openghg_inversions.serialization#

Shared serialization helpers for modern OpenGHG inversion artifacts.

This module contains the storage mechanics shared by modern artifact containers. It saves and eagerly loads xarray DataTree objects, converts ArviZ InferenceData groups to and from trees, and expands pandas MultiIndex coordinates into representations supported by NetCDF and Zarr.

Object-specific modules remain responsible for schema names, schema versions, required child nodes, and metadata validation. Prepared artifacts can use the CF compression-by-gathering convention through explicit encoding and decoding helpers. The older project-specific MultiIndex restoration remains deliberately forgiving so existing or partially malformed inversion-output artifacts remain loadable without reintroducing invalid indexes. Versioned schemas may request strict restoration, which rejects missing, malformed, empty, or inapplicable restoration metadata.

openghg_inversions.serialization.decode_cf_multiindexes(ds: Dataset, index_names: str | Iterable[str]) → Dataset#

Decode named CF gathering coordinates into pandas MultiIndexes.

The requested coordinates and their compress metadata are validated before calling cf_xarray. Names are always explicit: this deliberately avoids cf_xarray’s auto-detection path, which can drop unrelated coordinates when no compressed indexes are present.

Parameters:

  • ds – Dataset containing CF compression-by-gathering coordinates.

  • index_names – Explicit gathered-coordinate name or names to decode.

Returns:

A new Dataset with the requested coordinates decoded as MultiIndexes.

Raises:

ValueError – If no names are supplied or a requested coordinate has missing, malformed, ambiguous, or inapplicable CF compress metadata.

openghg_inversions.serialization.encode_cf_multiindexes(ds: Dataset, index_names: str | Iterable[str]) → Dataset#

Encode named MultiIndexes using CF compression by gathering.

This codec is intended for versioned prepared-input artifacts. Unlike reset_serialisation_multiindexes(), it emits the interoperable CF compress attribute and requires callers to name every index they intend to encode.

Parameters:

  • ds – Dataset containing the MultiIndex dimensions.

  • index_names – Explicit dimension name or names to encode.

Returns:

A new Dataset with the requested MultiIndexes encoded as integer gathering coordinates.

Raises:

ValueError – If no names are supplied, a requested dimension is not a named MultiIndex, level names are invalid or ambiguous, or existing compress metadata would be overwritten.

openghg_inversions.serialization.encode_multiindexes_for_storage(ds: Dataset) → Dataset#

Expand all semantic MultiIndexes and attach versioned schema metadata.

Parameters:

ds – Dataset containing zero or more pandas MultiIndex dimensions.

Returns:

A serialization copy with ordinary level coordinates and declarations of their owner, order, uniqueness, and reconstruction policy.

Raises:

ValueError – If an index has missing, repeated, or duplicate semantic labels.

openghg_inversions.serialization.inferencedata_from_datatree(dt: DataTree) → InferenceData#

Reconstruct ArviZ InferenceData from a group DataTree.

Parameters:

dt – DataTree containing root attributes and one child dataset per InferenceData group.

Returns:

Reconstructed InferenceData with root attributes and valid serialized MultiIndexes restored.

openghg_inversions.serialization.inferencedata_to_datatree(idata: InferenceData) → DataTree#

Convert ArviZ InferenceData groups to a serializable DataTree.

Parameters:

idata – InferenceData whose root attributes and groups should become the tree root and child nodes.

Returns:

DataTree containing the InferenceData root attributes and one child dataset per group.

openghg_inversions.serialization.load_inferencedata(file_path: str | Path) → InferenceData#

Load InferenceData and restore every valid declared MultiIndex.

Malformed declarations are removed and left expanded rather than guessed, matching restore_declared_multiindexes()’ forgiving default.

Parameters:

file_path – NetCDF file or Zarr store written by save_inferencedata().

Returns:

Fully loaded InferenceData with valid semantic indexes reconstructed.

openghg_inversions.serialization.normalise_declared_multiindex(ds: Dataset, dim: str, level_names: str | Iterable[str]) → Dataset#

Normalize a MultiIndex or its declared expanded form.

The owning dimension and ordered semantic level names are explicit. An already-indexed dataset is validated in place; an expanded representation is reconstructed only when every level is one-dimensional on dim and the resulting labels are unique.

openghg_inversions.serialization.open_datatree_loaded(file_path: str | Path) → DataTree#

Open and eagerly load a DataTree artifact.

NetCDF is first attempted with h5netcdf for compatibility with modern inversion outputs, then with xarray’s default engine. Loading happens while the file context is open, so the returned tree owns its data and does not retain references to closed file handles.

Parameters:

file_path – NetCDF or Zarr DataTree artifact to load.

Returns:

Fully loaded DataTree.

Raises:

  • OSError – If no backend can open the artifact.

  • RuntimeError – If opening fails due to a backend runtime error.

  • ValueError – If no backend can interpret the artifact.

openghg_inversions.serialization.reset_serialisation_multiindexes(ds: Dataset) → Dataset#

Expand xarray MultiIndexes before DataTree serialization.

Parameters:

ds – Dataset that may contain pandas MultiIndex dimensions.

Returns:

Dataset with MultiIndexes reset and restoration metadata stored in an attribute. The input dataset is not modified.

Raises:

ValueError – If a MultiIndex level is unnamed and cannot be restored unambiguously.

openghg_inversions.serialization.restore_declared_multiindexes(ds: Dataset, *, strict: bool = False) → Dataset#

Restore MultiIndexes from explicit storage declarations.

Parameters:

  • ds – Dataset carrying expanded coordinates and MULTIINDEX_DIMS_ATTR metadata.

  • strict – Raise a focused error for invalid metadata or semantic labels. By default, invalid declarations are removed and their coordinates remain expanded.

Returns:

Dataset with every valid declared MultiIndex restored explicitly.

openghg_inversions.serialization.restore_serialisation_multiindexes(ds: Dataset, *, strict: bool = False) → Dataset#

Restore valid MultiIndexes expanded during DataTree serialization.

In the default non-strict mode, malformed restoration metadata is discarded instead of raising. This is a compatibility guarantee for existing InversionOutput artifacts: valid index records are restored, while invalid records leave their level coordinates expanded. Strict mode raises for missing, empty, malformed, or inapplicable records.

Parameters:

  • ds – Dataset that may carry MULTIINDEX_DIMS_ATTR metadata.

  • strict – Raise for malformed restoration metadata instead of leaving index levels expanded. Strict restoration also requires at least one recorded MultiIndex dimension. The default preserves forgiving historical inversion-output loading.

Returns:

Dataset with valid recorded MultiIndexes restored and the private serialization attribute removed. The input dataset is not modified.

Raises:

ValueError – If strict is true and restoration metadata is malformed or cannot be applied to the dataset.

openghg_inversions.serialization.save_datatree(dt: DataTree, output_file: str | Path, output_format: Literal['netcdf', 'zarr'] | None = None) → None#

Save a DataTree to NetCDF or Zarr.

This writes the tree, replacing an existing destination artifact.

Parameters:

  • dt – DataTree to persist.

  • output_file – Requested artifact path. When output_format is supplied, the matching .nc or .zarr suffix is added or replaces an existing suffix as needed.

  • output_format – Storage format. When omitted, infer the format from an existing .nc or .zarr suffix.

Raises:

ValueError – If the format cannot be inferred or is unsupported.

openghg_inversions.serialization.save_inferencedata(idata: InferenceData, output_file: str | Path, output_format: Literal['netcdf', 'zarr'] | None = None) → None#

Save InferenceData through the declared MultiIndex boundary.

Parameters:

  • idata – InferenceData whose groups and root attributes should be saved.

  • output_file – Destination NetCDF file or Zarr store.

  • output_format – Explicit backend, or None to infer it from the path.