openghg_inversions.inversion_data#

class openghg_inversions.inversion_data.RhimeMergedData(fp_all: dict, site_options: _SiteOptions)#

Bases: object

Merged RHIME data and complete site-aligned metadata between stages.

Parameters:

  • fp_all – Merged per-site datasets plus shared flux, boundary-condition, and calibration entries.

  • site_options – Complete site-aligned acquisition options retained after retrieval or filtering.

Notes

This is a supported orchestration handoff. Its datasets remain backend-neutral and may be Dask-backed; later stages must treat them as borrowed.

property averaging_period: tuple[str | None, ...]#

Retained averaging periods aligned to sites.

fp_all: dict#
property platform: tuple[str | None, ...]#

Retained observation platforms aligned to sites.

site_options: _SiteOptions#
property sites: tuple[str, ...]#

Retained site names.

class openghg_inversions.inversion_data.RhimePreparedInputs(inv_inputs: Dataset, basis_functions: FluxWeightedBasis, site_metadata: Dataset)#

Bases: object

Modern RHIME preparation and durable serialization contract.

Site labels and site-aligned metadata are owned by site_metadata. Integer inv_inputs.site_indicator values are derived as zero-based project-schema positions into its site coordinate; they are distinct from CF compression-by-gathering indexes. The invariant is site_metadata.site[site_indicator] == nmeasure.site. site_names remains available in memory for existing model code, but is regenerated from site_metadata rather than serialized.

Parameters:

  • inv_inputs – Canonical inversion inputs consumed by RHIME model builders.

  • basis_functions – Retained flux basis object used to derive output-boundary basis and flux arrays.

  • site_metadata – Dataset indexed by the authoritative site coordinate. Every variable contains exactly one value per site, and averaging_period is required. Observation metadata that is genuinely constant per site may also be stored here. Values that vary within a site, such as satellite or aircraft release locations, must instead remain observation-aligned arrays. Such arrays may be carried alongside the inversion arrays without implying that model builders consume them.

Raises:

ValueError – If site metadata, measurement indexing, or multi-source labels are inconsistent.

property averaging_period: tuple[str | None, ...]#

Return averaging periods aligned to sites.

property basis_artifact_path: str | None#

Return the provenance-only basis artifact path.

property basis_artifact_source: str#

Return retained basis provenance, defaulting to generated.

basis_functions: FluxWeightedBasis#
classmethod from_datatree(dt: DataTree) → Self#

Construct prepared RHIME inputs from a version-1 DataTree.

Parameters:

dt – DataTree using the openghg_inversions.rhime_prepared_inputs schema.

Returns:

Reconstructed prepared inputs with the canonical inversion-input MultiIndexes and embedded basis object restored, including retained multisource basis ordering.

Raises:

  • ValueError – If the prepared schema, site metadata, serialized MultiIndex, site indicators, or source labels are malformed.

  • KeyError – If a required child node is missing.

classmethod from_legacy_inputs(inv_inputs: Dataset, basis_functions: FluxWeightedBasis, sites: Sequence[str], averaging_period: Sequence[str | None], basis_artifact_source: str | None = None, basis_artifact_path: str | None = None, site_lats: Sequence[float] | None = None, site_lons: Sequence[float] | None = None) → Self#

Adapt the former positional fields to the labeled-data contract.

Parameters:

  • inv_inputs – Canonical inversion inputs.

  • basis_functions – Retained basis object.

  • sites – Site labels in indicator-decoding order.

  • averaging_period – Observation periods aligned to sites.

  • basis_artifact_source – Optional basis provenance value.

  • basis_artifact_path – Optional basis provenance path.

  • site_lats – Optional legacy release latitudes, one per site.

  • site_lons – Optional legacy release longitudes, one per site.

Returns:

Prepared inputs using labeled site metadata.

inv_inputs: Dataset#
classmethod load(file_path: str | Path) → Self#

Load prepared RHIME inputs from a NetCDF or Zarr artifact.

Parameters:

file_path – Prepared-input artifact previously written by save.

Returns:

Fully loaded prepared RHIME inputs with no open file handles.

Raises:

  • OSError – If the artifact cannot be opened.

  • RuntimeError – If all available storage backends fail at runtime.

  • ValueError – If the artifact schema or metadata is invalid.

  • KeyError – If a required child node is missing.

save(output_file: str | Path, output_format: Literal['netcdf', 'zarr'] | None = None) → None#

Save prepared RHIME inputs to NetCDF or Zarr.

Parameters:

  • output_file – Destination artifact path. Saving writes and may overwrite this artifact.

  • output_format – Storage format. When omitted, infer it from a .nc or .zarr suffix. An explicit format adds or replaces the corresponding suffix.

Raises:

ValueError – If metadata is invalid or the output format cannot be inferred.

site_metadata: Dataset#
property sites: tuple[str, ...]#

Return retained site labels in indicator-decoding order.

to_datatree() → DataTree#

Convert prepared RHIME inputs to the versioned DataTree schema.

The basis object is embedded in the artifact. basis_artifact_path is retained only as provenance and is never read while serializing or reconstructing the prepared inputs.

Returns:

DataTree containing inv_inputs, basis_functions, and site_metadata child nodes.

Raises:

ValueError – If site metadata or inversion inputs no longer satisfy the prepared-input invariants.

validated() → Self#

Return a freshly canonicalized copy of these prepared inputs.

This re-establishes the semantic invariants after possible in-place mutation of the contained xarray objects.

Returns:

Prepared inputs normalized from the current xarray values.

openghg_inversions.inversion_data.data_processing_surface_notracer(species: str, sites: Sequence[str] | str, domain: str, averaging_period: list[str | None] | str | None, start_date: str, end_date: str, obs_data_level: list[str | None] | str | None = None, platform: list[str | None] | str | None = None, inlet: Sequence[str | slice | None] | str | None = None, instrument: list[str | None] | str | None = None, max_level: Sequence[int | None] | int | None = None, calibration_scale: str | None = None, met_model: list[str | None] | str | None = None, fp_model: str | None = None, fp_height: list[str | None | Literal['auto']] | Literal['auto'] | str | None = None, fp_species: str | None = None, emissions_name: list | None = None, use_bc: bool = True, bc_input: str | None = None, bc_store: str | None = None, obs_store: str | list[str] | None = None, footprint_store: str | list[str] | None = None, emissions_store: str | None = None, split_by_sectors: bool = False, averagingerror: bool = True, save_merged_data: bool = False, merged_data_name: str | None = None, merged_data_dir: str | None = None, output_name: str | None = None, flux_non_finite_check: Literal['lazy', 'count'] = 'lazy') → tuple[dict, list, list, list, list, list]#

Retrieve and prepare surface or column datasets from OpenGHG stores.

Use for forward simulations and model-data comparisons that do not use tracers.

Parameters:

  • species – Atmospheric trace gas species of interest e.g. “co2”

  • sites – Measurement station/site abbreviation, or a sequence of them, e.g. "MHD" or ["MHD", "TAC"]. NOTE: for satellite, pass as “satellitename-obs_region” eg “GOSAT-BRAZIL” and pass corresponding platform as “satellite”

  • domain – Model domain region of interest; e.g. “EUROPE”

  • averaging_period – Averaging period to apply to mole fraction data, either scalar or aligned to sites.

  • start_date – Date from which to gather data; e.g. “2020-01-01”

  • end_date – Date until which to gather data; e.g. “2020-02-01”

  • obs_data_level – ICOS observation data level, either scalar or aligned to sites. For non-ICOS sites use None.

  • platform – Observation platform, either scalar or aligned to sites.

  • inlet – Observation inlet selector, either scalar or aligned to sites. Entries may be strings, legacy slice selectors, or None.

  • instrument – Observation instrument, either scalar or aligned to sites.

  • max_level – Maximum atmospheric level to extract, either scalar or aligned to sites. This is required for satellite/site-column data.

  • calibration_scale – Convert measurements to defined calibration scale

  • met_model – Meteorological model used in the LPDM, either scalar or aligned to sites.

  • fp_model – LPDM used for generating footprints.

  • fp_height – Inlet height used in footprints for corresponding sites.

  • fp_species – Species name associated with footprints in the object store

  • emissions_name – List of keywords args associated with emissions files in the object store. Corresponds to source in OpenGHG.

  • use_bc – Option to include boundary conditions in model

  • bc_input – Variable for calling BC data from ‘bc_store’ - equivalent of ‘emissions_name’ for fluxes.

  • bc_store – Name of object store to retrieve boundary conditions data from.

  • obs_store – Name of object store to retrieve observations data from.

  • footprint_store – Name of object store to retrieve footprints data from.

  • emissions_store – Name of object store to retrieve emissions data from.

  • flux_non_finite_check – Non-finite flux handling mode. "lazy" applies zero-fill lazily and records attrs; "count" computes count metadata once and warns if non-finite values are present.

  • split_by_sectors – If True, calculate sector-resolved fp_x_flux_sectoral in ModelScenario. If False (default), combine all flux sources into a single fp_x_flux pathway.

  • averagingerror – Adds the variability in the averaging period to the measurement error if set to True.

  • save_merged_data – Save forward simulations data and observations.

  • merged_data_name – Filename for saved forward simulations data and observations.

  • merged_data_dir – Directory path for for saved forward simulations data and observations.

  • output_name – Optional name used to create merged data name.

Returns:

containing

  • fp_all: dictionary containing flux data (key “.flux”), bc data (key “.bc”), and observations data (site short name as key)

  • sites: Updated list of sites. All put in upper case and if data was not extracted correctly for any sites, drop these from the rest of the inversion.

  • inlet: List of inlet height for the updated list of sites

  • fp_height: List of footprint height for the updated list of sites

  • instrument: List of instrument for the updated list of sites

  • averaging_period: List of averaging_period for the updated list of sites

Return type:

tuple

Raises:

  • SearchError – If no requested site has both observations and footprints.

  • ValueError – If aligned options have invalid lengths, emissions are not specified, observation units are unavailable or incompatible, or required error inputs are absent.

Notes

This function reads OpenGHG stores, emits progress messages and warnings, and may save a merged-data artifact. The first retained scenario defines the unit target requested for later sites; fp_all[".units"] stores that unit’s scale against mol/mol.

openghg_inversions.inversion_data.load_merged_data(merged_data_dir: str | Path, species: str | None = None, start_date: str | None = None, output_name: str | None = None, merged_data_name: str | None = None, output_format: Literal['pickle', 'netcdf', 'zarr', 'zarr.zip'] | None = None) → dict#

Load fp_all dictionary from a file in merged_data_dir.

The name of the pickle file can be specified using merged_data_name, or a standard name will be created given species, start_date, and output_name.

If merged_data_name is not given, then species, start_date, and output_name must be provided.

This function tries to automatically find a compatible format of merged data, if a format is not specified. First, it checks for data in “zarr” (or zipped zarr) format, then in netCDF, and finally in pickle.

Note: if data is stored in a zarr ZipStore, then the data is eagerly loaded, since the data needs to loaded before the zip file is closed.

Parameters:

  • merged_data_dir – path to directory where merged data will be saved

  • species – species of inversion

  • start_date – start date of inversion period

  • output_name – output name parameter used for inversion run

  • merged_data_name – name to use for saved data.

  • output_format – format of data to load (if not specified, this will be inferred).

Returns:

fp_all dictionary

openghg_inversions.inversion_data.prepare_rhime_inputs(*, species: str, sites: list[str], domain: str, averaging_period: Sequence[str | None] | str | None, start_date: str, end_date: str, output_name: str, flux_sources: list[str], split_by_sectors: bool = False, bc_store: str = 'user', obs_store: str = 'user', footprint_store: str = 'user', emissions_store: str = 'user', met_model: Sequence[str | None] | str | None = None, fp_model: str | None = None, fp_height: Sequence[str | None] | str | None = None, fp_species: str | None = None, inlet: Sequence[str | slice | None] | str | None = None, instrument: Sequence[str | None] | str | None = None, max_level: Sequence[int | None] | int | None = None, calibration_scale: str | None = None, obs_data_level: Sequence[str | None] | str | None = None, platform: Sequence[str | None] | str | None = None, use_tracer: bool = False, use_bc: bool = True, fp_basis_case: str | None = None, basis_directory: str | None = None, bc_basis_case: str = 'NESW', bc_basis_directory: str | Path | None = None, country_directory: str | None = None, bc_input: str | None = None, basis_algorithm: str = 'weighted', nbasis: int = 100, filters: Any = None, fix_basis_outer_regions: bool = False, averaging_error: bool = True, bc_freq: str | None = None, reload_merged_data: bool = False, save_merged_data: bool = False, merged_data_dir: str | None = None, merged_data_name: str | None = None, basis_output_path: str | None = None, min_error: Literal['percentile', 'residual'] | dict[str, float] | None | int | float = 0.0, min_error_options: Mapping[str, Any] | None = None, flux_non_finite_check: Literal['lazy', 'count'] = 'lazy') → RhimePreparedInputs#

Prepare modern RHIME inputs without exposing legacy fixedbasis containers.

Observation filters are applied once to merged data before basis loading or generation. The same filtered site datasets and aligned metadata are then used for sensitivity construction.

Parameters:

  • species – Primary gas or tracer name used for object-store lookup and output naming.

  • sites – Requested observation site names.

  • domain – Model domain name.

  • averaging_period – Observation averaging period, either scalar or site-aligned.

  • start_date – Inclusive inversion start date.

  • end_date – Exclusive inversion end date.

  • output_name – Base output name used for data and basis artifacts.

  • flux_sources – OpenGHG flux source values requested for the run.

  • split_by_sectors – Whether to keep sector-resolved sensitivity inputs with a source provenance coordinate. Semantic sector names are applied later by the model specification.

  • inlet – Inlet selector, either scalar or aligned to sites. Entries may be strings, legacy slice selectors, or None.

  • fp_height – Footprint inlet height, either scalar or aligned to sites.

  • instrument – Observation instrument, either scalar or aligned to sites.

  • platform – Observation platform, either scalar or aligned to sites.

  • obs_data_level – Observation data level, either scalar or aligned to sites.

  • met_model – Footprint meteorological model, either scalar or aligned to sites.

  • max_level – Maximum column level, either scalar or aligned to sites. Entries must be integers or None.

  • min_error – Numeric minimum error or "residual"/"percentile" calculation method.

  • min_error_options – Calculated minimum-error options. The only supported key is boolean by_site.

  • use_tracer – Unsupported placeholder for tracer inversions, where an additional species constrains the primary species through linked forward models.

  • flux_non_finite_check – Non-finite flux handling mode. "lazy" applies zero-fill lazily and records attrs; "count" computes count metadata once and warns if non-finite values are present.

Returns:

Modern RHIME prepared inputs containing canonical inv_inputs and a retained BasisFunctions object.

Raises:

ValueError – If site options are empty, duplicated, misaligned, or have invalid types, or if minimum-error options are invalid.

openghg_inversions.inversion_data.prepare_rhime_inputs_from_xarray(data: DataTree | Mapping[str, Dataset], *, basis_functions: FluxWeightedBasis, sites: Sequence[str] | None = None, averaging_period: str | Sequence[str | None] | Mapping[str, str | None] | None = None, bc_freq: str | None = None, min_error: str | dict[str, float] | float = 0.0, min_error_per_site: bool = False, start_date: str | datetime | datetime64 | Timestamp | None = None) → RhimePreparedInputs#

Create canonical RHIME inputs from source-neutral xarray site data.

Input must be an ordered mapping or a DataTree with one direct child Dataset per site. Mapping keys and DataTree child names define site order unless sites explicitly selects and orders them. Direct Dataset, dense Dataset(site, time), and pre-stacked nmeasure layouts are rejected.

Each site dataset must follow the module-level variable contract. Cached fp_x_flux or fp_x_flux_sectoral is projected through basis_functions.sensitivity before site-time observations are gathered to nmeasure. Existing H is accepted directly. The retained operator and prior flux are preserved; artifact provenance may be added to a new BasisFunctions value in the returned prepared inputs.

Every row is active. Per-site time coordinates must be explicit, nonempty, unique datetime64 values without NaT; valid non-monotonic order is preserved. Observation variables must have exactly dimension ("time",). Required observations, projected H, and optional H_bc must be finite. Required observation and error fields, selected H or cache, and optional H_bc must have the same exact nonempty unit string as mf across all sites; no conversion is performed. Release-coordinate pairs are all-or-none across retained sites. Labels on an explicit source dimension are nonempty, unique Python or NumPy strings and are never coerced from bytes or numbers. Repeated source values in a gathered source-specific state MultiIndex remain valid.

Persist and reopen canonical artifacts with RhimePreparedInputs.save and load(); do not pass serialized nmeasure data back to this adapter. The reserved fixed_baseline input is rejected until a reusable semantic Baseline component is available; use sampled H_bc or omit it.

Parameters:

  • data – Site data as a DataTree or site-to-Dataset mapping.

  • basis_functions – Self-contained retained basis object, including the prior/reference flux used for posterior reconstruction. Multisector data require matching ordered source coordinates on retained flux and any source-specific operator.

  • sites – Optional site selection and order.

  • averaging_period – One common period, a site-aligned sequence or mapping, or None.

  • bc_freq – Frequency used to expand sampled H_bc contributions.

  • min_error – Minimum model error accepted by make_inv_inputs.

  • min_error_per_site – Whether calculated minimum errors vary by site. Defaults to False, matching the OpenGHG-backed RHIME preparation route.

  • start_date – Optional frequency anchor passed to make_inv_inputs.

Returns:

Canonical prepared inputs accepted by openghg_inversions.rhime.run_rhime_from_prepared_inputs().

Raises:

  • TypeError – If the input container or mapping values are unsupported, or basis_functions is not a BasisFunctions object.

  • ValueError – If site data violate the documented variable, dimension, source-order, or site-alignment contract, or contain the deferred fixed_baseline extension.