openghg_inversions.inversion_data.get_data#

Retrieve and merge observations, footprints, fluxes, and boundary data.

data_processing_surface_notracer expands scalar site options, keeps their positions aligned during retrieval failures, assembles per-site model scenarios, and can save the resulting merged-data artifact. Retrieval and optional saving have external object-store and filesystem side effects; loading saved merged data is handled by openghg_inversions.inversion_data.serialise. The first successful scenario supplies the unit target forwarded to later OpenGHG ModelScenario merges.

openghg_inversions.inversion_data.get_data.add_obs_error(sites: list[str], fp_all: dict, add_averaging_error: bool = True) → None#

Create mf_error variable.

The mf_error variables contains either mf_repeatablility, mf_variability or the square root of the sum of the squares of both, if add_averaging_error is True.

This function modifies fp_all in place, adding mf_error and making sure that both mf_repeatability and mf_variability are present.

Note: if averaging_period is specified in data_processing_surface_notracer, then OpenGHG will add an mf_variability variable with the standard deviation of the obs over the specified period. If mf_variability is already present (for instance, for Picarro data), then the existing variable is over-written. If the averaging_period matches the frequency of the data, this will make mf_variability zero (since the stdev of one value is 0).

Parameters:

  • sites – list of site names to process

  • fp_all – dictionary of ModelScenario objects, keyed by site names

  • add_averaging_error – if True, combine repeatability and variability to make mf_error variable. Otherwise, mf_error will equal mf_repeatability if it is present, otherwise it will equal mf_variability.

Returns:

None, modifies fp_all in place.

openghg_inversions.inversion_data.get_data.convert_to_list(x: Iterable[Any] | str | slice | int | Integral | None, length: int, name: str | None = None) → list[Any]#

Convert a scalar or sequence to a list of the expected size.

Parameters:

  • x – Scalar string/integer/slice/None to broadcast, or an iterable to copy.

  • length – Required output length.

  • name – Optional argument name used in error messages.

Returns:

A new list of the requested length.

Raises:

ValueError – If an iterable has the wrong length, or if x is neither a supported scalar nor an iterable.

openghg_inversions.inversion_data.get_data.data_processing_surface_notracer(species: str, sites: Sequence[str] | str, domain: str, averaging_period: list[str | None] | str | None, start_date: str, end_date: str, obs_data_level: list[str | None] | str | None = None, platform: list[str | None] | str | None = None, inlet: Sequence[str | slice | None] | str | None = None, instrument: list[str | None] | str | None = None, max_level: Sequence[int | None] | int | None = None, calibration_scale: str | None = None, met_model: list[str | None] | str | None = None, fp_model: str | None = None, fp_height: list[str | None | Literal['auto']] | Literal['auto'] | str | None = None, fp_species: str | None = None, emissions_name: list | None = None, use_bc: bool = True, bc_input: str | None = None, bc_store: str | None = None, obs_store: str | list[str] | None = None, footprint_store: str | list[str] | None = None, emissions_store: str | None = None, split_by_sectors: bool = False, averagingerror: bool = True, save_merged_data: bool = False, merged_data_name: str | None = None, merged_data_dir: str | None = None, output_name: str | None = None, flux_non_finite_check: Literal['lazy', 'count'] = 'lazy') → tuple[dict, list, list, list, list, list]#

Retrieve and prepare surface or column datasets from OpenGHG stores.

Use for forward simulations and model-data comparisons that do not use tracers.

Parameters:

  • species – Atmospheric trace gas species of interest e.g. “co2”

  • sites – Measurement station/site abbreviation, or a sequence of them, e.g. "MHD" or ["MHD", "TAC"]. NOTE: for satellite, pass as “satellitename-obs_region” eg “GOSAT-BRAZIL” and pass corresponding platform as “satellite”

  • domain – Model domain region of interest; e.g. “EUROPE”

  • averaging_period – Averaging period to apply to mole fraction data, either scalar or aligned to sites.

  • start_date – Date from which to gather data; e.g. “2020-01-01”

  • end_date – Date until which to gather data; e.g. “2020-02-01”

  • obs_data_level – ICOS observation data level, either scalar or aligned to sites. For non-ICOS sites use None.

  • platform – Observation platform, either scalar or aligned to sites.

  • inlet – Observation inlet selector, either scalar or aligned to sites. Entries may be strings, legacy slice selectors, or None.

  • instrument – Observation instrument, either scalar or aligned to sites.

  • max_level – Maximum atmospheric level to extract, either scalar or aligned to sites. This is required for satellite/site-column data.

  • calibration_scale – Convert measurements to defined calibration scale

  • met_model – Meteorological model used in the LPDM, either scalar or aligned to sites.

  • fp_model – LPDM used for generating footprints.

  • fp_height – Inlet height used in footprints for corresponding sites.

  • fp_species – Species name associated with footprints in the object store

  • emissions_name – List of keywords args associated with emissions files in the object store. Corresponds to source in OpenGHG.

  • use_bc – Option to include boundary conditions in model

  • bc_input – Variable for calling BC data from ‘bc_store’ - equivalent of ‘emissions_name’ for fluxes.

  • bc_store – Name of object store to retrieve boundary conditions data from.

  • obs_store – Name of object store to retrieve observations data from.

  • footprint_store – Name of object store to retrieve footprints data from.

  • emissions_store – Name of object store to retrieve emissions data from.

  • flux_non_finite_check – Non-finite flux handling mode. "lazy" applies zero-fill lazily and records attrs; "count" computes count metadata once and warns if non-finite values are present.

  • split_by_sectors – If True, calculate sector-resolved fp_x_flux_sectoral in ModelScenario. If False (default), combine all flux sources into a single fp_x_flux pathway.

  • averagingerror – Adds the variability in the averaging period to the measurement error if set to True.

  • save_merged_data – Save forward simulations data and observations.

  • merged_data_name – Filename for saved forward simulations data and observations.

  • merged_data_dir – Directory path for for saved forward simulations data and observations.

  • output_name – Optional name used to create merged data name.

Returns:

containing

  • fp_all: dictionary containing flux data (key “.flux”), bc data (key “.bc”), and observations data (site short name as key)

  • sites: Updated list of sites. All put in upper case and if data was not extracted correctly for any sites, drop these from the rest of the inversion.

  • inlet: List of inlet height for the updated list of sites

  • fp_height: List of footprint height for the updated list of sites

  • instrument: List of instrument for the updated list of sites

  • averaging_period: List of averaging_period for the updated list of sites

Return type:

tuple

Raises:

  • SearchError – If no requested site has both observations and footprints.

  • ValueError – If aligned options have invalid lengths, emissions are not specified, observation units are unavailable or incompatible, or required error inputs are absent.

Notes

This function reads OpenGHG stores, emits progress messages and warnings, and may save a merged-data artifact. The first retained scenario defines the unit target requested for later sites; fp_all[".units"] stores that unit’s scale against mol/mol.