openghg_inversions.flux_sanitization#

Helpers for applying a consistent non-finite flux policy.

OpenGHG flux data can contain NaN or infinite values even though those values do not have a scientific interpretation for flux weighting. This module normalises those values to zero early enough to protect generated and retained basis workflows.

The sanitation policy is stored in one namespaced xarray attribute as compact JSON. xarray attrs are the metadata channel that survives NetCDF-like serialisation, but they should contain simple serialisable values rather than Python objects. Keeping the metadata behind a small dataclass gives callers a typed interface without spreading many string attribute names through the code.

class openghg_inversions.flux_sanitization.FluxNonFiniteMetadata(schema_version: int = 1, policy: str = 'zero_fill', fill_value: float = 0.0, checked: str = 'not_counted', context: str | None = None, source: str | None = None, count: int | None = None, total: int | None = None, fraction: float | None = None)#

Bases: object

Machine-readable metadata describing non-finite flux handling.

The dataclass is the in-memory representation used by this module. Its serialised form is a JSON object stored under NONFINITE_METADATA_ATTR so the metadata can survive NetCDF/Zarr roundtrips without placing a Python object in DataArray.attrs.

Variables:

  • schema_version (int) – JSON metadata schema version.

  • policy (str) – Non-finite handling policy. Currently only "zero_fill" is written locally.

  • fill_value (float) – Value used to replace non-finite cells.

  • checked (str) – Whether the array was lazily sanitised without counting values or audited with an exact count.

  • context (str | None) – Human-readable processing context where the policy was applied.

  • source (str | None) – Optional flux source label.

  • count (int | None) – Exact non-finite count when checked is "computed".

  • total (int | None) – Exact total cell count when checked is "computed".

  • fraction (float | None) – Exact non-finite fraction when checked is "computed".

checked: str#
context: str | None#
count: int | None#
declares_zero_fill() → bool#

Return whether this metadata declares a zero-fill policy.

Returns:

True when the policy and fill value describe replacement with zero.

fill_value: float#
fraction: float | None#
classmethod from_attrs(attrs: Mapping[Any, Any]) → FluxNonFiniteMetadata | None#

Parse local non-finite metadata from xarray attrs.

Parameters:

attrs – Attribute mapping from a DataArray or Dataset.

Returns:

Parsed metadata when local JSON metadata is present, otherwise None.

classmethod from_json(value: object) → FluxNonFiniteMetadata#

Parse metadata from a JSON string or mapping.

Parameters:

value – JSON string, bytes, or mapping stored in xarray attrs.

Returns:

Parsed metadata.

Raises:

ValueError – If the stored value is not valid metadata.

policy: str#
schema_version: int#
source: str | None#
to_json() → str#

Return compact JSON suitable for storing in xarray attrs.

Returns:

JSON object encoded as a compact string.

to_json_dict() → dict[str, str | int | float | None]#

Return a JSON-serialisable dictionary.

Returns:

Dictionary containing only primitive or null values suitable for JSON encoding in xarray attrs.

total: int | None#
exception openghg_inversions.flux_sanitization.NonFiniteFluxWarning#

Bases: UserWarning

Warning emitted when non-finite flux values are replaced.

openghg_inversions.flux_sanitization.copy_flux_nonfinite_attrs(target: _XarrayObjectT, flux: Dataset | DataArray) → _XarrayObjectT#

Copy machine-readable non-finite flux metadata onto an output object.

Parameters:

  • target – Dataset or DataArray receiving metadata.

  • flux – Dataset or DataArray whose attrs may contain local non-finite flux metadata.

Returns:

A shallow copy of target with local JSON non-finite metadata copied from flux when present.

openghg_inversions.flux_sanitization.sanitize_flux_nonfinite(flux: DataArray, *, context: str, source: str | None = None, check: Literal['lazy', 'count'] = 'lazy', trust_attrs: bool = True, warn: bool = False) → DataArray#

Replace non-finite flux values with zero and record the policy.

The default check="lazy" creates a lazy xarray/dask graph and does not compute a count. Use check="count" when exact non-finite counts are needed for audit logs, accepting that this computes the finite mask.

Parameters:

  • flux – Flux field to sanitize.

  • context – Human-readable processing context used in metadata and warnings.

  • source – Optional flux source label to record in metadata.

  • check – "lazy" to avoid computing counts, or "count" to compute exact count/total/fraction metadata.

  • trust_attrs – If True, return flux unchanged when local attrs already declare a zero-fill policy. A previous lazy replacement cannot be upgraded to an exact count because the original values are gone.

  • warn – Emit a warning when values are replaced or a lazy fallback is applied.

Returns:

Sanitized flux with non-finite values replaced by 0.0 and local JSON policy metadata attached.

Raises:

ValueError – If check is not "lazy" or "count".

Warns:

NonFiniteFluxWarning – If values cannot be counted because a lazy replacement already occurred, or if warn is True and values are replaced or a lazy fallback is applied.