Ramsden real-data validation

Outcome

The experimental implementation at model commit f0d27d7 closely reproduces the retained original January 2019 inversion. The annualized UK fossil-fuel methane posterior mean changes by only 0.96%, all 49 regional 95% posterior intervals overlap, and the regional fossil-fuel posterior-mean correlation is 0.980.

Observation-space posterior latent means are also nearly identical. The original-versus-modern correlations are 0.9983 for methane and 0.999963 for ethane.

The validation also reproduces the principal scientific warning in the original analysis: both ethane model-error parameters concentrate at their 50 ppt upper bound. Modern ethane 90% posterior-predictive coverage is only 32.9% at MHD and 40.4% at TAC. This supports the implementation as a faithful, runnable historical comparison; it does not establish that the current ethane likelihood is scientifically production-ready.

What was tested

The validation used retained real-data observations and sensitivity matrices for January 2019:

Final validation configuration

Item	Configuration
Code	Experimental model at f0d27d7.
Period	1–31 January 2019, nominal four-hour observations.
Methane observations	331 values at MHD, TAC, RGL, BSD, and HFD; numeric units ppb.
Ethane observations	117 values at MHD and TAC; numeric units ppt.
Spatial state	49 retained quadtree basis regions, with identical reconstructed maps for both channels.
Methane design	Fossil and non-fossil source terms plus four boundary terms.
Ethane design	Fossil source term only plus four boundary terms.
Coupling	Shared 49-region fossil methane scaling state; no non-fossil ethane contribution.
Likelihood	Independent Gaussian channels with measurement error plus inferred absolute site-month model error; min_error=0.
Sampler	Four NumPyro NUTS chains; 1,000 tuning and 1,000 retained draws per chain; target acceptance 0.9.

Priors followed the paper’s printed specification:

• fossil and non-fossil scaling: zero-truncated Gaussian with mean 1 and standard deviation 0.5;

• methane model error: Uniform(10, 50) ppb by site and month;

• ethane model error: Uniform(20, 50) ppt by site and month;

• methane boundary scaling: zero-truncated Gaussian with mean 1 and standard deviation 0.05;

• ethane boundary scaling: zero-truncated Gaussian with mean 1 and standard deviation 0.5; and

• physical molar ethane:methane ratio: 0.075 * Uniform(0.1, 2.7) independently by region.

Terminology and truth

This is a real-data inversion. There is no known state-space or flux truth and no flux-recovery score.

The closest comparator is the retained original January 2019 posterior. It uses the same retained observations, designs, basis state, and month, but a historical sampler and a different effective methane model-error prior.

The paper is a second, less direct comparator. It reports monthly inversions summarized as annual 2015–2019 UK totals. January’s annualized flux rate and an annual result are not like-for-like.

Observation-space bias and RMSE below compare deterministic posterior latent means with retained observations. Posterior-predictive coverage instead uses simulated posterior observations and includes likelihood uncertainty. Neither kind of observation-space agreement demonstrates emission accuracy.

Input provenance and limitations

The retained output supplied observations, timestamps, source-separated designs, boundary designs, site stacking, state ordering, and the gridded basis representation. Its provenance identifies:

• UKGHG/EDGAR fossil and non-fossil methane sources;

• an ethane fossil inventory already scaled by the reference ratio 0.075;

• a zero non-fossil ethane contribution; and

• a 49-region sensitivity-driven quadtree basis.

The exact historical measurement-error arrays were not retained. Methane within-bin variability and ethane repeatability were reconstructed from frozen observation products. One missing TAC ethane repeatability value was filled with that site’s positive median. This is the main input-level limitation on a strict original-versus-modern comparison.

The available current OpenGHG object store did not contain usable footprint and observation products for rebuilding this historical case. The validation therefore converted the retained sensitivities into canonical prepared datasets rather than rerunning modern retrieval and preparation.

Sampling diagnostics

The final four-chain run completed both likelihoods and posterior-predictive generation with zero divergences.

Modern sampling diagnostics

Diagnostic	Value
Maximum unrounded R-hat	1.0051
Minimum bulk effective sample size	1,447
Minimum tail effective sample size	556
Mean acceptance probability	0.941
Maximum tree depth	8
Divergences	0

The paper used an adaptive random-walk Metropolis-Hastings sampler, discarded the first 50%, and retained every 100th subsequent sample. It did not report chain count, R-hat, effective sample sizes, or equivalent modern convergence diagnostics, so convergence cannot be compared directly.

Flux-space results

The retained per-region UK prior-flux weights were recovered by dividing each stored regional posterior flux trace by its scaling trace and taking the median over draws. Their sums exactly reproduce the stored fossil and non-fossil country priors. Applying the same weights to the modern posterior is valid because basis-map and state ordering were checked exactly.

The table reports annualized January methane flux rates in Tg CH4 yr-1, not twelve-month annual totals.

Annualized January UK methane flux

Sector	Retained original mean (95% interval)	Modern mean (95% interval)	Change in mean
Fossil fuel	0.28094 (0.21625–0.35400)	0.27825 (0.21405–0.35210)	-0.96%
Non-fossil	1.57795 (1.34910–1.80753)	1.65733 (1.41293–1.89559)	+5.03%
Total	1.85888 (1.61077–2.10658)	1.93557 (1.69621–2.17447)	+4.13%

The fossil-fuel aggregate, which is the quantity the added ethane channel is intended to constrain, is reproduced within 1%. The larger aggregate change is in the methane-only non-fossil sector.

Regional state results

The following metrics compare posterior summaries for the same 49 basis regions. Scaling states and ratio multipliers are dimensionless.

Regional state comparison

Parameter	Correlation of posterior means	RMSE of posterior means	Regions with overlapping 95% intervals
Fossil methane scaling	0.9799	0.1409	49/49
Non-fossil methane scaling	0.8889	0.1444	49/49
Ratio multiplier	0.9369	0.2087	49/49

The physical molar ratio means moles of ethane divided by moles of fossil-fuel methane. Because the retained ethane design already contains 0.075, it equals 0.075 * ratio_multiplier.

Physical ethane:methane ratio

Summary	Retained original	Modern
Unweighted mean over all region-draw values	0.08209	0.08320
UK fossil-methane-flux-weighted mean	0.08969	0.09121
Weighted 95% interval	0.07061–0.11703	0.07125–0.11663

The weighted mean changes by 1.7%.

Observation-space results

The table compares deterministic posterior latent means with observations. Methane bias and RMSE are in ppb; ethane values are in ppt.

Site-level posterior latent-mean fit

Gas	Site	Retained original bias, RMSE	Modern bias, RMSE
CH4	MHD	-1.610, 6.561	-1.883, 7.242
CH4	TAC	-2.498, 9.929	-2.702, 10.163
CH4	RGL	-3.402, 15.198	-2.041, 14.480
CH4	BSD	-0.629, 5.530	+0.393, 5.292
CH4	HFD	+0.867, 10.640	+0.564, 10.406
C2H6	MHD	-3.267, 229.104	-3.982, 229.018
C2H6	TAC	+11.159, 165.840	+11.510, 165.908

Overall methane RMSE changes from 10.126 to 10.004 ppb, and overall ethane RMSE from 206.038 to 206.003 ppt. Pointwise original-versus-modern latent-mean correlations are 0.99831 for methane and 0.999963 for ethane.

Modern 90% posterior-predictive coverage is 93.3–100% across methane sites, 32.9% for ethane at MHD, and 40.4% for ethane at TAC. Historical coverage cannot be computed on the same definition because the retained file does not contain equivalent likelihood-level predictive draws or the original measurement-error vector.

Model-error and boundary results

The four boundary states for each channel are numerically close. They are not assigned compass names here because the retained boundary-column ordering cannot be established safely from the output alone.

Boundary posterior means

Channel	Retained original	Modern
CH4	1.00177, 1.00357, 0.95488, 0.99873	1.00065, 1.00294, 0.96018, 0.99906
C2H6	1.10707, 1.19605, 0.00531, 0.80497	1.10547, 1.18075, 0.00531, 0.80432

Monthly site-level absolute model-error means are:

Absolute model-error posterior means

Channel and site order	Retained original	Modern
CH4: MHD, TAC, RGL, BSD, HFD (ppb)	6.591, 7.861, 4.686, 4.090, 9.108	10.218, 10.628, 10.365, 10.286, 10.685
C2H6: MHD, TAC (ppt)	49.964, 49.898	49.964, 49.896

The methane values are not like-for-like: the modern validation imposed the paper’s printed 10 ppb lower bound, whereas the retained implementation used a zero lower bound. Both implementations give virtually identical ethane model error and press the 50 ppt upper bound.

Primary figure

Panel A compares annualized UK methane flux rates in Tg CH4 yr-1. Error bars are 95% posterior intervals. Retained-original and modern bars represent January states; hatched paper bars are annual 2019 values shown only for scale. Panels B and C compare deterministic posterior latent-mean RMSE by site for methane in ppb and ethane in ppt.

Comparison with Ramsden et al. (2022)

The modern regional posterior-mean physical ratios span 0.0386–0.1997, with median 0.0674. This is inside the paper’s 2015–2019 regional range 0.009–0.2 and is consistent in scale with the independent ratios quoted there: approximately 0.06 for UK gas leaks and 0.088 (0.04–0.18) for sampled North Sea plumes. Those external observations are sparse and local, so this is context rather than a validation score.

The paper’s annual 2019 methane estimates and the modern January annualized state are:

Paper context versus the modern January state

Sector	Paper annual 2019	Modern January annualized state
Fossil fuel	0.25 (0.23–0.28)	0.278 (0.214–0.352)
Non-fossil	1.90 (1.78–2.04)	1.657 (1.413–1.896)
Total	2.15 (2.03–2.28)	1.936 (1.696–2.174)

Units are Tg CH4 yr-1 and intervals are 95% posterior intervals. All interval pairs overlap, but seasonal and annual averaging prevent a stronger reproduction claim.

The paper also reports that adding ethane lowers fossil emissions by about 15% relative to a methane-only inversion and reduces fossil interval width by 15% on average and up to 35%. The present validation has no matched methane-only counterfactual, so those claims were not tested.

Paper inconsistencies affecting interpretation

Three inconsistencies in the paper matter for exact reproduction:

1. The methods state a methane model-error prior of Uniform(10, 50) ppb, but the results report an overall mean of 7.75 ppb and say 75% of site-month means lie between 5 and 10 ppb. Those results are impossible under the printed prior. The retained Uniform(0, 50) implementation resolves the contradiction.

2. The paper prints the ethane model-error bounds in ppb, while its ethane figures use pmol mol-1, equivalent to ppt. The retained data and configuration also support ppt.

3. The text describes one fixed-ratio sensitivity case as April 2019, while the corresponding figure caption says May 2015.

Implementation differences

The modern port preserves the paper-shaped equations, not the historical software stack:

• PyMC/NumPyro NUTS and ArviZ replace the custom adaptive random-walk sampler and bespoke output format.

• Prepared canonical datasets replace the historical loaders, cache, and configuration parser.

• Source labels, state indexes, basis maps, unit scales, ratio provenance, and boundary configuration are validated explicitly.

• The direct physical ratio and the multiplier of a pre-scaled tracer design are represented separately.

• The module does not reproduce historical gridded/country post-processing.

The historical generalized branch also contains documented correctness problems, including a stale boundary proposal state, brittle posterior slicing, and omission of the sampled ratio from ethane gridded/country output. The comparison therefore uses explicit state variables, methane aggregate fluxes, boundary states, physical ratios, and observation latent means rather than the suspect stored national ethane result.

Assessment and follow-up

The evidence supports computational fidelity of the modern forward model and posterior target for this retained January case:

• fossil UK methane flux is reproduced within 1%;

• all regional 95% posterior intervals overlap;

• fossil and ratio spatial patterns agree strongly;

• boundary states and observation-space latent means are nearly identical; and

• the distinctive ethane model-error ceiling is reproduced.

It does not establish full reproduction of the paper. That would require:

• all 60 monthly inversions for 2015–2019;

• the exact original measurement-error inputs;

• a matched methane-only counterfactual; and

• the paper’s annual aggregation and uncertainty comparison.

The low ethane posterior-predictive coverage should remain visible in any future milestone work. A generic linked-tracer API should preserve the successful shared-state and ratio contracts while making data preparation, unit provenance, predictive diagnostics, and tracer-aware output explicit.

Reference

A. E. Ramsden et al. (2022), “Quantifying fossil fuel methane emissions using observations of atmospheric ethane and an uncertain emission ratio”, Atmospheric Chemistry and Physics 22, 3911–3929, doi:10.5194/acp-22-3911-2022.