# Archived atlas methods

The current merged-data release is documented in [MERGED_RELEASE.md](MERGED_RELEASE.md). This page preserves the methods of the original nine-country release, available at [global/archive.html](https://phuybers3.github.io/enso-yield-atlas/global/archive.html).

# Global agricultural atlas: yields and ENSO response

The first release is available at https://phuybers3.github.io/enso-yield-atlas/global/. We provide a world map, dedicated country pages, and regional observation panels. We retain the existing ENSO explorer at the site root. We map observed average yields, reporting coverage, and fitted yield responses to ENSO. The separate trend layer remains planned.

## Release 2026-09-26

The frozen source release contains wheat, maize, paddy rice, and milled rice from the supplied HarvestStat Asia v0.1 and v0.2 packages. The interface combines the two rice forms into one Rice (paddy equivalent) choice. The current inputs cover Bangladesh, India, Indonesia, Japan, South Korea, Malaysia, Sri Lanka, Thailand, and Vietnam. Other countries remain selectable on the world map and have an explicit missing-coverage page. We do not create national observations or assign national values to subnational polygons. The current packages contain stable subnational groups; testing a real national-only input awaits the expanded database.

| Product | Eligible crop–season–basis series | Regions with at least one record | Eligible annual observations |
|---|---:|---:|---:|
| Wheat | 560 | 478 | 9,971 |
| Maize | 1,282 | 561 | 20,423 |
| Rice, paddy | 422 | 231 | 14,962 |
| Rice, milled | 1,147 | 467 | 18,878 |

The table above describes the original source forms. The combined rice view retains 1,559 series with observations and 33,393 annual records, after selecting one source form for each region–season–area-basis combination. Its local-default selection maps 617 rice regions across all nine countries.

Counts span all reported seasons and both area bases. One map selection chooses one season and one denominator per country. Series are not independent outcomes, and several seasons or rice forms can describe overlapping agricultural activity.

## Calculation and selection rules

An eligible observation has complete area and production reporting, positive area, finite nonnegative yield and production, and yield consistent with production divided by the recorded area to relative tolerance 1e-5. We preserve a zero yield when the standardized source supplies zero production with complete positive area. Source-coded missing values remain missing. The source packages already removed several known zero placeholders. We exclude 4,130 incompletely reported rows and 319 rows without a usable area basis across these four products.

We apply the documented East Java 2017 paddy correction from `verified_correction.json`. We join the Vietnamese Summer Rice label to Autumn Rice, preserving the Northern and Southern qualifiers. All other product and season labels are preserved. Japanese production is derived from planted area and published yield, as documented in the supplied package. Malaysian rice includes changes between planted and harvested denominators; we split those records into separate series.

The displayed mean is the arithmetic mean of eligible annual yields in the requested period. We never average crop components, seasons, or area bases together. Prepared periods are 1991–2020, 2001–2020, 2011–2020, 2015–2024, and available record. For a fixed period, completeness is the observation count divided by the full requested number of years. For available record, completeness uses the inclusive span from the first to the last valid observation. A selectable 80% rule filters map values without hiding the underlying series. The default requires one valid year. Actual dates and counts accompany every value.

Country-specific season defaults prefer Annual, Calendar Year, and All (Season), then Rabi for wheat. Otherwise the default is the season with the most eligible observations. Defaults are calculated from the full record and remain fixed when users change period. A country’s default area basis is the basis with the most observations in its selected season. An explicit season or area-basis choice restricts the map. Missing selections remain missing when moving between countries. Changing crop resets season to the documented default; period, area basis, measure, and coverage requirement are retained.

Yield and coverage use full-range color scales across countries for a given selection. Those values are never clipped. The ENSO response layer uses a fixed diverging −50% to +50% scale, with end colors including values beyond those limits; exact values remain in the regional panel and downloads. Gray denotes no mapped value under the current selection. The location list distinguishes unavailable country coverage, an unavailable crop or season, no eligible observations in the period, and insufficient temporal completeness. No data category asserts that a crop is absent.

## Combined rice on a paddy-equivalent basis

The crop selector offers one **Rice (paddy equivalent)** choice. We divide milled yield and production by **0.67**, equivalent to multiplying by about **1.4925**, and leave reported paddy weights and all land areas unchanged. Cadoni and Angelucci’s [2013 FAO MAFAP rice analysis](https://www.fao.org/fileadmin/templates/mafap/documents/technical_notes/NIGERIA/NIGERIA_Technical_Note_RICE_EN_Jul2013.pdf) uses a 0.67 milled/paddy ratio. We adopt that ratio as a fixed approximation for harmonizing this atlas, not as a measured recovery for every country or year. [IRRI describes variation in milling recovery](https://www.knowledgebank.irri.org/step-by-step-production/postharvest/milling/producing-good-quality-milled-rice/milling-yields), including typical modern multistage recovery of 65–70%. Conversion uncertainty is not included in the displayed yield levels.

We group source series by country, stable unit, season, and area basis. When both rice forms have eligible observations, we prefer the entire paddy series. When paddy has no eligible observations, we use the milled series. Selection uses the full available record, so changing the displayed period does not switch source forms. We never sum the forms or splice their years. The ten overlapping source series are in South Korea; the paddy choice omits one additional Busan year available only in the milled record. India supplies the retained milled series; all eight other countries supply paddy. Planted and harvested area bases, seasons, and missing years stay distinct.

Maps, means, raw-yield time series, observation tables, and CSVs use the same converted weights. Counts, reporting periods, areas, and crop-window ENSO exposures are unchanged. Regional notes identify the original source form and conversion. Observation CSVs include `source_crop`, `display_weight_basis`, `weight_conversion_factor`, `milled_per_paddy_ratio`, `original_yield_t_ha`, and `original_production_t`. Fitted-observation CSVs include the converted yield and intercept plus the original yield and conversion fields. Original country–crop JSON downloads retain the source units. The mathematical transformation is evaluated at load time; no source file or frozen release checksum changes. Its machine-readable definition and references are in `global/data/rice-conversion.json`.

Multiplying every yield in one series by a positive constant adds `log(weight_conversion_factor)` to the log-yield intercept and leaves its time slope, ENSO coefficients, coefficient covariance, residuals, fitting years, and validation scores unchanged. The combined view applies that intercept shift explicitly so fitted yield levels and trend-adjusted observations use consistent units. ENSO percentage responses and their intervals are exactly those of the retained source series. No new regression or bootstrap is needed for this constant conversion. Choosing paddy rather than milled in an overlapping Korean series can change the result because those are distinct source records; the two are never averaged.

Existing global links with `crop=rice-paddy` or `crop=rice-milled` open the combined rice view and are normalized to `crop=rice`, retaining the region, season, period, denominator, model, and scenario selections. The separate legacy ENSO explorer and original source data remain available in their original reporting forms. The original four-product catalog stays frozen; the application adds the combined rice label and union of source seasons in memory. Climate windows and fits are always looked up using the retained source product.

Run `node tests/global_rice.cjs` (also part of `npm test`) to check every converted observation, mean, and retained model, including intercept shifts, unchanged percentage responses/intervals, unchanged adjusted residuals, duplicate handling, missing versus zero yields, and source immutability. Interface tests cover unified country coverage, old-link migration, paddy versus converted milled notes, and converted/original values in downloads.

## Yield and ENSO through time

The regional chart overlays observed yield in green (left axis, t/ha) and crop-season Niño 3.4 in orange (right axis, °C). We use the frozen NOAA PSL ERSSTv6 monthly anomalies from the agricultural analysis, relative to 1981–2010. We average all months in the existing country–crop–season window and align the average to the crop reporting year. For example, Indian Rabi uses November of the preceding year through April; Whole Year retains the existing following-year assumption. Every window and its calendar caveats accompany the index export. Annual windows whose crop-specific timing is unresolved are identified in the source note.

We scale the right axis so the two series have equal vertical standard deviations and align their means, using years with both values over the full displayed record. Raw values retain their original units. Both axes expand to include every value and ENSO zero without clipping. A dotted line marks zero Niño 3.4. Constant or single-year records use an automatic scale, identified below the chart. Changing the selected period changes emphasis and downloaded rows while retaining the full record and its axis scale. Yield gaps interrupt the yield line; an ENSO value requires every month of its window, so missing climate months also create gaps. ENSO can remain visible in years with missing yields.

The observation table and CSV include the seasonal index. The CSV also identifies the window, index product, and anomaly baseline. The separate climate supplement is `global/data/enso/2026-09-26/nino34.json`; the frozen agricultural files are unchanged. The NOAA source snapshot is `provenance/nino34_20260924.txt`, and the window definitions come from `source/atlas_payload.json.gz`. Both source hashes are stored in the supplement. Run `python3 scripts/build_global_enso.py` to regenerate it or add `--check` to verify it. The exporter reproduces all 240,768 ENSO exposures in the existing fits. The JavaScript tests independently compare the 51,882 fitted exposures for the crop windows in this atlas and verify the scaling, units, missing values, and reporting-year offsets.

## ENSO response layer

Choose **ENSO response · % yield** to map the fitted change relative to neutral crop-season Niño 3.4, at a fixed time trend. The initial selection is +1°C and the linear model. Brown indicates lower yield, teal higher yield, and gray no available estimate under the selection. All regions use the same selected response form. Dashed boundaries identify exposures outside each fitted series’ observed ENSO range. The support selector can exclude those extrapolations.

The response forms fit `log(yield) = a + trend*(year−2000)/10 + g(N)`, with `g(N) = βN` (linear), `βN + δN²` (quadratic), or `βN + h max(N,0)` (separate warm/cold slopes). We display `100 × [exp(g(N)) − 1]`. This is a change in fitted geometric yield relative to neutral; it is not an arithmetic expected yield or a causal effect. No alternate model is substituted where the selected model fails. The index is the frozen ERSSTv6 Niño 3.4 series already used in the yield time series. Other index choices will require additional fitted supplements.

We refit the frozen global observations separately for every exact country, reporting unit, crop, season, area basis, and prepared period. We do not import coefficients from legacy series that joined area-basis definitions. Fits require at least 20 positive-yield observations with complete crop-window ENSO exposure, five warm and five cold exposures, and a full-rank design. Missing source rows and zero yields cannot enter the logarithmic model; the panel reports their omission and retains all eligible observed yields in the raw time series. The ten-year periods have no eligible ENSO fits and explain why. Season and basis defaults remain the same as for descriptive yields.

| Product | Available-record eligible series | 1991–2020 | 2001–2020 |
|---|---:|---:|---:|
| Wheat | 347 | 317 | 226 |
| Maize | 522 | 460 | 323 |
| Rice, paddy | 380 | 331 | 257 |
| Rice, milled | 610 | 572 | 376 |

These counts span seasons and area bases; a map selection uses only one of each per country. Across the three prepared periods, 4,721 eligible series-period combinations support all three forms, or 14,163 fitted designs. Series are overlapping agricultural outcomes, not independent regional experiments.

**Uncertainty.** We draw 400 pairs-bootstrap samples of non-overlapping three-calendar-year blocks anchored at 1980. Each requested period has a master calendar covering its observed years across the database. Each draw resamples that master set of blocks with replacement and applies the same block multiplicities to every unit in the period, retaining missingness and paired yield/index values within each unit. The resulting observation count can vary by draw. We refit all coefficients, including the time trend, and reject draws with fewer than 12 paired observations or a rank-deficient design. We export the sample covariance of the ENSO coefficients when at least 360 draws remain. For a selected exposure, the variance is `v' Cov v`, where `v = N`, `(N,N²)`, or `(N,max(N,0))`. We form a normal interval in log response around the original point estimate and transform its endpoints to percent. The shaded curve band consists of pointwise intervals, not a simultaneous confidence band.

These intervals account for sampling variability under the selected block scheme and model. They do not cover crop-calendar uncertainty, alternate ENSO trajectories, model selection, or future-year yield variability. Three years is a fixed block-length assumption, not an estimated dependence length. The **95% interval excludes zero** filter uses unadjusted pointwise intervals, not field significance or a multiplicity-controlled discovery rate. Insufficient bootstrap draws leave the point estimate available but its interval unavailable. At neutral ENSO the response and its interval are exactly zero by construction.

**Validation.** We also hold out five-calendar-year blocks under two layouts anchored at 1980 and 1982, removing the adjacent year on each side from training. Each fold requires 12 training years, three warm and three cold exposures, and full rank for all three ENSO models and the trend-only baseline. We pool squared log-yield prediction errors by held-out observation. Each layout needs three eligible folds and 15 predictions. Skill is `1 − MSE_model/MSE_trend`; the **Positive holdout skill** filter requires positive skill in both layouts. This is distinct from the uncertainty interval and does not establish skill for a model chosen after inspecting results. The global layer does not automatically choose a winning model or require a nonlinear model to beat the linear model.

**Event peak scenario.** The +3°C button uses the frozen 1996–1999 monthly trajectory around the 1997–98 El Niño. Its original peak is +1.85°C in November 1997 in this index/baseline. We multiply the whole profile by `selected peak / 1.85` and shift its peak to the selected year and month, then average the aligned months in each crop window for the selected reporting year (peak year or following year). A +3°C peak is therefore usually a smaller crop-season average. The regional panel shows the monthly trajectory and shaded crop window. This historical shape scaled in amplitude is an illustrative scenario, not an ENSO forecast, and its resulting crop-season exposure is checked against the observed exposure range. A direct crop-season mean remains a separate choice.

Each regional ENSO panel shows one selected model and the actual fitted observations against Niño 3.4. For visibility, the scatter removes the jointly fitted linear time trend and expresses yield relative to the fitted neutral level; its vertical axis is linear in log response with percent tick labels. The model band and scenario marker use the same transformation. Raw observed yields and the ENSO index remain in the separate time-series chart below. The selected-fit CSV contains the retained years, raw yields, crop-season index, adjusted values, coefficients, and scenario interval. The map export contains all selected source units, including unavailable or filtered ones with reasons; `mapped` identifies the colored subset.

The fitted supplement lives under `global/data/fits/2026-09-26/`. Four crop JSON files map `(country|unit|season|basis)` to each prepared period’s fit or explicit failure reason. `methods.json` records definitions, source hashes, bootstrap design, and the scenario profile. Coefficients and covariance use full floating-point precision. These files leave the frozen agricultural release and legacy explorer unchanged. Crop fits are loaded only in ENSO map mode; individual observed series are still loaded only when a region opens.

Install `scripts/global-requirements.txt`, then run `python scripts/build_global_fits.py` to rebuild from the committed agricultural and climate snapshots. Run `python tests/global_fits.py` for numerical/source checks and `npm test` for response calculations, scenario/calendar alignment, filters, plotting, downloads, missing data, and navigation. The numerical checks compare all fitted coefficients with the exact retained records and compare the weighted bootstrap solve against explicitly duplicated observations. The baseline/source hashes and period-specific retained years are independently checked.

## Navigation

Regional observations open in a side panel over the world or country map; on smaller screens the panel sits along the bottom. The map remains clickable. Choose another reporting unit on the map, from the panel selector, or with Previous/Next. The panel retains the observations, chart, source notes, and downloads. Closing it or pressing Escape restores focus to the region in the location list when available.

Opening a panel adds one browser-history entry. Switching regions or changing its selections replaces that entry, so one browser Back returns to the originating map and its selections; Forward reopens the last panel. Zoom, map position, page scroll, search text, and list scroll are retained within the session. Back to map on a directly opened regional link stays in the atlas, using its specified backdrop. Shareable URLs preserve the region and selectors; camera position and scroll remain local to the session.

## Data interface

Files under `global/data/2026-09-26/` form a frozen release. `catalog.json` lists products, countries, source seasons, reporting levels, prepared periods, source-file checksums, and exclusion counts. `<crop>.json` contains a summary per region, crop, season, and area basis, with mean, count, actual dates, and completeness for each period. `geometry/<country>.json` contains stable reporting boundaries and their lineage metadata. `observations/<country>-<crop>.json` contains source identifiers, excluded-record counts, and observations in the column order recorded inside each file. `manifest.json` hashes every release JSON file.

The compound series key is `(country, region_id, crop, season, yield_basis)`. Years must be unique within a series. Region IDs are stable source identifiers and must have a corresponding geometry. Missing means are JSON null. An observation array contains reporting year, yield in tonnes per hectare, area in hectares, production in tonnes, and a Boolean correction flag. The displayed region names do not define identifiers. Parent/child navigation is currently world → country → stable source unit; these units are the finest statistical geography in the current packages.

We simplify statistical boundaries at 700 m in a local equal-area projection. The original grouping remains the analytical geography. The one unmatched Indian boundary without a statistical unit is left uncolored. Natural Earth 1:110m country outlines supply global context; very small territories may be absent at that scale. National coverage supplied by the expanding database will require a national reporting unit with source observations and its own geometry. Mixed national and subnational inputs must define non-overlapping coverage before inclusion.

The application uses hash routes, such as `#/country/JP?crop=wheat&period=available&metric=yield&minimum=1&basis=default&season=default&release=2026-09-26`. Regional routes add the stable unit ID and `view=world` or `view=country` to identify the underlying map. Links without `view` retain a country backdrop. The release is part of the link. Unknown release IDs produce an explicit error. We load crop summaries and the relevant geometry for the world view, and load observations only when a regional panel opens. No API, credentials, external tiles, or runtime third-party scripts are required.

## Reproduction and checks

The frozen application runs on a static HTTP server or GitHub Pages. Run `npm ci` then `npm test` with Node.js 18 or later to validate all period summaries, release hashes, route round trips, selection uniqueness, and country/region interfaces. The interface tests use a DOM implementation and include the accessible no-WebGL fallback. Navigation checks additionally exercise browser history, camera and scroll preservation, direct links, regional switching, and keyboard close/focus behavior with a camera-aware map stub. Browser checks additionally verify the actual interactive map and layout.

To regenerate the release, install Python dependencies from `scripts/global-requirements.txt`. Obtain the original agricultural packages and Natural Earth’s `ne_110m_admin_0_countries.geojson`, then run:

```sh
python scripts/build_global_data.py --inputs /path/to/4_ag --world /path/to/ne_110m_admin_0_countries.geojson
python tests/global_rules.py
npm test
```

The original source packages are unchanged. Input hashes in the catalog identify the agricultural files used. The world outline source is https://raw.githubusercontent.com/nvkelso/natural-earth-vector/master/geojson/ne_110m_admin_0_countries.geojson; the exported geometry is frozen and hashed in this release. The exporter is intentionally an adapter for the current packages. The larger database can populate the same release contract without changing the navigation.

MapLibre GL JS 5.20.0 and its stylesheet are vendored from the npm package. Its BSD-3-Clause license is in `global/vendor/maplibre-LICENSE.txt`. Natural Earth outlines are public domain. Statistical boundaries and agricultural observations retain the source terms described in the repository README and provenance notes. No new blanket data license is asserted.