Why any of this exists · a question about farms, not about satellites
Small-scale gold mining spread across Ghanaian farmland. Does it lift rural incomes — cash, local demand — or wreck the farms — poisoned water, lost land, labour pulled away?
It is an empirical question, and answering it needs something nobody has: for every survey village, in every year, how much of the surrounding land was being mined. Ghana’s household survey records the farms. It records nothing about the mines.
What the economics needs
mine exposure(village, year) · 1,143 villages · 2007…2017
What exists off the shelfnothing at this resolution · so we build it
Two repositories, one chainmine-maps measures the mining → mines-gsps estimates the effect
Step 0 · what ships in git & what you stage
git repository
pipeline code · configs · src/Makefile · launchers + small reference data: hand verdicts CSV, Barenblitt polygons, GADM boundaries
clone · ships ✓
Drive catalog
satellite imagery: raw Sentinel-2 years and GeoMAD annual composites ~200 GB · the human-facing source of truth
stage_data.py · rclone · 76 GB in 6 min
Quest /projects/p32315
proc tiles · training labels · urban masks · EA coordinates · trained weights shared group-readable working copy
read in place · no copy
GCS bucket
durable outputs: probability rasters · detections · panels …/repro_20260710/
durable team store
The repo holds the code, Drive holds the imagery, the shared cluster folder holds labels and models, GCS holds durable outputs. Every step that follows reads from and writes to these four homes.
Stage 1 · Sentinel-2 · 10 m · launched 2015 · finds the mines
sourceCOPERNICUS/S2_SR_HARMONIZED
annual median
Stage 2 · Landsat GeoMAD · 30 m · back to 1984 · dates the mines
grid EPSG:6933 · 30 m · uint16
bands blue green red nir swir1 swir2
years 2007 … 2017 (+2024)
output in/raw/DEA_GeoMAD/Ghana/{year}/
Drive catalog · in/raw/2024-S2/staged in 6 min
35 files76 GBGhana · 2024 · 10-band · 10 m
Drive catalog · in/raw/DEA_GeoMAD/Ghana/~11 GB per year
543 files / year11 yearszero failures on restage
Step 2 · one deterministic 0.5° WGS-84 grid for everything
117 half-degree tiles cover Ghana (≈55 km squares). Rebuilt from raw: 117-task array, ≤8 min per tile, 20 at a time → ~50 min wall-clock. 02_preproc.py:43
12 channels on disk+ NBR computed on the fly= 13-band detector input
Step 2b · the frozen percentile spine — GeoMAD 2007…2017 re-normed on 2017
in/proc/{year}-GeoMAD/chunk_X_Y.tif · 8-band float3221 in-region tiles × 10 years + full 117-tile 2017 reference
Every historical year is stretched on the 2017 curve, so what the dating model sees in 2009 is on the same scale as what it learned on in 2017. Early-year (2007–2011) mosaics carry a known artifact — the levels are a signal under test.
~3 min per year6 SR bands + NDVI + MNDWI
Step 3 · the detector's training deck — labelled in Nigeria, applied to Ghana
window 128 × 128 px · stride 64 (50% overlap)
skip >10% nodata in any band
route mask.sum()>0 → POS · urban>25% → URB · else REST
train_pos0.50
mask.sum() > 0
train_neg_urban0.15
bare urban ground · the classic false alarm
train_neg_rest0.35
everything else
TFRecords · 87 shards from the 30-tile production set696 s
image float32 [−1,1] 128×128×Cmask uint8 128×128×1mixed at train time via sample_from_datasets
Ghana labelling roundin progress · not wired in
Six coders tracing galamsey polygons in Google Earth Pro over 311 EA circles, landing as shapefiles in Drive. This would let the detector train on Ghana instead of transferring from Nigeria.
14 of 311 EAs complete · audited 2026-08-20 · no in/Ghana/labels/ in the repo yet
Every Ghana number in this pipeline is zero-shot transfer. The training set contains zero Ghana pixels — the detector learned in Nigeria and is applied across a border it has never seen. ml-roadmap item 8
Step 4b supervision · a different ground truth — Barenblitt candidates, hand-ruled
loss Σ[ V · w · BCE(logit) ] / ΣV · pos_weight 10 on mine pixelsbias σ output bias = log(0.05/0.95) = −2.94 · the rare-class priorsplit 40 km spatial blocks held out — no passing by memorizing a neighborhood
val patch AUC 0.8264recorded 0.8199applied to Ghana → 6,458 detections
Step 4b · the dater — a scene classifier that sees only a 400 m window
back-caster · 400 m GeoMAD scene → verdict04b_train_backcaster.py · 2017 composite
why 400 m a fixed window, so a mine's size can't give the answer awayCV spatial-block hold-out · seeded CPU training is deterministic
val AUC0.9433 · site-weighted 0.8466 · F1 0.9021
trains in ~4 minutesidentical to 4 decimals on two independent runsout/backcasting/runs/verdict_*/best.keras
Step 5 · sweep Ghana, average the votes, trace the outlines
chunk_2_3_probability.tif · float32 0–1
05_inference_seg.py · best.keras
windows 128 px · stride 64 · batch 512
merge overlapping predictions averaged
cost 88 min for all 117 tiles on one CPU node
06_postproc_seg.py · vectorize~33 min
threshold≥ 0.95 (F1-optimal, prod)
morphologyclose 3×3 + fill holes
components8-connectivity
polygonizerasterio.shapes
merge nearbybuffer ±20 m
area filter≥ 10 000 m² (1 ha)
detections.shp · Ghana 2024 production product7,283 polygons
+ per-chunk KML · merged KML · KMZre-vectorizes byte-identically from the archived rasters
The archived rasters, not the raw imagery, are the product's source of record. Regenerating from freshly rebuilt tiles gives 15,486 detections — the original tiles were normalized on a Ghana-specific curve, the current recipe uses the Nigeria curve.
Step 6 · the two stages meet — every detection, every year, one verdict
chop · BC_CHOP=106_inference_backcaster.py
32,496 pieces from 7,283 detections400 m · min 625 m²→ pieces.gpkg
score · each piece × each year · pred_prob13 min · all 11 years
predictions_polygon_year.csvpiece_id, lon, lat, area_m2, year, pred_prob, kept
Each year's composite is re-normalized with the shared 2017 percentiles so the input distribution matches training. Replicated series correlates 0.980 with the original product; the validated quantity is the held-out 2017 TRUE−FALSE gap (≈ +0.84), the year-by-year levels are under audit.
Step 7 · from satellite output to an economics variable
GSPS enumeration areas · Western Region · EPSG:32630
07_postproc_backcaster.py
share_mined[ea, year] = area( polys with pred_prob ≥ τ in year ∩ buffer[ea] ) / area( buffer[ea] )
buffer 10 km · τ = 0.6 · EAs with no mine ⇒ share 0, kept (the comparison group)
ea_buffer_year_share_long.csv / .dta1,143 EAs × 11 years
+ _wide.csv+ _summary.jsonruns in seconds — the cheapest place to experiment
ea_buffer_emergence.png · mean share vs yearlevels under audit
consumed byGSPS difference-in-differences · the treatment series 1,143 villages in every variant — exactly the GSPS schema
Step 8 · the seam — where one repository hands off to the other
mine-maps · measures the mining
mines-gsps · estimates the effect
household buffer shares · ghana_buffer_share_*.csvDrive → make exposure
schema id_XY, lat, lon, share_mined_p090, share_mined_p095, share_mined_p099, share_mined_p0998rows 15,636–15,688 — one per unique GSPS household coordinatefetch src/build/fetch_exposure.sh · rclone from gdrive-gprl:mlgis-macrodev/4-data processing/gsps-exposure/ghana
4 detectors × 3 buffer radii × 2 vintages = 24 files, all present on disk
the analysis picks exactly one: AES2 · 5 km · p099the other 23 are robustness
ea_buffer_year_share_wide.csvmerged · unused
cols id_XY, lat, lon, share_2007 … share_2017rows 1,143 Western-region EA clusters
The step-7 back-cast panel is merged in (as share_mined_bc_10km_2007…2017) — and then never referenced by the estimation. It is scaffolding for a staggered design that has not been built. Today's treatment is a single 2019→2024 change.
The two repos meet at a coordinate join, not a shared database: exposure is matched to households on latitude/longitude rounded to 4 decimal places, because the panel's own id_XY and the exposure file's id_XY disagree (only 920 of 1,143 agree). Everything downstream inherits that join.
Step 9 · the household side — four survey waves, one roster, one rule
the master roster · 01-1_hh_wave_roster.doFPrimary × wave
rows 21,378 household-wave observationshouseholds 7,229 unique FPrimarythe rule every later step left-merges, then assert _N == $N_ROSTER
Membership is decided once, up front. No merge is allowed to add or drop a row — a ratchet that makes it impossible to quietly lose households while building variables.
01_build_gsps_master.do · eight helpers, in order→ ghana_gsps_proc.dta
Stata · ~19 MB outputevery step ends in the same assertion
how a harvest becomes a number01-3_crops.do
reported harvest“12 bowls of maize” — local units, per plot
unit → kgpooled lookup from Conversion_Units.xlsx
kg → valuefixed 2010 Ghana FAO producer prices, USD/tonne
÷ areavalue per hectare of the maize+rice+cassava basket
logln_mrc_value_per_ha — the outcome the model estimates
prices maize 340.8 · rice 488.3 · cassava 168.8 USD/tonne (2010)why fixed so a change in the number means a change in output, not in prices
freeze lat/lon fixed at first-observed waveflag haversine drift → mover_{1,5,10}km
Exposure must be a property of a place, not of a household's choice to move. Freezing coordinates makes exposure time-invariant per household — and stage 2 asserts it: any within-household variation in the share columns halts the build.
Step 10 · who counts as treated, and who is left to compare them with
share_mined_p099 · AES2 · 5 km · 2024rural non-movers
p75
0 (no mining detected)max 0.0768
median 0 — more than half the sample has no detected mining at allp75 0.0000141 — the cut-off is effectively “any detection at all”mean 0.00266 in 2024 vs 0.00048 in 2019 — roughly 5× in five yearsskew 5.04
defining D · analysis_did.doa single binary, wave-invariant
gen byteD = share_mined_p099_aes2_5km_24 > p75
drop if share_mined_p099_aes2_5km_19 > p75
Read it in words: treated = became mined between 2019 and 2024. Households already in the top quartile in 2019 are thrown out entirely, so the contrast is newly-arrived mining against none — not more mining against less.
The same drop also removes rows with missing 2019 exposure, because in Stata missing compares as larger than any number. The code comments on this; it is a real, deliberate side effect.
from roster to regression · every restriction, in orderanalysis_did.log · 2026-07-26
outcome ln_mrc_value_per_ha — log value/ha, maize+rice+cassavaabsorb household (FPrimary) + yearcluster FPrimary — 1,232 clustersbase year2019, the last wave before the mining surge
Household fixed effects mean each household is compared to its own past; year effects absorb everything that hit Ghana as a whole. What is left in βt is the divergence between newly-mined and never-mined households.
Stata 18 · reghdfe + ftoolsrun 2026-07-26
βt with 95% confidence intervalsbase 2019 = 0
N = 3,222F(3, 1231) = 0.78Prob > F = 0.5074within R² 0.0021
What this does and does not show
Nothing here is statistically significant. The joint test on all three interaction terms gives F = 0.78, p = 0.51 — the data cannot reject “mining changed nothing”.
The point estimates all lean negative, and the post-period one is large: β2023 = −0.388, about a 33% fall in value per hectare. But its confidence interval runs from −0.89 to +0.11, so it is equally consistent with a large loss and a small gain.
The pre-period coefficients are not tightly zero either (−0.13 and −0.13, both with standard errors around 0.17), so the parallel-trends evidence is weak rather than reassuring. The honest summary: underpowered. 3,222 observations and 1,232 clusters, with treatment defined by a threshold that amounts to “any detection at all”. The code says so itself — the plot note reads “SEs clustered at the HH level (preliminary)”.
What would sharpen it — already scaffolded, not yet built
· the staggered design: mine onset year per household from the step-6 back-cast
(columns merged, never referenced) → a proper event time, not one 2019→2024 jump
· non-farm outcomes: the NFE enterprise panel is merged and never estimated
· Ghana labels → a detector trained in-country, so exposure stops being zero-shot
· reproducibility: no pyproject / requirements / lockfile for the Python stage
· the 1 km grid products (population, nightlights, built-up) need a
grid-cell → household buffering helper that does not exist yet