Skip to content

Pipeline

Outputs & File Structure

Outputs on disk are the primary interface of SpectralBridge. Downstream analysis should rely on these files rather than return values from the Python API.

Canonical outputs

Per-flightline contract

Naming stems come from spectralbridge.paths.FlightlinePaths and spectralbridge.utils.naming.get_flightline_products. Sensor-specific stems come from SensorProductPaths.

Output type Canonical filename pattern Description Notes / guarantees
Raw ENVI (when available) <flight_id>_envi.(img|hdr|parquet) Direct export of the NEON HDF5 reflectance cube. Used when present to seed later stages; parquet sidecar is written when exported.
BRDF model JSON <flight_id>_brdf_model.json Scene-level BRDF coefficient tables written before BRDF application. Includes iso/vol/geo, kernel settings, ndvi_binning_enabled, and ndvi_edges.
BRDF + topographic corrected ENVI <flight_id>_brdfandtopo_corrected_envi.(img|hdr|json|parquet) Physics-informed normalization and correction JSON produced before sensor resampling. One corrected set per flightline; parquet mirrors the corrected ENVI cube.
Sensor-resampled ENVI + Parquet <flight_id>_<sensor>_envi.(img|hdr|parquet) Reflectance cubes resampled into the configured target sensor frame. Sensor stems match FlightlinePaths.sensor_products.
Merged Parquet <flight_id>_merged_pixel_extraction.parquet Master table that merges Parquet sidecars across stages into one analysis-ready spectral library. Exactly one per flightline and treated as the primary success signal.
QA artefacts <flight_id>_qa.png, <flight_id>_qa.json, optional <flight_id>_qa.pdf Visual and numeric QA summaries aligned to the merged outputs. PNG and JSON are expected for completed runs; PDF is optional.
QA metrics parquet <flight_id>_qa_metrics.parquet Structured QA metrics by band and sensor. Emitted alongside QA outputs when QA calculation runs.
Synthetic sensor regression diagnostic qa_plots/<merged_stem>__MS_vs_Landsat_FIXED.(png|json) Scatter panels compare wavelength-matched synthetic MicaSense and Landsat products; the JSON records the displayed slope, intercept, correlation, R², and sample count. Both axes come from the same corrected NEON source. This is a descriptive convolution diagnostic, not empirical sensor calibration.
Stage QA qa/stages/<order>_<stage>/stage_qa.(json|html) plus optional overview.png Focused report for one canonical stage with explicit checks and provenance. Deterministic and restart-safe; missing diagnostics are recorded as NOT EVALUATED.
Combined stage QA qa/combined/combined_qa.(json|html|pdf) plus pipeline_evolution.png Cross-stage status, pipeline evolution, evidence-backed synthesis, and a printable multi-page summary. Does more than concatenate stage reports; unsupported translation/Landsat diagnostics remain explicit. The PDF is intended for download and flightline-to-flightline comparison.

Success criteria

What a successful run looks like

  • The merged parquet exists and is readable: <flight_id>_merged_pixel_extraction.parquet.
  • The QA PNG renders with its matching JSON: <flight_id>_qa.png and <flight_id>_qa.json.
  • Sensor-specific ENVI and parquet products exist as configured; absence can reflect configuration rather than failure.
  • If the merged parquet and QA outputs are present, the flightline completed successfully.

Restart safety

Idempotence and validation

process_one_flightline and go_forth_and_multiply skip stages whose outputs already exist and validate, so re-running the pipeline does not recompute completed products unless outputs are missing or invalid.

Stage QA uses the same principle: reports are reused when the schema, parameters, thresholds, software version, and input/output artifact fingerprints match.

Drone polygon workflows also attempt to write CSV sidecars next to parquet outputs for portability. The parquet files remain the authoritative outputs.

How to use these files

Load parquet first

Use merged parquet outputs directly for most analysis tasks.

Inspect QA before modeling

Review QA PNG and JSON outputs to confirm spectral health and calibration quality.

Treat ENVI as diagnostic

Intermediate ENVI products remain useful for inspection, but many workflows only need parquet and QA outputs.