Pipeline
Pipeline Overview & Stages¶
SpectralBridge transforms NEON HDF5 directional reflectance into corrected, sensor-harmonized, and analysis-ready outputs through restart-safe, file-based stages.
The orchestrators process_one_flightline and go_forth_and_multiply validate outputs before skipping them, which makes reruns restart-safe and auditable.
Execution order
Stage order and idempotence¶
- Download HDF5 via
stage_download_h5. - Export the raw ENVI cube.
- Build correction JSON.
- Apply BRDF and topographic correction.
- Resample all configured sensors.
- Export Parquet sidecars.
- Merge to
_merged_pixel_extraction.parquet. - Render QA PNG and JSON outputs.
Each stage checks whether expected outputs exist and validate before deciding to skip or recompute them. Recovery mode exists for raw ENVI exports when corrected products are already present.
1. Data acquisition¶
Inputs
NEON API paths or local HDF5 files.
Outputs
Cached HDF5 tiles stored under the selected --base-folder.
Downloads are handled by stage_download_h5 and triggered automatically by go_forth_and_multiply.
2. HDF5 to ENVI export¶
Inputs
*_directional_reflectance.h5 plus per-pixel geometry and metadata.
Outputs
*_envi.img/.hdr and the matching parquet sidecar when exported.
stage_export_envi_from_h5 writes the raw ENVI pair using canonical naming in FlightlinePaths, with optional brightness offsets.
3. Topographic and BRDF correction¶
Inputs
Directional or raw ENVI exports, DEM-derived slope and aspect, and solar/view geometry.
Outputs
<flight_id>_brdfandtopo_corrected_envi.(img|hdr|json).
stage_build_and_write_correction_json writes the parameter JSON, and stage_apply_brdf_and_topo applies the combined correction before downstream resampling.
The current NEON path writes the corrected cube in fixed non-overlapping spatial chunks. For drone workflows, correction only runs when required ancillary geometry is available for that flight.
4. Sensor harmonization¶
Inputs
BRDF-and-topo-corrected ENVI plus sensor spectral response functions.
Outputs
<flight_id>_<sensor>_envi.(img|hdr|parquet).
stage_convolve_all_sensors delegates to the configured resample method and iterates through the sensor products registered in FlightlinePaths.
5. Parquet extraction and merging¶
Inputs
Any ENVI cube produced by earlier stages.
Outputs
Per-product parquet files plus <flight_id>_merged_pixel_extraction.parquet.
_export_parquet_stage builds the per-product sidecars, and merge_flightline uses DuckDB to consolidate them with schema validation before optional QA rendering.
6. Quality assurance¶
Inputs
Merged parquet plus supporting ENVI files.
Outputs
<flight_id>_qa.png, <flight_id>_qa.json, and optional PDF output.
QA artefacts come from render_flightline_panel and follow the same canonical stems described in Outputs & File Structure.
Next steps