Skip to content

Usage

Command-Line Interface

The CLI provides restart-safe entry points for downloading NEON flightlines, running the full pipeline, generating QA outputs, and repairing missing intermediate products.

Legacy CLI aliases still forward to the same implementation, but the primary command names are spectralbridge-*.

Entry points

Available commands

spectralbridge-download

Download NEON HDF5 flightlines into a workspace.

spectralbridge-pipeline

Run the full NEON processing pipeline end to end.

spectralbridge-qa

Re-render QA PNG/JSON outputs for processed flightlines.

spectralbridge-recover-raw

Backfill raw ENVI exports when corrected outputs already exist.

spectralbridge-qa-summary

Build a multi-page PDF summary of recursive drone QA PNG outputs.

spectralbridge-merge-duckdb

Merge per-product parquet outputs into a flightline-level master table.

spectralbridge-bulk

Catalog completed flightlines, query them virtually, and run population-aware synthetic translation analyses.

spectralbridge-drone-production

Inventory a remote ExportPackage collection, process independent drone flights, or run the complete strict campaign-to-bulk closeout workflow.

spectralbridge-drone-production

spectralbridge-drone-production bulk i:/iplant/home/shared/.../summer-2023-10cm-10k \
  --years 2023 2024 \
  --work-dir /home/jovyan/data-store/SpectralBridge_Drone_2023_2024 \
  --upload-results

The inventory subcommand performs no H5 transfer. For a year-specific source name containing one standalone year token, every requested year is resolved as a sibling collection; a higher campaign root is scanned directly. The inventory and campaign reports retain per-year source, manifest, discovery, completion, and reuse evidence, and strict mode cannot silently accept a missing requested year. campaign stops after validated canonical flight outputs. bulk adds exact combined-population preflight, compact analysis, normal bulk figures/report, checksummed packaging, and optional verified upload. Authentication uses the caller's normal non-interactive gocmd configuration. See remote drone production.

spectralbridge-bulk

Runs independent cross-run analysis against completed or minimally staged flightline products. Automatic mode recursively finds scientifically identifiable flightline directories beneath arbitrary storage folders and reuses the target-sensor ENVI products required by the selected profile. Generic units use spectralbridge_flightline.json; canonical NEON names remain supported. If no flightline directories are found, automatic mode falls back to prebuilt merged Parquets.

spectralbridge-bulk /data/spectralbridge_completed_runs \
  --output-dir /data/spectralbridge_bulk_analysis \
  --input-mode auto \
  --analysis translation \
  --translation-pair MicaSense_to-match_OLI_and_OLI-2__to__Landsat_8_OLI \
  --threads <N> \
  --memory-limit <XGB> \
  --temp-directory /scratch/spectralbridge_bulk

The source tree is read only, the output directory must be fresh and external, and streaming defaults to one flightline at a time with bounded aligned raster windows. Normal analysis writes compact sufficient statistics rather than pixel caches. --diagnostic-sample-size enables an optional globally bounded deterministic sample. --sensor and --translation-pair are repeatable selectors. Invalid flightlines are recorded and excluded by default; use --on-invalid error for strict behavior. --materialize-observations is retained only as an explicit legacy dataset-build option. Use --preflight-only to inspect products, selected bytes, compact-output estimates, compatible pairs, and deterministic exclusions without reading raster pixels. --input-kind applies only to merged-Parquet compatibility mode. See Build a bulk cross-run analysis.

Existing polygon spectral library

Pass an existing merged polygon Parquet separately when spectral-library products are required. Start with --preflight-only to inspect its schema, counts, source size, largest groups, expected pages, and estimated scans without writing PDFs. A non-preflight run with both plotting flags omitted creates only compact summaries and outlier diagnostics. Summary plots are explicit; the complete multipage suite requires a second flag. The default renders every visualization-valid trace into bounded raster layers. --spectral-max-traces-per-group is the only sampling control and is never enabled implicitly.

spectralbridge-bulk /data/completed_products \
  --output-dir /data/bulk_analysis \
  --spectral-library /data/library/polygons_merged_pixel_extraction.parquet \
  --spectral-stage corr \
  --make-summary-plots \
  --make-full-spectral-reports \
  --spectral-panels-per-page 4 \
  --spectral-trace-batch-size 2000 \
  --spectral-y-scale global_robust \
  --spectral-plot-y-quantiles 0.005 0.995

The schema adapter recognizes physical-wavelength columns such as corr_b001_wl0450nm, sorts them numerically, and rejects ambiguous stages or duplicate wavelengths. Use --species-field and --spectral-stage only when automatic selection cannot be unambiguous. Species can be ordered with --species-sort count_desc or alphabetical. The default global_robust y-scale preserves comparison between panels while clipping only the display at the configured quantiles; the full suite also writes a separate full-range audit PDF. global_full uses the actual common range, and per_group_robust exposes more within-species structure. Finite negative values remain plot-valid by default. Use repeatable --spectral-nodata-value for known sentinels and --spectral-plot-minimum-reflectance only for an explicit visualization threshold; neither changes regression validity.

spectralbridge-download

Downloads NEON HDF5 flightlines into a workspace directory using the package download helpers.

NEON's download endpoint requires an API token. Export NEON_API_TOKEN (or NEON_TOKEN) in the shell before running either the download command or the full pipeline. Keep the token out of notebooks, configuration files, prompts, and version control.

export NEON_API_TOKEN="<your-token>"
spectralbridge-download SOAP \
  --year-month 2021-06 \
  --flight NEON_D17_SOAP_DP1_L057-1_20210615_directional_reflectance \
  --output data
  • Required: positional site, --year-month, and one or more --flight values
  • Optional: --product defaults to DP1.30006.001
  • Optional: --output defaults to data, with downloads written under <output>/<site>/

spectralbridge-pipeline

Runs the main NEON pipeline and writes all canonical outputs for each flightline.

spectralbridge-pipeline \
  --base-folder output_demo \
  --site-code NIWO \
  --year-month 2023-08 \
  --product-code DP1.30006.001 \
  --flight-lines NEON_D13_NIWO_DP1_L020-1_20230815_directional_reflectance \
  --engine thread \
  --max-workers 2

Required options

--base-folder, --site-code, --year-month, --product-code, and --flight-lines.

Parallel defaults

The pipeline defaults to --engine ray and --max-workers 8.

Resampling controls

--resample-method accepts convolution, legacy, or resample.

Merge controls

Use --merge-memory-limit, --merge-threads, --merge-row-group-size, and --merge-temp-directory to tune DuckDB merging.

Outputs include raw ENVI exports when available, corrected ENVI, sensor-resampled products, per-product parquet sidecars, _merged_pixel_extraction.parquet, and QA files. See Outputs & File Structure for the full contract.

spectralbridge-qa

Recomputes QA PNG and JSON outputs for existing flightline folders.

spectralbridge-qa --base-folder output_demo --quick
  • --base-folder is required
  • --quick uses the deterministic 25k-pixel path; --full disables that subsampling shortcut
  • --n-sample defaults to 100000
  • --rgb-bands accepts values such as 660,560,490 or symbolic R,G,B
  • --save-json / --no-save-json controls whether _qa.json is written alongside the PNG
  • --out-dir copies the generated QA files elsewhere after rendering

spectralbridge-recover-raw

Scans a base folder for HDF5 plus corrected ENVI outputs and backfills missing raw ENVI exports so restart-safe downstream stages can continue cleanly.

spectralbridge-recover-raw \
  --base-folder output_demo \
  --brightness-offset 0.0

spectralbridge-qa-summary

Builds a multi-page PDF summary by recursively finding drone QA PNG files.

spectralbridge-qa-summary drone_outputs --output-pdf drone_outputs/qa_summary.pdf
  • Required: positional base_dir
  • Optional: --output-pdf to override the default output path
  • Optional: --pattern defaults to *__qa.png

spectralbridge-merge-duckdb

Merges per-product parquet tables into a master parquet and can optionally render QA alongside the merge output.

spectralbridge-merge-duckdb --help

Use this command when you need direct control over parquet merging outside the main pipeline orchestration.