Skip to content

Tutorial

Cloud and HPC workflows

SpectralBridge is designed for large flight lines and restart-safe reruns, which makes it a good fit for scratch-based cloud, JupyterHub, and cluster environments where compute and storage policies matter as much as the code itself.

Stage data in

Bring only the HDF5 inputs you need into fast local or scratch storage for the current job.

Process locally

Write ENVI, parquet, merge, and QA artefacts into the working directory where chunked stages can spill safely.

Archive outputs

Move final products back to persistent storage once the restart-safe run is complete.

Storage pattern

  1. Stage a small batch of NEON HDF5 inputs into scratch or fast local storage.
  2. Run the pipeline there so correction, parquet extraction, merge, and QA all share the same fast workspace.
  3. Copy the completed outputs you care about back to object storage or shared research storage.
  4. Clean intermediate scratch only after the outputs are validated.

This pattern keeps failures isolated and works well with the package’s skip-aware rerun behavior.

Engine choice

When to use Ray, thread, or process execution

ray

Best for larger multi-flightline or managed compute environments. This remains the default execution backend for the NEON CLI.

thread

Best for first-pass debugging, lightweight runs, or situations where you want to avoid Ray initialization entirely.

process

Useful when you want local multi-process execution without the Ray runtime.

spectralbridge-pipeline ... --engine ray --max-workers 8
spectralbridge-pipeline ... --engine thread --max-workers 1
spectralbridge-pipeline ... --engine process --max-workers 2

Memory and temp space

How to reduce pressure safely

  • lower --max-workers before changing scientific settings
  • lower --parquet-chunk-size when extraction or polygon filtering is memory-bound
  • set --merge-temp-directory to local scratch for large parquet merges
  • avoid keeping many scenes or notebooks open against the same working directory
  • treat reruns as normal; the pipeline is built to skip validated outputs

Batch pattern

One job per flight line

This is the safest default because failed jobs stay isolated and completed jobs can be rerun without recomputing validated outputs.

Shared post-processing

Use downstream DuckDB, pandas, or QA summary tools once the per-flightline artefacts are already on disk.

Example

Minimal SLURM job script

#!/bin/bash
#SBATCH --job-name=spectralbridge
#SBATCH --mem=64G
#SBATCH --cpus-per-task=8

module load python

BASE=$SCRATCH/spectralbridge_${SLURM_JOB_ID}
mkdir -p "$BASE"

spectralbridge-pipeline \
  --base-folder "$BASE" \
  --site-code NIWO \
  --year-month 2023-08 \
  --product-code DP1.30006.001 \
  --flight-lines NEON_D13_NIWO_DP1_L020-1_20230815_directional_reflectance \
  --engine ray \
  --max-workers 8

Adapt the worker count and memory request to the size of the scene and the storage available on the cluster.

Where to go next

Pipeline stages See which stages dominate runtime and disk usage. Working with parquet outputs Use the authoritative tabular products downstream. Troubleshooting Recover safely from partial jobs or cluster interruptions.