Usage
Working with parquet outputs¶
Parquet sidecars and merged parquet tables are the main analysis-ready outputs for tabular workflows in SpectralBridge.
Authoritative tables
They are the intended interface for DuckDB, pandas, and other columnar tools.
Restart-safe exports
The pipeline validates and reuses good parquet outputs instead of recomputing them blindly.
Large-scene friendly
They support filtering and aggregation without loading a whole raster cube into memory.
Why parquet
Why the pipeline writes columnar outputs¶
ENVI remains the raster authority for image-style access, but parquet is the practical entry point for most analysis, validation, and merge workflows. It compresses well, reads efficiently by column, and works cleanly with DuckDB and pandas.
That makes parquet especially useful when you want to summarize reflectance, inspect metadata, or join outputs across stages without materializing the full scene in Python memory.
File contract
What files you should expect¶
Typical per-product sidecars and merged outputs include names such as:
*_envi.parquet
*_brdfandtopo_corrected_envi.parquet
*_landsat_oli_envi.parquet
*_merged_pixel_extraction.parquet
Per-product parquet sidecars
These sit beside the corresponding ENVI products and store one row per pixel-band observation for that product.
Merged parquet
This combines raw, corrected, and sensor-resampled products into the per-flightline table named <flight_id>_merged_pixel_extraction.parquet.
Common columns include:
flightline_idrow,col,x,ybandwavelength_nmfwhm_nmreflectance
DuckDB
Inspect outputs without loading everything¶
DuckDB is usually the best first tool for large flight lines because it can query parquet lazily.
import duckdb
duckdb.query("""
SELECT *
FROM '..._brdfandtopo_corrected_envi.parquet'
LIMIT 5
""").df()
Check the size of a product:
duckdb.query("""
SELECT COUNT(*) AS nrows
FROM '..._landsat_oli_envi.parquet'
""").df()
Summarize reflectance by wavelength:
duckdb.query("""
SELECT wavelength_nm, AVG(reflectance) AS mean_reflectance
FROM '..._landsat_oli_envi.parquet'
GROUP BY wavelength_nm
ORDER BY wavelength_nm
""").df()
Python access
Use pandas carefully and keep ENVI for raster-style work¶
Pandas
Pandas works well once you have narrowed the data down to a manageable size.
import pandas as pd
df = pd.read_parquet("..._merged_pixel_extraction.parquet")
df.head()
Raster and cube views
If you need full spatial cube behavior, the ENVI .img/.hdr outputs are still the better interface. Parquet-to-xarray workflows usually work best after pivoting or aggregation.
For big flight lines, prefer DuckDB for filtering and aggregation first, then collect smaller results into pandas.
Where to go next