Inspect and report a spectral library

A spectral library is a table of spectra with grouping information, such as plant species, polygon, flightline or site. Start with input preflight before requesting large plots; retain the supplied table as the source of truth.

Check cross-flight wavelength grids first

This adapter requires one compatible physical wavelength grid. A cross-flight Parquet union can contain several native grids with structural nulls and repeated band indices. spectral_stage="corr" selects a stage; it does not reconstruct or harmonize those grids. Follow the Parquet data-model guide before passing such a table. Pixel counts are not independent plant counts.

Visualize an existing polygon spectral library

The intentional merged polygon spectral library is different from the removed temporary observation cache. Pass its existing Parquet path separately: it is opened read-only and is never copied into the bulk output. The adapter inspects the footer before analysis, detects the species and available hierarchy fields, and selects one compatible wavelength-bearing spectral stage. It prefers corr_*_wl*nm columns when present; multiple other stages require an explicit choice. Wavelengths are parsed from column names, numerically sorted, and kept paired to their original values. Duplicate wavelengths, duplicate band indices, non-numeric spectral columns, and schemas without physical wavelengths fail explicitly.

from spectralbridge import inspect_spectral_library_preflight, run_bulk_pipeline
from spectralbridge.bulk import SpectralLibraryPlotConfig

plots = SpectralLibraryPlotConfig(
    spectral_stage="corr",
    species_sort="count_desc",       # or "alphabetical"
    panels_per_page=4,
    trace_batch_size=2_000,
    summary_band_batch_size=32,
    max_traces_per_group=None,        # default: show every valid trace
    raster_dpi=150,
    plot_y_quantiles=(0.005, 0.995),
    species_y_scale="global_robust",
    spectral_plot_minimum_reflectance=None,
)

library = "/data/library/polygons_merged_pixel_extraction.parquet"
preflight = inspect_spectral_library_preflight(library, config=plots)
print(preflight["schema"])
print(preflight["counts"])
print(preflight["largest_species"])
print(preflight["expected_pages"])

result = run_bulk_pipeline(
    "/data/completed_products",
    "/data/bulk_analysis",
    spectral_library=library,
    make_summary_plots=True,
    make_full_spectral_reports=True,
    spectral_library_config=plots,
)
print(result["preflight"]["spectral_library_schema"])
print(result["spectral_library"]["reports"])

Use the public preflight call before launching a bulk run. It reads the Parquet in place and returns the source size and signature, detected fields and spectral stage, row and hierarchy counts, largest and median species sizes, groups over 10,000 and 100,000 valid spectra, trace-cap state, expected pages, and estimated logical scans. It writes no summaries or PDFs. The recommended production sequence is: (1) preflight, (2) run with both plot flags false to create compact summaries and inspect robust bounds/extremes, (3) enable summary plots, then (4) enable the full suite.

make_summary_plots=True writes the between-species median overview and species observation-count report. make_full_spectral_reports=True also writes the species low-alpha ensemble, nested quantile envelopes, and available site, flightline, and within-species hierarchy reports. It also implies the two summary PDFs. No report is generated by default because large libraries can require substantial sequential I/O and rendering time.

The trace alpha is deterministic:

alpha = min(0.03, max(0.003, 0.2 / sqrt(rendered trace count)))

This makes tens of spectra individually legible and progressively lowers alpha for dense groups. It is a qualitative spectral trace density, not a normalized probability density. The raw trace layer is rendered batch by batch into a fixed-size raster; titles, axes, annotations, and the median remain vector PDF elements. By default every complete finite spectrum contributes. Sampling is used only when max_traces_per_group (or --spectral-max-traces-per-group) is explicitly set, in which case a seeded DuckDB hash order makes the cap reproducible.

The compact products are species_summary.parquet, species_band_summary.parquet, species_quantiles.parquet, species_median_spectra.parquet, group_counts.parquet, species_plot_ranges.parquet, and extreme_spectra.parquet. Quantiles default to 2.5, 10, 25, 50, 75, 90, and 97.5 percent and use DuckDB approx_quantile in bounded band batches.

Visualization validity is intentionally separate from regression validity. Every selected-stage value must be present and finite; explicit nodata sentinels from recognized Parquet metadata and nodata_values (default -9999) are excluded within nodata_tolerance. Finite negative corrected reflectance remains visible by default. Set spectral_plot_minimum_reflectance only when a visualization-specific lower threshold is scientifically warranted. This does not change translation/regression validity.

The primary species report defaults to one common robust range, estimated from the 0.5th and 99.5th percentiles without loading the library into memory. Values outside it are clipped graphically only: they remain in analytical medians, quantiles, min/max, and counts. Plot-range provenance records the bounds, method, value and spectrum exceedance counts, and per-species indicators. The bounded extreme_spectra.parquet ranks up to max_extreme_spectra_per_species observations per species with source context. species_y_scale="global_full" uses one observed full range, while "per_group_robust" maximizes within-species detail at the cost of direct cross-species scale comparison. The separate full-range audit PDF is always part of the full suite; other scale combinations are not generated automatically.

Memory is bounded by one Arrow trace batch plus at most the configured page's fixed-size panel rasters during plotting, and by one configured band batch during summary calculation. The companion JSON records source signature, package version, run ID, detected schema, counts, wavelength range, exact plot configuration, alpha rule, rasterization, sampling, visualization validity, plot ranges, extreme diagnostics, report-cost estimates, report paths, page counts, and sizes.

The full suite uses these canonical names when the corresponding grouping field exists:

• spectral_library_species_variability.pdf
• spectral_library_species_variability_full_range.pdf
• spectral_library_species_quantiles.pdf
• spectral_library_species_medians.pdf
• spectral_library_observation_counts.pdf
• spectral_library_hierarchical_variability.pdf
• spectral_library_site_variability.pdf
• spectral_library_flightline_variability.pdf

These reports are additional products. Dataset census, translation, cross- sensor, LOSO, and other QA/statistical outputs remain unchanged.

For raster-population processing, return to bulk analysis.

For merged polygon libraries containing different native wavelength grids, use the plant-balanced spectral atlas. It audits actual flight/source support, aggregates pixels into independent GlobalID plants, and preserves unsupported atmospheric gaps before making species panels.

Paired PFT structure under sensor reduction

For synthetic sensor representations of the same NEON observations, compare identical polygon/year units and acquisition histories before attributing a change to spectral resolution. The plant spectral atlas workflow reports paired OLI effect, direction and ranking retention with cluster-aware uncertainty; the primary grouping is now recorded researcher cover_category; older taxonomic groupings remain historical only. These are conditional group-structure results, not independent satellite/drone validation or classification accuracy. The supplied MicaSense export fails its physical-coordinate gate and is excluded from biological inference.

For abundance questions, use the separate MESMA readiness assessment. It evaluates held-out mixture geometry and fractions; a previous spectral difference test or synthetic sensor-retention result is not an abundance validation. The current library supports controlled diagnostics, with operational material fractions blocked by source/purity/background/reference-data gaps.

For the observation model, label systems, empirical atlas and current ecological evidence, start with the spectral-library overview. This page retains the stable preflight/report workflow.