Bulk orchestration¶

Module: spectralbridge.pipelines.bulk.

Orchestration for the independent production bulk-analysis pipeline.

Discovery, the virtual DuckDB dataset, provenance, and scientific analyses live under the dedicated spectralbridge.bulk package.

API glossary · Workflow / scientific context

Signatures and documentation below are collected from the current source. Advanced objects are not newly declared stable by inclusion here.

Audience: public. Also exported as spectralbridge.pipelines.run_bulk_pipeline, spectralbridge.run_bulk_pipeline.

run_bulk_pipeline ¶

run_bulk_pipeline(input_path: str | Path, output_dir: str | Path | None = None, *, input_kind: BulkInputKind = 'full', input_mode: BulkInputMode = 'auto', analysis: str | AnalysisProfile = 'translation', sensors: tuple[str, ...] | list[str] | None = None, translation_pairs: Sequence[str | TranslationPair] | None = None, product_registry: ProductRegistry = DEFAULT_PRODUCT_REGISTRY, identity_parsers: Sequence[FlightlineIdentityParser] = DEFAULT_IDENTITY_PARSERS, on_invalid: str = 'exclude', minimum_reflectance: float = 0.0, require_translation_pairs: bool = True, materialize_observations: bool = False, row_group_size: int = 50000, memory_limit: str | None = None, threads: int | None = None, temp_directory: str | Path | None = None, preflight_only: bool = False, extraction_chunk_size: int = 2048, extraction_workers: int = 1, diagnostic_sample_size: int = 0, diagnostic_seed: int = 0, spectral_library: str | Path | None = None, make_summary_plots: bool = False, make_full_spectral_reports: bool = False, spectral_library_config: SpectralLibraryPlotConfig | None = None, force: bool = False) -> dict[str, Any]

Catalog completed flightlines and run population analyses.

Parameters:

Name	Type	Description	Default
`input_path`	`str \| Path`	Existing source tree or merged Parquet to inventory; input products are read in place.	required
`output_dir`	`str \| Path \| None`	Directory for generated products; bulk outputs must be outside the read-only input tree.	`None`
`input_kind`	`BulkInputKind`	Select eligible 'full', 'polygon', or 'both' table products; discovery still records companions for transparency.	`'full'`
`input_mode`	`BulkInputMode`	'auto' prefers completed flightline directories and falls back to merged Parquets; alternatives are 'flightline_outputs' and 'merged_parquet'.	`'auto'`
`analysis`	`str \| AnalysisProfile`	Registered analysis profile name or AnalysisProfile controlling required products and permissible sensor pairs.	`'translation'`
`sensors`	`tuple[str, ...] \| list[str] \| None`	Optional sensor-name selection passed to the product registry; None retains profile-compatible sensors.	`None`
`translation_pairs`	`Sequence[str \| TranslationPair] \| None`	Explicit registered source-to-target relationships to analyze; None uses the selected registry/profile defaults.	`None`
`product_registry`	`ProductRegistry`	ProductRegistry describing recognized filenames, sensor roles, band contracts and translation relationships.	`DEFAULT_PRODUCT_REGISTRY`
`identity_parsers`	`Sequence[FlightlineIdentityParser]`	Ordered parsers for canonical scientific identity from filenames or manifests; outer run-folder names are not scientific identifiers.	`DEFAULT_IDENTITY_PARSERS`
`on_invalid`	`str`	'exclude' records invalid sources; 'error' stops instead of continuing with an excluded population.	`'exclude'`
`minimum_reflectance`	`float`	Lower screening bound in the stored reflectance units; used for regression validity, not automatic unit conversion.	`0.0`
`require_translation_pairs`	`bool`	Require eligible registered pairs; False permits a modified profile without a translation-pair requirement.	`True`
`materialize_observations`	`bool`	Explicit opt-in to pixel-scale observation materialization; normal bulk runs retain compact sufficient statistics only.	`False`
`row_group_size`	`int`	Positive rows per written Parquet row group; None in the merge workflow delegates to its established default.	`50000`
`memory_limit`	`str \| None`	DuckDB memory budget string, such as "4GB"; None leaves the connection default.	`None`
`threads`	`int \| None`	Positive DuckDB thread budget; None delegates to the established database default.	`None`
`temp_directory`	`str \| Path \| None`	DuckDB spill directory; bulk temporary storage must be outside the read-only source tree.	`None`
`preflight_only`	`bool`	Return catalog/eligibility and cost information without executing the population analyses.	`False`
`extraction_chunk_size`	`int`	Positive bounded raster window size in pixels for sufficient-statistics extraction.	`2048`
`extraction_workers`	`int`	Positive number of independent flightline extraction workers; storage bandwidth can limit useful concurrency.	`1`
`diagnostic_sample_size`	`int`	Maximum diagnostic pixel sample count; zero disables row-level diagnostic sample output.	`0`
`diagnostic_seed`	`int`	Random-number seed for reproducible sampling, splits or resampling; it does not create independent observations.	`0`
`spectral_library`	`str \| Path \| None`	Existing merged polygon Parquet with wavelength-bearing columns; read in place without copying the source dataset.	`None`
`make_summary_plots`	`bool`	Write compact spectral-summary figures in addition to numerical summaries.	`False`
`make_full_spectral_reports`	`bool`	Explicitly enable potentially expensive full trace reports; inspect preflight costs before enabling.	`False`
`spectral_library_config`	`SpectralLibraryPlotConfig \| None`	SpectralLibraryPlotConfig controlling group fields, stage, validity and rendering; None uses its defaults.	`None`
`force`	`bool`	Regenerate the requested outputs instead of reusing an otherwise valid checkpoint or report.	`False`

Returns:

Type	Description
`dict[str, Any]`	Status, accepted/rejected/duplicate counts, pair counts, manifest/catalog/database and analysis paths; preflight returns before fits.

Raises:

Type	Description
`FileNotFoundError`	If a required source or completed artifact is absent.
`ValueError`	If the supplied options, dimensions, support or provenance violate the input contract.

Notes

The source tree is treated as read only. Scientific identity is recovered through configurable identity parsers, never arbitrary outer run-folder names. Duplicate flightline IDs in different source directories are cataloged and excluded from analysis.

input_mode='auto' prefers canonical completed-flightline directories and falls back to prebuilt merged Parquets. Completed-flightline mode needs only the target products required by the selected profile and relationship; it reads them in bounded chunks and writes only compact, restart-safe sufficient statistics beneath output_dir. Pixel observations are materialized only when materialize_observations=True is explicitly set. Canonical drone Landsat-like application products are inventoried and may be fitted as explicitly labeled application-verification diagnostics. They are not independent evidence for updating the coefficient registry that produced them.