Bulk Models¶

Module: spectralbridge.bulk.models.

Data contracts for production bulk analysis.

API glossary · Workflow / scientific context

Signatures and documentation below are collected from the current source. Advanced objects are not newly declared stable by inclusion here.

Audience: public. Also exported as spectralbridge.bulk.SourceFileRecord, spectralbridge.pipelines.bulk.SourceFileRecord.

SourceFileRecord `dataclass` ¶

SourceFileRecord(source_id: str, candidate_id: str, canonical_flightline_id: str | None, identity_source: str | None, site: str | None, acquisition_date: str | None, source_directory: str, source_path: str, relative_path: str, input_kind: str, status: str, reason: str | None, row_count: int | None, column_count: int | None, size_bytes: int, modified_time_ns: int, schema_sha256: str | None, available_sensors_json: str, translation_eligible: bool, product_role: str = 'merged_parquet', sensor_name: str | None = None, header_path: str | None = None, dimensions_json: str = '{}', source_signature_sha256: str | None = None, qa_status: str | None = None, reason_code: str | None = None, matching_group: str | None = None, processing_stage: str | None = None, wavelengths_json: str = '[]', dtype: str | None = None, product_key: str | None = None, storage_format: str | None = None, product_semantics: str | None = None, extraction_mode: str | None = None, schema_json: str = '[]')

One source product discovered in the read-only input tree.

Attributes:

Name	Type	Description
`source_id`	`str`	Deterministic catalog key for this source, candidate directory or exclusion record; not a new ecological sampling unit.
`candidate_id`	`str`	Deterministic catalog key for this source, candidate directory or exclusion record; not a new ecological sampling unit.
`canonical_flightline_id`	`str \| None`	Canonical scientific flightline identifier; histories from the same underlying flight must retain this identity.
`identity_source`	`str \| None`	Parser or manifest provenance explaining how the scientific flightline identity was recovered.
`site`	`str \| None`	Scientific site code recovered from the canonical identity or manifest, not an arbitrary outer directory name.
`acquisition_date`	`str \| None`	ISO acquisition date associated with this scientific flightline; None indicates unavailable metadata.
`source_directory`	`str`	Authoritative local source directory; bulk consumers treat its completed products as immutable.
`source_path`	`str`	Backing input product path retained for traceability and restart checks.
`relative_path`	`str`	Source path relative to the catalog input root for portable provenance.
`input_kind`	`str`	Select eligible 'full', 'polygon', or 'both' table products; discovery still records companions for transparency.
`status`	`str`	Recorded stage-specific outcome or eligibility state; inspect reason/detail and the relevant workflow before interpreting it scientifically.
`reason`	`str \| None`	Explicit reason for exclusion, failure or unavailable status; None when no reason applies.
`row_count`	`int \| None`	Recorded table dimension; None means the dimension could not be established.
`column_count`	`int \| None`	Recorded table dimension; None means the dimension could not be established.
`size_bytes`	`int`	Size or estimated storage requirement in bytes; estimates describe planned artifacts rather than guaranteed disk use.
`modified_time_ns`	`int`	Recorded filesystem modification timestamp in nanoseconds for source-change detection.
`schema_sha256`	`str \| None`	SHA-256 fingerprint of the recorded schema or source identity used for provenance/restart validation.
`available_sensors_json`	`str`	JSON-encoded available sensors retained in compact catalog rows; decode before inspecting nested fields.
`translation_eligible`	`bool`	Whether the recorded product satisfies the corresponding translation availability/eligibility or producer-to-bulk gate; does not certify empirical sensor calibration.
`product_role`	`str, default='merged_parquet'`	Scientific role of this product within the analysis profile, distinct from its storage format.
`sensor_name`	`str \| None, default=None`	Canonical sensor identity used to match registered products and translation relationships.
`header_path`	`str \| None, default=None`	Companion raster header path needed to interpret dimensions, wavelengths and georeferencing.
`dimensions_json`	`str, default='{}'`	JSON-encoded dimensions retained in compact catalog rows; decode before inspecting nested fields.
`source_signature_sha256`	`str \| None, default=None`	SHA-256 fingerprint of the recorded schema or source identity used for provenance/restart validation.
`qa_status`	`str \| None, default=None`	Recorded QA classification; missing or not-evaluated status is not a pass.
`reason_code`	`str \| None, default=None`	Machine-readable exclusion/review reason from the workflow vocabulary.
`matching_group`	`str \| None, default=None`	Registry key declaring products that can be paired on common spatial/observational support.
`processing_stage`	`str \| None, default=None`	Recorded processing lineage, such as raw, corrected or translated; stages are not silently interchangeable.
`wavelengths_json`	`str, default='[]'`	JSON-encoded wavelengths retained in compact catalog rows; decode before inspecting nested fields.
`dtype`	`str \| None, default=None`	Recorded numerical storage dtype of the source product.
`product_key`	`str \| None, default=None`	Unique registry key identifying this descriptor or translation relationship.
`storage_format`	`str \| None, default=None`	Persisted representation such as raster or tabular storage, independent of scientific semantics.
`product_semantics`	`str \| None, default=None`	Declared meaning of the product values and their role in the evidence chain.
`extraction_mode`	`str \| None, default=None`	'full' writes pixel Parquet, 'polygon' requires polygons; None preserves the entry point's documented default extraction behavior.
`schema_json`	`str, default='[]'`	JSON-encoded schema retained in compact catalog rows; decode before inspecting nested fields.

Audience: public. Also exported as spectralbridge.bulk.FlightlineRecord, spectralbridge.pipelines.bulk.FlightlineRecord.

FlightlineRecord `dataclass` ¶

FlightlineRecord(candidate_id: str, canonical_flightline_id: str | None, identity_source: str | None, site: str | None, acquisition_date: str | None, acquisition_year: int | None, source_directory: str, canonical_merged_parquet: str | None, polygon_merged_parquet: str | None, selected_source_ids_json: str, selected_source_paths_json: str, qa_products_json: str, metadata_products_json: str, available_sensors_json: str, processing_stages_json: str, row_count: int | None, size_bytes: int, source_directory_size_bytes: int, schema_fingerprints_json: str, brightness_state_json: str, correction_state_json: str, translation_eligible: bool, status: str, rejection_reason: str | None, duplicate_status: str, duplicate_candidate_count: int, source_provenance_json: str, input_mode: str = 'merged_parquet', corrected_product_json: str = '{}', target_products_json: str = '{}', qa_status: str = 'missing', stage_qa_status_json: str = '{}', missing_products_json: str = '[]', analysis_eligibility_json: str = '{}', estimated_cache_bytes: int = 0, estimated_analysis_output_bytes: int = 0, estimated_materialized_pixel_bytes: int = 0, cache_observations: str | None = None, extraction_status: str = 'not_required', analysis_profile: str = 'translation', processing_completeness: str = 'unknown', product_availability_json: str = '{}', exclusion_reason_codes_json: str = '[]', exclusion_context_json: str = '[]', translation_available: bool = False, translation_availability_json: str = '{}', scientific_status: str = 'not_evaluated', scientific_blockers_json: str = '[]')

Canonical scientific unit recovered from one source directory.

Attributes:

Name	Type	Description
`candidate_id`	`str`	Deterministic catalog key for this source, candidate directory or exclusion record; not a new ecological sampling unit.
`canonical_flightline_id`	`str \| None`	Canonical scientific flightline identifier; histories from the same underlying flight must retain this identity.
`identity_source`	`str \| None`	Parser or manifest provenance explaining how the scientific flightline identity was recovered.
`site`	`str \| None`	Scientific site code recovered from the canonical identity or manifest, not an arbitrary outer directory name.
`acquisition_date`	`str \| None`	ISO acquisition date associated with this scientific flightline; None indicates unavailable metadata.
`acquisition_year`	`int \| None`	Calendar acquisition year used for inventory grouping and completeness checks.
`source_directory`	`str`	Authoritative local source directory; bulk consumers treat its completed products as immutable.
`canonical_merged_parquet`	`str \| None`	Authoritative existing merged product path, or None when the corresponding full/polygon table is unavailable.
`polygon_merged_parquet`	`str \| None`	Authoritative existing merged product path, or None when the corresponding full/polygon table is unavailable.
`selected_source_ids_json`	`str`	JSON-encoded selected source ids retained in compact catalog rows; decode before inspecting nested fields.
`selected_source_paths_json`	`str`	JSON-encoded selected source paths retained in compact catalog rows; decode before inspecting nested fields.
`qa_products_json`	`str`	JSON-encoded qa products retained in compact catalog rows; decode before inspecting nested fields.
`metadata_products_json`	`str`	JSON-encoded metadata products retained in compact catalog rows; decode before inspecting nested fields.
`available_sensors_json`	`str`	JSON-encoded available sensors retained in compact catalog rows; decode before inspecting nested fields.
`processing_stages_json`	`str`	JSON-encoded processing stages retained in compact catalog rows; decode before inspecting nested fields.
`row_count`	`int \| None`	Recorded table dimension; None means the dimension could not be established.
`size_bytes`	`int`	Size or estimated storage requirement in bytes; estimates describe planned artifacts rather than guaranteed disk use.
`source_directory_size_bytes`	`int`	Size or estimated storage requirement in bytes; estimates describe planned artifacts rather than guaranteed disk use.
`schema_fingerprints_json`	`str`	JSON-encoded schema fingerprints retained in compact catalog rows; decode before inspecting nested fields.
`brightness_state_json`	`str`	JSON-encoded brightness state retained in compact catalog rows; decode before inspecting nested fields.
`correction_state_json`	`str`	JSON-encoded correction state retained in compact catalog rows; decode before inspecting nested fields.
`translation_eligible`	`bool`	Whether the recorded product satisfies the corresponding translation availability/eligibility or producer-to-bulk gate; does not certify empirical sensor calibration.
`status`	`str`	Recorded stage-specific outcome or eligibility state; inspect reason/detail and the relevant workflow before interpreting it scientifically.
`rejection_reason`	`str \| None`	Explicit reason for exclusion, failure or unavailable status; None when no reason applies.
`duplicate_status`	`str`	Recorded stage-specific outcome or eligibility state; inspect reason/detail and the relevant workflow before interpreting it scientifically.
`duplicate_candidate_count`	`int`	Number of candidate directories sharing the canonical scientific flightline identity.
`source_provenance_json`	`str`	JSON-encoded source provenance retained in compact catalog rows; decode before inspecting nested fields.
`input_mode`	`str, default='merged_parquet'`	'auto' prefers completed flightline directories and falls back to merged Parquets; alternatives are 'flightline_outputs' and 'merged_parquet'.
`corrected_product_json`	`str, default='{}'`	JSON-encoded corrected product retained in compact catalog rows; decode before inspecting nested fields.
`target_products_json`	`str, default='{}'`	JSON-encoded target products retained in compact catalog rows; decode before inspecting nested fields.
`qa_status`	`str, default='missing'`	Recorded QA classification; missing or not-evaluated status is not a pass.
`stage_qa_status_json`	`str, default='{}'`	JSON-encoded stage qa status retained in compact catalog rows; decode before inspecting nested fields.
`missing_products_json`	`str, default='[]'`	JSON-encoded missing products retained in compact catalog rows; decode before inspecting nested fields.
`analysis_eligibility_json`	`str, default='{}'`	JSON-encoded analysis eligibility retained in compact catalog rows; decode before inspecting nested fields.
`estimated_cache_bytes`	`int, default=0`	Size or estimated storage requirement in bytes; estimates describe planned artifacts rather than guaranteed disk use.
`estimated_analysis_output_bytes`	`int, default=0`	Size or estimated storage requirement in bytes; estimates describe planned artifacts rather than guaranteed disk use.
`estimated_materialized_pixel_bytes`	`int, default=0`	Size or estimated storage requirement in bytes; estimates describe planned artifacts rather than guaranteed disk use.
`cache_observations`	`str \| None, default=None`	Optional path to explicitly materialized pixel observations; absent from ordinary compact bulk analysis.
`extraction_status`	`str, default='not_required'`	Recorded stage-specific outcome or eligibility state; inspect reason/detail and the relevant workflow before interpreting it scientifically.
`analysis_profile`	`str, default='translation'`	Registered analysis profile name or AnalysisProfile controlling required products and permissible sensor pairs.
`processing_completeness`	`str, default='unknown'`	Recorded assessment of required product completeness under the selected analysis profile.
`product_availability_json`	`str, default='{}'`	JSON-encoded product availability retained in compact catalog rows; decode before inspecting nested fields.
`exclusion_reason_codes_json`	`str, default='[]'`	JSON-encoded exclusion reason codes retained in compact catalog rows; decode before inspecting nested fields.
`exclusion_context_json`	`str, default='[]'`	JSON-encoded exclusion context retained in compact catalog rows; decode before inspecting nested fields.
`translation_available`	`bool, default=False`	Whether the recorded product satisfies the corresponding translation availability/eligibility or producer-to-bulk gate; does not certify empirical sensor calibration.
`translation_availability_json`	`str, default='{}'`	JSON-encoded translation availability retained in compact catalog rows; decode before inspecting nested fields.
`scientific_status`	`str, default='not_evaluated'`	Recorded stage-specific outcome or eligibility state; inspect reason/detail and the relevant workflow before interpreting it scientifically.
`scientific_blockers_json`	`str, default='[]'`	JSON-encoded scientific blockers retained in compact catalog rows; decode before inspecting nested fields.

Audience: public. Also exported as spectralbridge.bulk.ExclusionRecord.

ExclusionRecord `dataclass` ¶

ExclusionRecord(exclusion_id: str, canonical_flightline_id: str | None, source_path: str, site: str | None, acquisition_date: str | None, analysis_profile: str, product_role: str | None, sensor_name: str | None, offending_files_json: str, reason_code: str, detail: str, processing_stage: str | None)

One deterministic, machine-readable bulk exclusion.

Attributes:

Name	Type	Description
`exclusion_id`	`str`	Deterministic catalog key for this source, candidate directory or exclusion record; not a new ecological sampling unit.
`canonical_flightline_id`	`str \| None`	Canonical scientific flightline identifier; histories from the same underlying flight must retain this identity.
`source_path`	`str`	Backing input product path retained for traceability and restart checks.
`site`	`str \| None`	Scientific site code recovered from the canonical identity or manifest, not an arbitrary outer directory name.
`acquisition_date`	`str \| None`	ISO acquisition date associated with this scientific flightline; None indicates unavailable metadata.
`analysis_profile`	`str`	Registered analysis profile name or AnalysisProfile controlling required products and permissible sensor pairs.
`product_role`	`str \| None`	Scientific role of this product within the analysis profile, distinct from its storage format.
`sensor_name`	`str \| None`	Canonical sensor identity used to match registered products and translation relationships.
`offending_files_json`	`str`	JSON-encoded offending files retained in compact catalog rows; decode before inspecting nested fields.
`reason_code`	`str`	Machine-readable exclusion/review reason from the workflow vocabulary.
`detail`	`str`	Human-readable context supporting the exclusion or decision record.
`processing_stage`	`str \| None`	Recorded processing lineage, such as raw, corrected or translated; stages are not silently interchangeable.

Audience: public. Also exported as spectralbridge.bulk.BulkAnalysisPaths, spectralbridge.pipelines.bulk.BulkAnalysisPaths.

BulkAnalysisPaths `dataclass` ¶

BulkAnalysisPaths(output_dir: Path)

Canonical, isolated output paths for one bulk-analysis run.

Attributes:

Name	Type	Description
`output_dir`	`Path`	Directory for generated products; bulk outputs must be outside the read-only input tree.

catalog_dir `property` ¶

catalog_dir: Path

Resolve the canonical catalog dir location.

Returns:

Type	Description
`Path`	Path computed as `self.output_dir / 'catalog'`. Accessing this property does not create or validate the artifact.

database_dir `property` ¶

database_dir: Path

Resolve the canonical database dir location.

Returns:

Type	Description
`Path`	Path computed as `self.output_dir / 'database'`. Accessing this property does not create or validate the artifact.

cache_dir `property` ¶

cache_dir: Path

Legacy pixel-materialization directory; unused by normal analysis.

statistics_dir `property` ¶

statistics_dir: Path

Resolve the canonical statistics dir location.

Returns:

Type	Description
`Path`	Path computed as `self.output_dir / 'statistics'`. Accessing this property does not create or validate the artifact.

flightline_statistics_dir `property` ¶

flightline_statistics_dir: Path

Resolve the canonical flightline statistics dir location.

Returns:

Type	Description
`Path`	Path computed as `self.statistics_dir / 'flightlines'`. Accessing this property does not create or validate the artifact.

sufficient_statistics `property` ¶

sufficient_statistics: Path

Resolve the canonical sufficient statistics location.

Returns:

Type	Description
`Path`	Path computed as `self.statistics_dir / 'translation_sufficient_statistics.parquet'`. Accessing this property does not create or validate the artifact.

diagnostic_sample `property` ¶

diagnostic_sample: Path

Resolve the canonical diagnostic sample location.

Returns:

Type	Description
`Path`	Path computed as `self.statistics_dir / 'diagnostic_sample.parquet'`. Accessing this property does not create or validate the artifact.

analyses_dir `property` ¶

analyses_dir: Path

Resolve the canonical analyses dir location.

Returns:

Type	Description
`Path`	Path computed as `self.output_dir / 'analyses'`. Accessing this property does not create or validate the artifact.

coefficients_dir `property` ¶

coefficients_dir: Path

Resolve the canonical coefficients dir location.

Returns:

Type	Description
`Path`	Path computed as `self.output_dir / 'coefficients'`. Accessing this property does not create or validate the artifact.

tables_dir `property` ¶

tables_dir: Path

Resolve the canonical tables dir location.

Returns:

Type	Description
`Path`	Path computed as `self.output_dir / 'tables'`. Accessing this property does not create or validate the artifact.

figures_dir `property` ¶

figures_dir: Path

Resolve the canonical figures dir location.

Returns:

Type	Description
`Path`	Path computed as `self.output_dir / 'figures'`. Accessing this property does not create or validate the artifact.

reports_dir `property` ¶

reports_dir: Path

Resolve the canonical reports dir location.

Returns:

Type	Description
`Path`	Path computed as `self.output_dir / 'reports'`. Accessing this property does not create or validate the artifact.

logs_dir `property` ¶

logs_dir: Path

Resolve the canonical logs dir location.

Returns:

Type	Description
`Path`	Path computed as `self.output_dir / 'logs'`. Accessing this property does not create or validate the artifact.

flightlines `property` ¶

flightlines: Path

Resolve the canonical flightlines location.

Returns:

Type	Description
`Path`	Path computed as `self.catalog_dir / 'flightlines.parquet'`. Accessing this property does not create or validate the artifact.

source_files `property` ¶

source_files: Path

Resolve the canonical source files location.

Returns:

Type	Description
`Path`	Path computed as `self.catalog_dir / 'source_files.parquet'`. Accessing this property does not create or validate the artifact.

source_products `property` ¶

source_products: Path

Resolve the canonical source products location.

Returns:

Type	Description
`Path`	Path computed as `self.catalog_dir / 'source_products.parquet'`. Accessing this property does not create or validate the artifact.

duplicates `property` ¶

duplicates: Path

Resolve the canonical duplicates location.

Returns:

Type	Description
`Path`	Path computed as `self.catalog_dir / 'duplicates.parquet'`. Accessing this property does not create or validate the artifact.

rejected_sources `property` ¶

rejected_sources: Path

Resolve the canonical rejected sources location.

Returns:

Type	Description
`Path`	Path computed as `self.catalog_dir / 'rejected_sources.parquet'`. Accessing this property does not create or validate the artifact.

exclusions `property` ¶

exclusions: Path

Resolve the canonical exclusions location.

Returns:

Type	Description
`Path`	Path computed as `self.catalog_dir / 'exclusions.parquet'`. Accessing this property does not create or validate the artifact.

exclusions_json `property` ¶

exclusions_json: Path

Resolve the canonical exclusions json location.

Returns:

Type	Description
`Path`	Path computed as `self.catalog_dir / 'exclusions.json'`. Accessing this property does not create or validate the artifact.

exclusions_csv `property` ¶

exclusions_csv: Path

Resolve the canonical exclusions csv location.

Returns:

Type	Description
`Path`	Path computed as `self.catalog_dir / 'exclusions.csv'`. Accessing this property does not create or validate the artifact.

manifest `property` ¶

manifest: Path

Resolve the canonical manifest location.

Returns:

Type	Description
`Path`	Path computed as `self.catalog_dir / 'bulk_manifest.json'`. Accessing this property does not create or validate the artifact.

database `property` ¶

database: Path

Resolve the canonical database location.

Returns:

Type	Description
`Path`	Path computed as `self.database_dir / 'spectralbridge_bulk.duckdb'`. Accessing this property does not create or validate the artifact.

observations `property` ¶

observations: Path

Optional portable materialization; absent in the default mode.

source_catalog `property` ¶

source_catalog: Path

Compatibility alias for the source-file catalog.

coefficients_parquet `property` ¶

coefficients_parquet: Path

Resolve the canonical coefficients parquet location.

Returns:

Type	Description
`Path`	Path computed as `self.coefficients_dir / 'candidate_translation_coefficients.parquet'`. Accessing this property does not create or validate the artifact.

coefficients_json `property` ¶

coefficients_json: Path

Resolve the canonical coefficients json location.

Returns:

Type	Description
`Path`	Path computed as `self.coefficients_dir / 'candidate_translation_coefficients.json'`. Accessing this property does not create or validate the artifact.

analysis_decisions `property` ¶

analysis_decisions: Path

Resolve the canonical analysis decisions location.

Returns:

Type	Description
`Path`	Path computed as `self.reports_dir / 'analysis_decisions.json'`. Accessing this property does not create or validate the artifact.

campaign_summary `property` ¶

campaign_summary: Path

Resolve the canonical campaign summary location.

Returns:

Type	Description
`Path`	Path computed as `self.reports_dir / 'campaign_summary.md'`. Accessing this property does not create or validate the artifact.

ensure_directories ¶

ensure_directories() -> None

Create the canonical output directory hierarchy.

Returns:

Type	Description
`None`	No return value; validation raises on unsupported configuration and directory creation persists the canonical hierarchy.

Explicitly exported constants and type aliases¶

These values describe package metadata, schemas or configuration; they are not observations from a production dataset.

Name	Source expression
`BULK_SCHEMA_VERSION`	`9`
`BulkInputKind`	`Literal['full', 'polygon', 'both']`
`BulkInputMode`	`Literal['auto', 'flightline_outputs', 'merged_parquet']`
`BulkSource`	`SourceFileRecord`

Bulk Models¶

SourceFileRecord dataclass ¶

FlightlineRecord dataclass ¶

ExclusionRecord dataclass ¶

BulkAnalysisPaths dataclass ¶

catalog_dir property ¶

database_dir property ¶

cache_dir property ¶

statistics_dir property ¶

flightline_statistics_dir property ¶

sufficient_statistics property ¶

diagnostic_sample property ¶

analyses_dir property ¶

coefficients_dir property ¶

tables_dir property ¶

figures_dir property ¶

reports_dir property ¶

logs_dir property ¶

flightlines property ¶

source_files property ¶

source_products property ¶

duplicates property ¶

rejected_sources property ¶

exclusions property ¶

exclusions_json property ¶

exclusions_csv property ¶

manifest property ¶

database property ¶

observations property ¶

source_catalog property ¶

coefficients_parquet property ¶

coefficients_json property ¶

analysis_decisions property ¶

campaign_summary property ¶

ensure_directories ¶

Explicitly exported constants and type aliases¶

SourceFileRecord `dataclass` ¶

FlightlineRecord `dataclass` ¶

ExclusionRecord `dataclass` ¶

BulkAnalysisPaths `dataclass` ¶

catalog_dir `property` ¶

database_dir `property` ¶

cache_dir `property` ¶

statistics_dir `property` ¶

flightline_statistics_dir `property` ¶

sufficient_statistics `property` ¶

diagnostic_sample `property` ¶

analyses_dir `property` ¶

coefficients_dir `property` ¶

tables_dir `property` ¶

figures_dir `property` ¶

reports_dir `property` ¶

logs_dir `property` ¶

flightlines `property` ¶

source_files `property` ¶

source_products `property` ¶

duplicates `property` ¶

rejected_sources `property` ¶

exclusions `property` ¶

exclusions_json `property` ¶

exclusions_csv `property` ¶

manifest `property` ¶

database `property` ¶

observations `property` ¶

source_catalog `property` ¶

coefficients_parquet `property` ¶

coefficients_json `property` ¶

analysis_decisions `property` ¶

campaign_summary `property` ¶