Merge Duckdb¶

Module: spectralbridge.merge_duckdb.

spectralbridge.merge_duckdb¶

DuckDB-backed merge utilities for the spectralbridge pipeline.

The public entry point :func:merge_flightline stitches every ENVI-derived Parquet table for a single flightline into one master artifact named <scene_prefix>_merged_pixel_extraction.parquet. The merged parquet includes pixel metadata, the full set of original/corrected wavelengths, and any resampled sensor bands.

Unless qa=False is passed, completing the merge automatically invokes qa_plots.render_flightline_panel to emit <scene_prefix>_qa.png in the same directory. This mirrors the default behaviour of python -m bin.merge_duckdb as documented in the CLI.

Typical usage::

from spectralbridge import merge_duckdb

merge_duckdb.merge_flightline(
    Path("/path/to/flightline"),
    out_name="custom_master.parquet",  # optional override
    qa=True,
)

The module intentionally minimises assumptions about directory layout. The default glob patterns match the canonical pipeline exports but can be overridden to support custom workflows.

API glossary · Workflow / scientific context

Signatures and documentation below are collected from the current source. Advanced objects are not newly declared stable by inclusion here.

Audience: advanced.

merge_flightline ¶

merge_flightline(flightline_dir: Path, out_name: str | None = None, original_glob: str = '**/*original*.parquet', corrected_glob: str = '**/*corrected*.parquet', resampled_glob: str = '**/*resampl*.parquet', write_feather: bool = False, emit_qa_panel: bool = True, ray_cpus: int | None = None, *, merge_memory_limit_gb: float | str | None = 64.0, merge_threads: int | None = 4, merge_row_group_size: int | None = None, merge_temp_directory: Path | None = None, is_polygon_mode: bool | None = None, filter_no_data_rows: bool = True) -> Path

Merge all pixel-level parquet tables for one flightline.

Parameters:

Name	Type	Description	Default
`flightline_dir`	`Path`	Directory containing the flightline's parquet outputs.	required
`out_name`	`str`	Custom name for the merged parquet. If None, defaults to: _merged_pixel_extraction.parquet	`None`
`original_glob`	`str`	Glob used to locate original reflectance parquet tables.	`'*/original*.parquet'`
`corrected_glob`	`str`	Glob used to locate corrected reflectance parquet tables.	`'*/corrected*.parquet'`
`resampled_glob`	`str`	Glob used to locate resampled sensor parquet tables.	`'*/resampl*.parquet'`
`write_feather`	`bool`	If True, writes a Feather copy of the merged table alongside the parquet.	`False`
`emit_qa_panel`	`bool`	If True, renders the standard QA panel (_qa.png) after merging.	`True`
`ray_cpus`	`int`	CPU budget forwarded to Ray validation of Parquet shards. Defaults to `None` which allows the Ray helper to choose the configured default.	`None`
`merge_memory_limit_gb`	`float or str`	Upper bound on DuckDB's memory usage. Floats are interpreted as GiB. Provide `None` to leave the default DuckDB behaviour unchanged.	`64.0`
`merge_threads`	`int`	Number of threads DuckDB should use for the merge. `None` keeps the engine default (usually `os.cpu_count()`).	`4`
`merge_row_group_size`	`int or None`	Target number of rows per Parquet row group in the merged output. If None, DuckDB will auto-determine the optimal size (better streaming performance).	`None`
`merge_temp_directory`	`Path`	Directory where DuckDB should spill temporary data. Defaults to a `.duckdb_tmp` subdirectory inside `flightline_dir`.	`None`

Returns:

Type	Description
`Path`	Path to the merged parquet file.

Audience: advanced.

merge_all_flightlines ¶

merge_all_flightlines(data_root: Path, *, flightline_glob: str = 'NEON_*', **kwargs) -> List[Path]

Run per-flightline merges across matching directories, collecting successful output paths and reporting failures.