DESCRIPTION

i.hyper.lib_relab is the third i.hyper.lib_* harvester (after i.hyper.lib_ecosis and i.hyper.lib_usgs): it parses a local download of the RELAB (Reflectance Experiment Laboratory) spectral database, run by Brown University for NASA's Keck/RELAB program, and writes it into the same shared, source-tagged database at $HOME/grassdata/hyperspeclib, using the exact same row schema as the other two harvesters — one pyarrow.dataset scan spans EcoSIS field spectra, USGS lab reference spectra, and RELAB lab reference spectra together.

RELAB's data release, like USGS's, has no live query API — it is a static downloaded archive. input_dir must point at the root of that download (e.g. ~/DBDATA/RelabDatabase2025Dec31), containing its catalogues/ and data/ subdirectories as distributed.

Downloading the data

  1. Go to the RELAB Spectral Database page: https://sites.brown.edu/relab/relab-spectral-database/ (Reflectance Experiment Laboratory, Department of Earth, Environmental and Planetary Sciences, Brown University, run for NASA's Keck/RELAB program).
  2. The page offers the two catalogue files (Sample_Catalogue / Spectra_Catalogue, needed to look up individual spectra by sample) and, per this module's own ReadMeFirst.txt, "a compressed zip file containing all the catalogues and data" as a single combined download — that single zip is the simplest option and is what this harvester expects.
  3. Extract the zip into a directory of your choice, e.g. ~/DBDATA/RelabDatabase<date>/ — it unpacks directly into catalogues/ (the TSV catalogue files) and data/ (one .asc/.txt pair per spectrum, bucketed into data/<pi_code>/<sample_id prefix>/) as siblings. This extracted root is what input_dir must point at.
  4. The full archive is several GB (34,564 spectra in the 2025-12-31 release used to build and verify this harvester); ingestion reads only the .asc files (not the equivalent .txt files, which carry the same data in a different unit/format) plus the small catalogue TSVs.

Why this needed its own parser (not shared with the other two)

RELAB's on-disk layout was studied via the reference parser in $HOME/dev/spectral/spectral/database/relab.py (the spectral Python package) and via the archive's own ReadMeFirst.txt — not used as a dependency, reimplemented here because RELAB's real local download turned out to be richer and more structured than either the reference parser or i.hyper.lib_usgs's per-sample-HTML approach: it ships two proper relational catalogue tables (tab-separated, one header row each) instead of requiring metadata to be scraped out of free text per spectrum.

This catalogue-driven approach means no per-file HTML scraping is needed at all: Spectra_Catalogue.txt is scanned once (a cheap metadata-only pass, joined against the in-memory Sample_Catalogue dict) to group every spectrum by category before touching a single .asc file, then each category's actual spectra are parsed and written in the same incremental, batched way as the other two harvesters.

Dataset grouping

RELAB has no chapter-like folder structure at all (its data/ directory is bucketed by PI code + sample-ID prefix, an organizational convenience with no semantic meaning) — so, analogous to i.hyper.lib_usgs's use of USGS's chapters, this harvester groups by Sample_Catalogue's GeneralType1 field (Mineral, Rock, Sediment, Glass, Biological, Mixture, Organic, Regolith, ... — about 26 categories observed in practice, including a few small/idiosyncratic ones and an Unclassified bucket for samples with no GeneralType1 recorded), sanitized into a filesystem-safe dataset_id (e.g. Chemical Compound -> Chemical_Compound) while keeping the original text as dataset_title. general_types= restricts ingestion to specific categories (matched against the sanitized form; default: all).

No geolocation, no live per-record web page

Same as i.hyper.lib_usgs: RELAB lab samples have no coordinates (a free-text Location/Origin field exists for some, not structured lat/lon), so longitude/latitude are always null for this source. And unlike EcoSIS, there is no confirmed single per-record web page or DOI to cite in the local download itself, so source_url/source_api_url are left null rather than guessed — per-record traceability instead comes entirely from extra_metadata's local_spectrum_file path plus the SpectrumID/SampleID identifiers, which any RELAB catalogue search can resolve back to the original record.

Incremental re-runs and streaming

Same conventions as the other two harvesters: a shared _manifest.json at the database root (keyed relab:<category>) tracks each category's source spectrum count (expected_total) to decide whether a re-run can skip it (-f to force), and records are batched (stream_batch_size, default 2000) and written incrementally via the same ParquetWriter/executemany() per-batch pattern.

NOTES

Use -k to keep going past a spectrum that raises a parse error instead of aborting the whole run. A spectrum whose catalogued .asc file is simply missing on disk (a real, if uncommon, gap in the archive itself — confirmed present in this session's actual download: 170 of 33,786 catalogued spectra) is always skipped silently regardless of -k, since that is an expected, not exceptional, condition.

EXAMPLES

Ingest a single (small) category, GeoParquet backend

i.hyper.lib_relab input_dir=$HOME/DBDATA/RelabDatabase2025Dec31 \
    output=/tmp/test_relab_lib format=parquet general_types=Coating
Loading RELAB catalogues (Sample_Catalogue, PI_Address, MeasurementTypes)…
16786 samples in Sample_Catalogue.
Scanning Spectra_Catalogue…
Found 10 spectra across 1 category/categories.
Wrote 10 record(s) for 'Coating' →
/tmp/test_relab_lib/source_database=relab/dataset_Coating.parquet
Done: 1 categor(y/ies) ingested (10 spectra), 0 already up to date, library
at /tmp/test_relab_lib (format=parquet).

One of those 10 rows, inspected directly:

record_id:      BIR1LH053
dataset_title:  Coating
organization:   Department of Geological Sciences, Brown University
n_bands:        3527
wavelengths[:3]: [1428.4, 1428.8, 1429.2]   (nm -- already native, no conversion)
extra_metadata:
  sample_id:          LH-JFM-053
  sample_name:        SB-01 chips
  general_type1:      Coating
  type1:              Iron Oxide
  sub_type:           Jarosite
  texture:            Chip
  origin:             Rio Tinto, Spain
  spectrometer_name:  Thermo Nexus 870 with PIKE AutoDiff
  measurement_mode:   Reflectance (Off-axis biconical FT-IR)
  spectra_catalogue_speccode:      BCF-FTIR2
  spectra_catalogue_researchtype:  Astrobiology, Student
  local_spectrum_file: .../data/jfm/lh/bir1lh053.asc

Full library: all categories into the shared database

i.hyper.lib_relab input_dir=$HOME/DBDATA/RelabDatabase2025Dec31
Loading RELAB catalogues (Sample_Catalogue, PI_Address, MeasurementTypes)…
16786 samples in Sample_Catalogue.
Scanning Spectra_Catalogue…
Found 33786 spectra across 26 category/categories.
Wrote 609 record(s) for 'Biological' → .../dataset_Biological.parquet
Wrote 507 record(s) for 'Glass' → .../dataset_Glass.parquet
...
Wrote 5016 record(s) for 'Sediment' → .../dataset_Sediment.parquet
Wrote 97 record(s) for 'Unclassified' → .../dataset_Unclassified.parquet
Done: 26 categor(y/ies) ingested (33616 spectra), 0 already up to date,
library at /home/yann/grassdata/hyperspeclib (format=parquet).

170 of the 33,786 catalogued spectra were skipped (their .asc file is not actually present in this download — a real, pre-existing gap in the archive, confirmed by checking each missing path directly, not a parsing failure). Re-running the same command afterward is fully idempotent (0 categor(y/ies) ingested ... 26 already up to date).

Querying across all three harvested sources at once

With i.hyper.lib_ecosis and i.hyper.lib_usgs having already populated the same library root, all three sources now live side by side under their own source_database= partition, queryable together in one scan:

import pyarrow.dataset as ds

dataset = ds.dataset("/home/yann/grassdata/hyperspeclib", format="parquet", partitioning="hive")
table = dataset.to_table(columns=["source_database"])
print(table.to_pandas()["source_database"].value_counts())
source_database
ecosis          229893
relab            33616
usgs_splib07      2457

228 total Parquet files, 2.5 GB, 265,966 spectra from three independently harvested sources, one unified schema.

SEE ALSO

i.hyper.lib_ecosis, i.hyper.lib_usgs, i.hyper.spectroscopy, i.hyper.endmembers

RELAB (Reflectance Experiment Laboratory), Brown University, operated for NASA's Keck/RELAB program.

GeoParquet specification: https://geoparquet.org

AUTHOR

Spectral Feature Extraction and Interpretation Engine