i.hyper.lib_relab is the third i.hyper.lib_* harvester
(after i.hyper.lib_ecosis and i.hyper.lib_usgs): it
parses a local download of the RELAB (Reflectance Experiment Laboratory)
spectral database, run by Brown University for NASA's Keck/RELAB program,
and writes it into the same shared, source-tagged database at
$HOME/grassdata/hyperspeclib, using the exact same row schema
as the other two harvesters — one pyarrow.dataset scan
spans EcoSIS field spectra, USGS lab reference spectra, and RELAB lab
reference spectra together.
RELAB's data release, like USGS's, has no live query API — it is a
static downloaded archive. input_dir must point at the root of that
download (e.g. ~/DBDATA/RelabDatabase2025Dec31), containing
its catalogues/ and data/ subdirectories as
distributed.
Sample_Catalogue
/ Spectra_Catalogue, needed to look up individual spectra
by sample) and, per this module's own ReadMeFirst.txt,
"a compressed zip file containing all the catalogues and data" as a
single combined download — that single zip is the simplest
option and is what this harvester expects.~/DBDATA/RelabDatabase<date>/ — it unpacks
directly into catalogues/ (the TSV catalogue files) and
data/ (one .asc/.txt pair per
spectrum, bucketed into data/<pi_code>/<sample_id
prefix>/) as siblings. This extracted root is what
input_dir must point at..asc files (not the equivalent
.txt files, which carry the same data in a different
unit/format) plus the small catalogue TSVs.RELAB's on-disk layout was studied via the reference parser in
$HOME/dev/spectral/spectral/database/relab.py (the
spectral Python package) and via the archive's own
ReadMeFirst.txt — not used as a dependency,
reimplemented here because RELAB's real local download turned out to be
richer and more structured than either the reference parser or
i.hyper.lib_usgs's per-sample-HTML approach: it ships two proper
relational catalogue tables (tab-separated, one header row each) instead
of requiring metadata to be scraped out of free text per spectrum.
catalogues/Sample_Catalogue.txt (~16,800 rows):
one row per physical sample — name, PI code, mineralogy
classification (GeneralType1/Type1/Type2/SubType),
particle size, texture, origin/locality, free-text description.catalogues/Spectra_Catalogue.txt (~33,800 rows):
one row per measured spectrum, foreign-keyed to
Sample_Catalogue by SampleID —
measurement date, instrument code (SpecCode), spectral
range, geometry angles, temperature, atmosphere, research context.catalogues/PI_Address.txt and
catalogues/MeasurementTypes.txt are small lookup
tables (PI code -> institution/full name; SpecCode ->
human-readable spectrometer name and measurement mode) joined in to
enrich organization and extra_metadata
respectively, rather than leaving raw codes as the only trace.data/<pi_code>/<sample_id prefix>/<SpectrumID>.asc,
lowercased (e.g. sample BF-BFJ-004, spectrum
C1BF04 -> data/bfj/bf/c1bf04.asc) — a
first line giving the number of data points, then that many
wavelength(nm) reflectance [stddev] lines
(space-delimited; the optional third stddev column, present in roughly
40% of files, is not carried into the common
wavelengths/values row schema), then
trailing free-text ignored by this module. Unlike USGS's
.txt files, RELAB's .asc wavelengths are
already in nanometers — no unit conversion needed.This catalogue-driven approach means no per-file HTML scraping is
needed at all: Spectra_Catalogue.txt is scanned once (a cheap
metadata-only pass, joined against the in-memory Sample_Catalogue
dict) to group every spectrum by category before touching a single
.asc file, then each category's actual spectra are parsed and
written in the same incremental, batched way as the other two
harvesters.
RELAB has no chapter-like folder structure at all (its data/
directory is bucketed by PI code + sample-ID prefix, an organizational
convenience with no semantic meaning) — so, analogous to
i.hyper.lib_usgs's use of USGS's chapters, this harvester groups
by Sample_Catalogue's GeneralType1 field
(Mineral, Rock, Sediment, Glass, Biological, Mixture, Organic, Regolith,
... — about 26 categories observed in practice, including a few
small/idiosyncratic ones and an Unclassified bucket for
samples with no GeneralType1 recorded), sanitized into a
filesystem-safe dataset_id (e.g. Chemical Compound
-> Chemical_Compound) while keeping the original text as
dataset_title. general_types= restricts ingestion to
specific categories (matched against the sanitized form; default:
all).
Same as i.hyper.lib_usgs: RELAB lab samples have no coordinates
(a free-text Location/Origin field exists for
some, not structured lat/lon), so longitude/latitude
are always null for this source. And unlike EcoSIS, there is no confirmed
single per-record web page or DOI to cite in the local download itself, so
source_url/source_api_url are left null rather than guessed
— per-record traceability instead comes entirely from
extra_metadata's local_spectrum_file path plus
the SpectrumID/SampleID identifiers, which any
RELAB catalogue search can resolve back to the original record.
Same conventions as the other two harvesters: a shared
_manifest.json at the database root (keyed
relab:<category>) tracks each category's source
spectrum count (expected_total) to decide whether a re-run
can skip it (-f to force), and records are batched
(stream_batch_size, default 2000) and written incrementally via the
same ParquetWriter/executemany() per-batch
pattern.
Use -k to keep going past a spectrum that raises a parse error
instead of aborting the whole run. A spectrum whose catalogued
.asc file is simply missing on disk (a real, if
uncommon, gap in the archive itself — confirmed present in this
session's actual download: 170 of 33,786 catalogued spectra) is always
skipped silently regardless of -k, since that is an expected, not
exceptional, condition.
i.hyper.lib_relab input_dir=$HOME/DBDATA/RelabDatabase2025Dec31 \
output=/tmp/test_relab_lib format=parquet general_types=Coating
Loading RELAB catalogues (Sample_Catalogue, PI_Address, MeasurementTypes)… 16786 samples in Sample_Catalogue. Scanning Spectra_Catalogue… Found 10 spectra across 1 category/categories. Wrote 10 record(s) for 'Coating' → /tmp/test_relab_lib/source_database=relab/dataset_Coating.parquet Done: 1 categor(y/ies) ingested (10 spectra), 0 already up to date, library at /tmp/test_relab_lib (format=parquet).
One of those 10 rows, inspected directly:
record_id: BIR1LH053 dataset_title: Coating organization: Department of Geological Sciences, Brown University n_bands: 3527 wavelengths[:3]: [1428.4, 1428.8, 1429.2] (nm -- already native, no conversion) extra_metadata: sample_id: LH-JFM-053 sample_name: SB-01 chips general_type1: Coating type1: Iron Oxide sub_type: Jarosite texture: Chip origin: Rio Tinto, Spain spectrometer_name: Thermo Nexus 870 with PIKE AutoDiff measurement_mode: Reflectance (Off-axis biconical FT-IR) spectra_catalogue_speccode: BCF-FTIR2 spectra_catalogue_researchtype: Astrobiology, Student local_spectrum_file: .../data/jfm/lh/bir1lh053.asc
i.hyper.lib_relab input_dir=$HOME/DBDATA/RelabDatabase2025Dec31
Loading RELAB catalogues (Sample_Catalogue, PI_Address, MeasurementTypes)… 16786 samples in Sample_Catalogue. Scanning Spectra_Catalogue… Found 33786 spectra across 26 category/categories. Wrote 609 record(s) for 'Biological' → .../dataset_Biological.parquet Wrote 507 record(s) for 'Glass' → .../dataset_Glass.parquet ... Wrote 5016 record(s) for 'Sediment' → .../dataset_Sediment.parquet Wrote 97 record(s) for 'Unclassified' → .../dataset_Unclassified.parquet Done: 26 categor(y/ies) ingested (33616 spectra), 0 already up to date, library at /home/yann/grassdata/hyperspeclib (format=parquet).
170 of the 33,786 catalogued spectra were skipped (their
.asc file is not actually present in this download — a
real, pre-existing gap in the archive, confirmed by checking each missing
path directly, not a parsing failure). Re-running the same command
afterward is fully idempotent (0 categor(y/ies) ingested ... 26
already up to date).
With i.hyper.lib_ecosis and i.hyper.lib_usgs having
already populated the same library root, all three sources now live side
by side under their own source_database= partition,
queryable together in one scan:
import pyarrow.dataset as ds
dataset = ds.dataset("/home/yann/grassdata/hyperspeclib", format="parquet", partitioning="hive")
table = dataset.to_table(columns=["source_database"])
print(table.to_pandas()["source_database"].value_counts())
source_database ecosis 229893 relab 33616 usgs_splib07 2457
228 total Parquet files, 2.5 GB, 265,966 spectra from three independently harvested sources, one unified schema.
RELAB (Reflectance Experiment Laboratory), Brown University, operated for NASA's Keck/RELAB program.
GeoParquet specification: https://geoparquet.org