DESCRIPTION

i.hyper.lib_gsti is the fourth i.hyper.lib_* harvester (after i.hyper.lib_ecosis, i.hyper.lib_usgs and i.hyper.lib_relab): it parses a local clone/download of the Global Spectra-Trait Initiative (GSTI) database – a collaborative repository of paired leaf hyperspectral reflectance and gas-exchange/photosynthetic-capacity trait measurements (Lamour et al., 2026, Earth System Science Data, doi:10.5194/essd-18-245-2026) – and writes it into the same shared, source-tagged database at $HOME/grassdata/hyperspeclib, using the exact same row schema as the other three harvesters – one pyarrow.dataset scan spans EcoSIS field spectra, USGS lab reference spectra, RELAB lab reference spectra and GSTI paired leaf spectra/traits together.

GSTI has no live query API – it is developed on GitHub and published as a static archive. input_dir must point at the root of a local clone/ download of that repository (e.g. ~/DBDATA/GlobalSpectraTraitInitiative/gsti), containing its Database/ subdirectory as distributed.

Downloading the data

  1. Clone the GSTI GitHub repository:

    git clone https://github.com/plantphys/gsti.git

    or download the archived release from ESS-DIVE: doi:10.15485/2530733 (Lamour et al., 2025).

  2. The curated, standardized database lives at Database/Database_p1.csv, Database_p2.csv and Database_p3.csv (split into three parts only to stay under GitHub’s file size limit; all three share the same 2197-column header). This directory is what input_dir, one level up, must resolve to (i.e. point input_dir at the clone root, not at Database/ itself).

  3. The repository also ships Datasets/ (raw, per-contributor-study data as originally submitted) and PLSR//R/ (model code) – this harvester only reads the curated Database/ CSVs, not the raw per-study data.

Row schema

Each Database_p*.csv row is one paired leaf observation: 46 metadata columns (site, species, phenology, plant type, photosynthetic pathway, soil, leaf traits – LMA, Narea, Nmass, Parea, Pmass, LWC – fitted photosynthetic parameters – Vcmax25, Jmax25, TPU25, Rday25 and their standard errors – fitting/instrument metadata, lat/lon/elevation/biome, authors, DOIs) followed by 2151 Wave_<nm> columns spanning 350-2500 nm reflectance.

Incremental re-runs and streaming

Same conventions as the other three harvesters: a shared _manifest.json at the database root (keyed gsti:<dataset_id>) tracks each dataset’s source record count (expected_total) to decide whether a re-run can skip it (-f to force), and records are batched (stream_batch_size, default 2000) and written incrementally via the same ParquetWriter/executemany() per-batch pattern.

NOTES

Use -k to keep going past a record that raises a build error instead of aborting the whole run. A record with no SampleID or with zero non-missing reflectance bands is always skipped silently regardless of -k, since that is an expected, not exceptional, condition.

GSTI's own missing-value convention is inconsistent across columns: most numeric trait columns use NA, but Dataset_DOI/Publication_DOI (and some other text columns) use a bare - instead. Both are treated as missing throughout this harvester.

EXAMPLES

Ingest a single (small) dataset, GeoParquet backend

i.hyper.lib_gsti input_dir=$HOME/DBDATA/GlobalSpectraTraitInitiative/gsti \
    output=/tmp/test_gsti_lib format=parquet datasets=Croft_2021
Scanning GSTI Database_p1/p2/p3.csv…
Found 24 observations across 1 dataset(s).
Wrote 24 record(s) for 'Croft_2021' →
/tmp/test_gsti_lib/source_database=gsti/dataset_Croft_2021.parquet
Done: 1 dataset(s) ingested (24 records), 0 already up to date, library at
/tmp/test_gsti_lib (format=parquet).

Full library: all datasets into the shared database

i.hyper.lib_gsti input_dir=$HOME/DBDATA/GlobalSpectraTraitInitiative/gsti
Scanning GSTI Database_p1/p2/p3.csv…
Found 7561 observations across 37 dataset(s).
Wrote 1285 record(s) for 'Coast_et_al_2019' → .../dataset_Coast_et_al_2019.parquet
Wrote 837 record(s) for 'Ojo_et_al_2024' → .../dataset_Ojo_et_al_2024.parquet
...
Done: 37 dataset(s) ingested (7561 records), 0 already up to date, library
at /home/yann/grassdata/hyperspeclib (format=parquet).

Re-running the same command afterward is fully idempotent (0 dataset(s) ingested ... 37 already up to date).

Querying across all four harvested sources at once

With i.hyper.lib_ecosis, i.hyper.lib_usgs and i.hyper.lib_relab having already populated the same library root, all four sources live side by side under their own source_database= partition, queryable together in one scan:

import pyarrow.dataset as ds

dataset = ds.dataset("/home/yann/grassdata/hyperspeclib", format="parquet", partitioning="hive")
table = dataset.to_table(columns=["source_database"])
print(table.to_pandas()["source_database"].value_counts())

SEE ALSO

i.hyper.lib_ecosis, i.hyper.lib_relab, i.hyper.lib_usgs, i.hyper.spectroscopy, i.hyper.endmembers

Global Spectra-Trait Initiative (GSTI): github.com/plantphys/gsti

Lamour, J. et al. (2026). The Global Spectra-Trait Initiative: A database of paired leaf spectroscopy and functional traits associated with leaf photosynthetic capacity. Earth System Science Data, 18(1), 245-265. doi:10.5194/essd-18-245-2026

GeoParquet specification: https://geoparquet.org

AUTHOR

Spectral Feature Extraction and Interpretation Engine