DESCRIPTION

i.hyper.speclookup is the query-side counterpart to the i.hyper.lib_* harvester family (i.hyper.lib_ecosis, i.hyper.lib_usgs, i.hyper.lib_relab): it reads the shared spectral library those modules build at $HOME/grassdata/hyperspeclib and answers two kinds of question against it, in one module because they share the same filtering and library-reading machinery:

  1. Metadata search — "what's in the library matching these criteria?" (source, category, free text, wavelength coverage, band count, geographic bounds). No query spectrum needed.
  2. Spectral similarity matching — "what in the library looks most like this spectrum?" — given one or more query spectra (from a vector map or CSV, in the same wl_<wavelength> column convention i.hyper.endmembers writes), ranks every matching library record by Spectral Angle Mapper, Pearson correlation, or Euclidean distance and returns the best top_n per query spectrum.

Both modes work identically regardless of whether the library was built with format=parquet or format=sqlite, and regardless of which harvester(s) populated it — format=auto (default) detects which backend is present.

Similarity matching can also be answered by an index server= instead of a local library tree, which is what makes matching against a quarter-million-record library practical (see Server mode below).

Filtering (applies to both modes)

All filters combine with AND. Parquet-backed filters (source_database, dataset_id, band count, bbox, free text) are pushed down into the scan itself (pyarrow.dataset predicate pushdown / SQLite WHERE); the wavelength-coverage filter is applied just after, per candidate row, since coverage is derived from the wavelengths array rather than a plain scalar column.

Metadata search mode (no query_vector/query_csv given)

Scans the library with the given filters, printing up to limit matches (source/category/record ID/title/band count) and, if output= is given, writing the same (lightweight, no spectra by default) rows to CSV or JSON. Add -s to include each matched record's full wavelengths/values arrays in the output file too.

Spectral similarity matching mode

Triggered by query_vector= (a GRASS point vector map, e.g. i.hyper.endmembers's vector output) and/or query_csv= (a CSV file, e.g. i.hyper.endmembers's CSV output) — both read wl_<wavelength> attribute/column names (the exact naming convention i.hyper.endmembers writes: wl_2265_19 for 2265.19, wl_neg_5_2 for -5.2) and treat each row/point as one independent query spectrum.

For every candidate library record (after the filters above), the record's own spectrum is linearly interpolated (numpy.interp) onto the query spectrum's wavelength grid, restricted to the wavelengths actually overlapping the candidate's own native range (no extrapolation) — candidates with fewer overlapping points than either min_overlap_bands (default 5, an absolute floor) or min_overlap_fraction (default 0, disabled) are skipped as not comparable. An absolute floor alone does not scale: 13 overlapping bands is a weak, edge-of-range match for a 255-band hyperspectral query (e.g. some USGS NIC4/Nicolet mineral records are only measured from ~2400nm onward, overlapping a typical VNIR/SWIR scene in just a narrow sliver at its very end) but would be unremarkable for a 20-band multispectral one — min_overlap_fraction expresses the requirement relative to the query's own band count instead (e.g. 0.15 = at least 15% of the query's bands must be covered), and both thresholds apply together when set. The resampled pair is then scored by similarity_method:

The top_n best-scoring candidates per query spectrum are printed and, if output= is given, written to CSV or JSON (one row per (query, rank) pair, with -s again controlling whether the matched spectra themselves are included).

Server mode

server= takes the base URL of a spectral index server (e.g. http://127.0.0.1:8977) and sends it each query spectrum, instead of reading and scoring library records locally. The server keeps the whole library resident as one L2-normalised float32 matrix and answers an exact nearest-neighbour search — not an approximate index — so a query costs tens of milliseconds whatever the library holds, where a local scan pays an interpolation per candidate record.

What changes in this mode:

Results carry the same column names as the local backend's, so anything reading this module's CSV/JSON (notably i.hyper.endmembers -i) works unchanged against either. record_id is the served library's own spectrum id and dataset_title its material name; organization and n_bands come back empty, as the index does not carry them.

Performance

The whole matching scan streams the library in batches (bounded memory regardless of how many candidates match, the same batching discipline the harvesters use when writing), so narrowing with source_database=/dataset_id= mainly saves time, not memory — matching one query spectrum against all ~2,450 USGS splib07a records takes a few seconds; matching against the full ~266,000-record library (no source/category filter) will scan proportionally longer. Filter to the source(s) most likely to contain what you're looking for when you can.

With server= that trade-off disappears: the search is a few whole-matrix products against a memory-mapped library, so the cost is set by the library's size rather than by how much of it the filters let through, and a filter saves nothing measurable. Measured against a 251,753-spectrum index on a 216-band 350-2500 nm grid, one query takes about 47 ms. On a real run, identifying all 20 endmembers of a 408-band drone cube cost about 5 s together with the N-FINDR extraction that produced them, against the minutes a local scan of the same library would need.

NOTES

query_vector and query_csv can both be given at once — their query spectra are simply concatenated (each still matched and ranked independently).

A CSV or vector map's non-wl_* columns (cat, x/y, lon/lat) are ignored for matching purposes; they exist only to identify each query point (used to label its result set) or for downstream reprojection in whichever module produced them.

EXAMPLES

Metadata search: free text, restricted to one source

i.hyper.speclookup search=Kaolinite source_database=usgs_splib07 limit=5
Library: /home/yann/grassdata/hyperspeclib (format=parquet)
Searching library metadata…
10 matching record(s) (limit=5).
[usgs_splib07/ChapterM_Minerals] splib07a_Antigorite_NMNH17958_BECKb_AREF: Minerals (476 bands, nm)
...

Note that search=Kaolinite also matches Antigorite here — USGS's own metadata classifies it under the "Kaolinite-Serpentine group" mineral family, and search= matches the full extra_metadata text, not just the primary sample name. Add dataset_id=ChapterM_Minerals to also restrict by chapter, or tighten search= to a more specific phrase if only the exact mineral name is wanted.

Metadata search with geographic + wavelength filters, JSON output

i.hyper.speclookup source_database=ecosis n=40 s=30 e=-100 w=-125 \
    min_wavelength=400 max_wavelength=2400 limit=5 \
    output=matches.json output_format=json
Library: /home/yann/grassdata/hyperspeclib (format=parquet)
Searching library metadata…
5 matching record(s) (limit=5).
[ecosis/0c23c6f5-6fdc-48a7-8977-40354c9f948d] 676479c0cc8e84e775975af7:
Fresh Leaf Spectra and Measured Traits from the Sierra Nevada (CA) in July 2023 (2151 bands, nm)
...
Wrote 5 record(s) → matches.json

Spectral similarity matching against a query CSV

Given a query CSV in i.hyper.endmembers's own wide format (first two columns are coordinates, the rest are wl_<wavelength> reflectance values — here built from a resampled real USGS Kaolinite spectrum, as a verification case):

i.hyper.speclookup query_csv=query_kaolinite.csv \
    source_database=usgs_splib07 dataset_id=ChapterM_Minerals top_n=3
Library: /home/yann/grassdata/hyperspeclib (format=parquet)
Matching 1 query spectrum/spectra against the library (method=sam)…
Scanned 1276 candidate record(s).
Top 3 match(es) for 'row1':
  1. [usgs_splib07/ChapterM_Minerals] splib07a_Kaolinite_CM5_BECKb_AREF (Minerals) -- sam=2e-06 (96 overlapping bands)
  2. [usgs_splib07/ChapterM_Minerals] splib07a_Kaolinite_CM3_BECKa_AREF (Minerals) -- sam=0.80745 (96 overlapping bands)
  3. [usgs_splib07/ChapterM_Minerals] splib07a_Kaolinite_GDS11_lt63um_BECKb_AREF (Minerals) -- sam=0.845591 (94 overlapping bands)

The query spectrum is (a coarsened resample of) Kaolinite_CM5's own real spectrum, and it correctly comes back as the top match at sam=2e-06 degrees (floating-point-level agreement) — with the next two matches being other real Kaolinite specimens in the library, confirming both the resampling/overlap logic and that SAM correctly favors genuinely similar mineralogy over unrelated samples.

Same query, against a vector map instead, with full-spectra CSV output

v.in.ascii input=query_kaolinite.txt output=query_pts \
    columns="x double precision, y double precision, wl_221_1 double precision, ..." \
    separator=pipe x=1 y=2
i.hyper.speclookup query_vector=query_pts source_database=usgs_splib07 \
    dataset_id=ChapterM_Minerals top_n=3 output=matches.csv -s
Matching 1 query spectrum/spectra against the library (method=sam)…
Scanned 1276 candidate record(s).
Top 3 match(es) for 'cat1':
  1. [usgs_splib07/ChapterM_Minerals] splib07a_Kaolinite_CM5_BECKb_AREF (Minerals) -- sam=2e-06 (96 overlapping bands)
  ...
Wrote 3 match(es) → matches.csv

matches.csv includes each match's rank, similarity score, full extra_metadata (mineral name, formula, locality, ...), and — because of -s — the matched record's own wavelengths/values arrays, ready for direct overplotting against the query spectrum.

Spectral similarity matching against an index server

Identify one spectrum against a whole served library, restricted to minerals, using continuum-removed SAM:

i.hyper.speclookup server=http://127.0.0.1:8977 \
    query_csv=endmembers.csv category=mineral \
    similarity_method=cr_sam top_n=5 \
    output=endmember_minerals.csv

Restricting the same query to one source, and keeping the reference-grade USGS library only:

i.hyper.speclookup server=http://127.0.0.1:8977 \
    query_csv=endmembers.csv source_database=usgs_splib07 \
    min_overlap_fraction=0.6 top_n=10

Comparing the similarity methods on the same query

i.hyper.speclookup query_csv=query_kaolinite.csv source_database=usgs_splib07 \
    dataset_id=ChapterM_Minerals top_n=1 similarity_method=correlation
i.hyper.speclookup query_csv=query_kaolinite.csv source_database=usgs_splib07 \
    dataset_id=ChapterM_Minerals top_n=1 similarity_method=euclidean
1. ... splib07a_Kaolinite_CM5_BECKb_AREF (Minerals) -- correlation=1.0 (96 overlapping bands)
1. ... splib07a_Kaolinite_CM5_BECKb_AREF (Minerals) -- euclidean=0.0 (96 overlapping bands)

All three metrics agree on the exact match, as expected; they differ in how sharply they penalize partial mismatches among the runners-up (SAM and correlation are shape-sensitive and albedo-tolerant, Euclidean is not).

SEE ALSO

i.hyper.lib_ecosis, i.hyper.lib_usgs, i.hyper.lib_relab, i.hyper.endmembers, i.hyper.spectroscopy

AUTHOR

Spectral Feature Extraction and Interpretation Engine