DESCRIPTION

i.hyper.endmembers identifies the most spectrally "pure"/extreme pixels (endmembers) in a hyperspectral 3D raster cube — the classic first step of a linear spectral unmixing workflow — and reports their locations and spectra. Unlike i.pysptools.unmix, which works on a 2D imagery group, this module reads directly from a GRASS 3D raster (as produced by i.hyper.import), and follows the same band-metadata and region-handling conventions as the rest of the i.hyper module family.

Endmember extraction needs the whole scene loaded at once (unlike per-pixel classification, it searches for extremes across the full image), so processing is not chunked; keep the computational region to your actual area of interest, or sample it with sample_size (below).

Extraction methods

PPI and FIPPI runtime scales with ppi_skewers and the number of valid pixels. Lower ppi_skewers for a quicker, less statistically robust estimate. (This is dominated by dense matrix multiplication, so it also benefits substantially from an optimized BLAS — e.g. OpenBLAS rather than the plain reference BLAS some distributions ship by default.)

Outputs

At least one of output / output_file is required.

Unmixing methods

Band metadata and compatibility

Band wavelength/FWHM/validity metadata is read the same way as i.hyper.spectroscopy: from i.hyper.import's grid3/<map>/hyper.json sidecar if present, else r3.info -h history, else per-band r.support metadata. Use -b to restrict processing to bands marked valid (valid=1) in that metadata, and min_wavelength/ max_wavelength to restrict to a spectral sub-range (e.g. to exclude noisy detector edges or strong water-vapor absorption windows), consistent with i.hyper.geology's convention.

Automatic exclusion of out-of-range bands (max_invalid_fraction)

-b/min_wavelength/max_wavelength rely on the cube's own metadata (or the caller) already knowing which bands are unreliable. That is not always true: a real EMIT scene's per-band valid=1 metadata did not flag two bands at ~1357nm and ~1417nm (classic strong water-vapor-absorption wavelengths) even though 61.7% and 28.1% of valid pixels there held a physically impossible out-of-range (outside 0.0-1.0) reflectance value — a sensor/atmospheric-correction artifact, not a measurement.

This matters more for extraction than it might seem: ATGP starts from the pixel with the single largest spectral vector norm, and N-FINDR/PPI/FIPPI all likewise select or refine endmembers by norm or simplex volume. An out-of-range value in even one band inflates that pixel's norm and can change which pixel looks most "extreme" — and because endmember algorithms specifically hunt for rare/extreme pixels, even a low-frequency artifact (one real case found here: just 0.14% of pixels in a third band) is disproportionately likely to land on exactly the pixels selected as endmembers, not average out harmlessly. Verified directly on this scene: excluding the two worst bands changed 2 of 6 ATGP-selected pixels, including the very first (max-norm) pick.

max_invalid_fraction (default 0.0) guards against this independent of -b/wavelength range/upstream metadata: after loading the cube, any band where more than this fraction of valid pixels has a value outside 0.0-1.0 is excluded entirely — from extraction, identification, and plotting — with a warning naming the wavelength and the fraction affected. The default of 0 (exclude on even a single contaminated pixel) is deliberately strict, for the reason above; raise it (e.g. to 0.01) if your sensor is known to have harmless, genuinely sparse per-pixel noise unrelated to a systematic band-level artifact. Set to 1.0 to disable entirely.

Storage scale (reflectance_scale)

Every 0.0-1.0 test in this module — max_invalid_fraction above, and the identification query sent to i.hyper.speclookup, which drops any value outside that range — assumes the cube holds reflectance as a fraction. Surface-reflectance products commonly ship scaled by 1000 or 10000 instead. Extraction itself does not care, since a common factor cancels out of every norm and simplex volume, but those two tests do: against a cube stored x10000 every band looks entirely out of range and the module stops with "All bands were excluded by max_invalid_fraction".

reflectance_scale (default 0) is the divisor that converts stored values to reflectance. At 0 it is detected from the cube's own value range and reported, using the same detector and the same thresholds as i.hyper.spectroscopy's option of the same name, so one cube resolves to the same scale in both modules. Give it explicitly to override a wrong detection.

Sampling instead of the whole region (sample_size)

By default the module loads every pixel of the computational region. It does that with r3.to.rast, one 2D raster per band, which reads the whole scene once per band. Measured on the C2RD Pourrieres drone mosaic (8471x7284x408): a single band slice takes 10-15 s, and loading the cube through all 408 of them took 1 h 24 min — at a 0.3 m region, already decimated tenfold, because at the native 0.03 m grid the loaded cube would be 61.7 M pixels x 408 bands, about 100 GB as float32.

sample_size=n instead reads n randomly chosen whole spectra with r3.what, which fetches only the voxels asked for, with no per-band pass over the scene and memory of only n x bands x 4 bytes. On the same cube, 50000 sampled spectra at native 0.03 m took 367 s, against the 1 h 24 min that loading the same cube took at a tenth of that resolution, and the whole run — sampling, N-FINDR for 20 endmembers and 20 identification queries — took 6 min 12 s. The sampled cells are drawn without replacement from the region's own cell grid and depend on random_seed, so a run is reproducible; coordinates and spectra of the selected endmembers are identical to what the whole-region path reports for the same pixel.

The cost is set by the cube's tile geometry, not by the band count. That mosaic is tiled 31x26x10 FCELL, so one 408-band spectrum spans 41 tiles and pulls about 1.3 MB to deliver 408 values. Scattered points share almost no tiles — 50000 points over 61.7 M pixels is one per 1234, where a tile column covers 806 — so the per-point cost is flat in the sample size and the reads add up to several times the size of the cube. Measured on this cube:

5000 points scattered over the scene   39.0 s   7.80 ms/point
5000 points inside one 300x300 block    2.8 s   0.57 ms/point

Locality is therefore worth a factor of 14, and a sample restricted to a smaller region costs far less per spectrum than a sample spread over a large one. Enlarging the raster3d tile cache (RASTER3D_DEFAULT_CACHE_SIZE/RASTER3D_MAX_CACHE_SIZE) does not help the scattered case, where the misses are compulsory rather than capacity misses: it measured slightly slower (45.5 s against 39.0 s), while on the clustered case it gave 1.5 s against 2.8 s.

What a sample costs is coverage, not accuracy. Endmember extraction hunts for the few most extreme spectra in the scene, so a material covering a thousandth of the region has about a 1 in 1000 chance per sampled pixel of being seen at all: if a known material is missing from the result, raise sample_size rather than lowering it. Sampling is refused together with abundance_prefix, since unmixing writes an abundance for every pixel of the region and so needs the whole cube.

A sample whose size reaches the region's pixel count degenerates to querying every pixel one by one, which is slower than the whole-region path; the module warns and carries on.

Cross-checking purity against local variability (homogeneity)

Extraction finds the most extreme spectra in the scene, which is not the same as the most pure. A pixel straddling two materials, or sitting on a shadow edge, is extreme in exactly the way N-FINDR and ATGP reward, and the spectrum it yields is a mixture no library will match well.

homogeneity=raster samples a 2D raster of local spectral variability at each selected pixel and attaches the value to every output as a homogeneity column (vector) or field (CSV/JSON), reporting it alongside the raster's own region mean:

Local variability from <mosaic_sam5> (region mean 0.305):
Endmember 1: homogeneity=0.012 (0.04x region mean)
Endmember 2: homogeneity=0.596 (1.95x region mean)

An endmember well below the region mean sits in a uniform neighbourhood and its spectrum is a plausible pure material; one well above it should be treated as a mixture or an edge, whatever its library match says.

Any 2D raster is accepted and the value is attached exactly as stored: this module does not know which way round the given measure runs. A variance, standard deviation or mean-SAM map from i.variance.opencl measures variability, so there the low values mark the homogeneous pixels, as in the example above. A null at an endmember location is reported as null and written as an empty field, never as zero. The raster is checked to exist before the cube is read, so a typo costs nothing.

Identification against the shared spectral library (-i)

-i identifies every extracted endmember against the shared local spectral library built by i.hyper.lib_ecosis/i.hyper.lib_usgs/ i.hyper.lib_relab, by calling i.hyper.speclookup directly — one real GRASS module invocation covering all endmembers at once (each endmember becomes one query row in a temporary CSV, and i.hyper.speclookup already ranks one result set per row), not a reimplementation of its matching logic. This is the natural second step after extraction: N-FINDR/ATGP/PPI/FIPPI tell you where the spectrally distinct materials in a scene are; -i gives each one a candidate real-world identity.

Before being written to that query CSV, each endmember's own values are checked against the physical reflectance range: any value outside 0.0-1.0 (e.g. a water-vapor-absorption-band retrieval artifact — the same kind of spike the plot's fixed y-axis clips from view) is left blank in that row rather than passed through as-is or written as a literal nan. A blank CSV cell is skipped cleanly by i.hyper.speclookup's own reader (float('') raises, same as any other unparseable value), simply dropping that one band from that endmember's query — whereas writing the literal text nan would not be caught (Python's float() parses the strings "nan"/"inf" successfully instead of raising, the same pitfall already fixed in the harvesters' own ingestion), and passing the raw out-of-range value through uncorrected would silently distort the SAM/correlation/Euclidean computation for every candidate compared against that endmember. This is not a cosmetic fix: on a real EMIT scene, a handful of such artifact values changed which library record several endmembers matched best, in one case improving the best match from sam=55.7 (a poor fit) to sam=9.4 (a much better one) once the corrupting values were excluded from the comparison.

Each identified endmember is labeled with the best match's source database, dataset, record ID, title, organization, the similarity score (accuracy) and number of overlapping bands used to compute it, and the confidence margin to the runner-up (precision) — attached to the vector map's attribute table (match_source, match_dataset, match_record, match_title, match_organization, match_score, match_overlap_bands, match_margin) and, more fully, to output_file (the same fields plus match_method and the matched record's complete match_extra_metadata JSON — mineral/plant name, formula, collection locality, and everything else the source harvester captured). An endmember with no library candidate above spec_min_overlap_bands gets a blank/null match rather than a fabricated one.

Plotting (-p / -w)

-p renders every extracted endmember's spectrum as a matplotlib PNG (solid line, one color per endmember). The y-axis is always fixed to 0.0-1.0, the physical range of a reflectance value — any excursion outside it in the underlying data (e.g. a water-vapor-absorption-band retrieval artifact) is a known data-quality issue and is simply clipped from view rather than being allowed to rescale the whole plot. The x-axis is clipped to the endmember's own wavelength range, even when a matched library record natively covers a much wider range — e.g. a Nicolet FTIR reference extending to 200,000+ nm would otherwise squeeze the whole comparison into an unreadable sliver. If -i was also given, each endmember's identified library match is overplotted in the same color as a dashed line, and the legend names the match's title, record ID, source, and similarity score alongside each endmember.

The PNG is written to plot_dir (default: the current GRASS location's own directory, $GISDBASE/$LOCATION_NAME/ — so it lands alongside whatever project the input raster belongs to without needing to specify one explicitly) as <output>_endmembers.png, or <input>_endmembers.png if no vector output was requested. Add -w to also open it in an interactive window (needs a display; the file is still written either way). -p alone (without output/output_file) is a valid, plot-only invocation.

NOTES

Column/field names are derived from each band's central wavelength, in wavelength_unit (nm by default), sanitized into valid SQL identifiers (e.g. 2265.19 nm becomes wl_2265_19).

Endmember coordinates are cell centers of whichever pixel was selected, computed directly from the active computational region — they are exact regardless of the region's resolution or extent relative to the raster's native footprint.

EXAMPLES

The examples below use an EMIT scene imported with i.hyper.import into its own location, emit_slovakia (raster_3d map emit_20240730, 244 bands, hyper.json sidecar metadata). Its native footprint covers nearly all of Slovakia (1727 x 2746 pixels, north=49.10 south=48.17 east=20.53 west=19.04) — endmember extraction needs the whole active region loaded into memory at once (see above), so the first step in any real session is always to restrict the region to the actual area of interest, not run against the full native extent:

g.region n=48.70 s=48.60 e=19.80 w=19.70 res=0:00:01.952037
g.region -g
projection=3
zone=0
n=48.7
s=48.6
w=19.7
e=19.8
nsres=0.000543478260869573
ewres=0.000543478260869573
rows=184
cols=184
cells=33856

Example 1: N-FINDR (default), vector map + CSV with longitude/latitude

i.hyper.endmembers input=emit_20240730 output=slovakia_endmembers \
    output_file=slovakia_endmembers.csv -l n_endmembers=6 \
    extraction_method=NFINDR
Extracting 244 bands (184x184 pixels)…
Extracting 6 endmembers using NFINDR…
Writing endmember vector map → slovakia_endmembers…
Writing endmember spectra (csv) → slovakia_endmembers.csv…
Done: 6 endmember(s) extracted with NFINDR.

The vector map has one point per endmember and one attribute column per band, named by its central wavelength:

v.info -c map=slovakia_endmembers
INTEGER|cat
DOUBLE PRECISION|x
DOUBLE PRECISION|y
DOUBLE PRECISION|wl_381_0056
DOUBLE PRECISION|wl_388_4092
DOUBLE PRECISION|wl_395_8158
...

And the CSV (-l puts longitude/latitude first, reprojected via m.proj from this location's native CRS — here already geographic, so the values are unchanged, but the same flag reprojects correctly from a projected/UTM location too):

lon,lat,wl_381_0056,wl_388_4092,wl_395_8158,wl_403_2254,...
19.70027174,48.62038043,0.00179211446,0.00183079974,0.00186951912,0.00190832582,...

Example 2: PPI + FCLS abundance mapping over the whole region

i.hyper.endmembers input=emit_20240730 output=slovakia_ppi \
    n_endmembers=5 extraction_method=PPI \
    abundance_prefix=slovakia_abund unmixing_method=FCLS random_seed=42
Extracting 244 bands (184x184 pixels)…
Extracting 5 endmembers using PPI…
Writing endmember vector map → slovakia_ppi…
Unmixing abundances using FCLS…
Done: 5 endmember(s) extracted with PPI.

This writes slovakia_abund_1 .. slovakia_abund_5, one abundance raster per endmember, each valued 0-1 and summing to 1 across the five rasters at every pixel (FCLS's non-negativity + sum-to-one constraints):

r.univar map=slovakia_abund_1 -g
n=33856
min=0
max=1
mean=0.0485496587901701

Example 3: restrict to valid bands in a wavelength range, export JSON

i.hyper.endmembers input=emit_20240730 output=slovakia_endmembers_b \
    output_file=slovakia_endmembers.json output_format=json \
    n_endmembers=8 extraction_method=ATGP \
    -b min_wavelength=450 max_wavelength=2400
Extracting 221 bands (184x184 pixels)…
Extracting 8 endmembers using ATGP…
Writing endmember vector map → slovakia_endmembers_b…
Writing endmember spectra (json) → slovakia_endmembers.json…
Done: 8 endmember(s) extracted with ATGP.

221 of the cube's 244 bands fall in the requested 450-2400nm range; no -l here, so the JSON's first two fields are the native projected X,Y (this location happens to be geographic, so numerically these look like lon/lat too, but for a projected/UTM location they would not be):

[
  {
    "x": 19.78233695652174,
    "y": 48.674728260869564,
    "wl_455_1703": 0.0431,
    "wl_462_5989": 0.0448,
    ...
  },
  ...
]

Example 4: identify each endmember against the shared spectral library

i.hyper.endmembers input=emit_20240730 output=slovakia_endmembers_id \
    output_file=slovakia_endmembers_id.csv -l -i n_endmembers=6 \
    extraction_method=NFINDR spec_source_database=usgs_splib07 \
    spec_dataset_id=ChapterV_Vegetation,ChapterM_Minerals,ChapterS_SoilsAndMixtures
Extracting 244 bands (184x184 pixels)…
Extracting 6 endmembers using NFINDR…
Identifying endmembers against the shared spectral library (i.hyper.speclookup)…
Identified 6 of 6 endmember(s) (method=sam).
Endmember 1: Vegetation (splib07a_LeafySpurge_Spurge-A2-Jun98_ASDFRa_AREF, usgs_splib07) -- sam=5.18419, margin=0.02959 (240 overlapping bands)
Endmember 2: Vegetation (splib07a_Marsh_SCAM50%..._CRMS121v32_ASDFRa_AREF, usgs_splib07) -- sam=3.71498, margin=0.2749 (234 overlapping bands)
Endmember 3: Vegetation (splib07a_LeafySpurge_Spurge-A1-Oct97_ASDFRa_AREF, usgs_splib07) -- sam=3.4407, margin=0.03746 (240 overlapping bands)
Endmember 4: Vegetation (splib07a_Marsh_SCAM50%..._CRMS121v32_ASDFRa_AREF, usgs_splib07) -- sam=1.37321, margin=0.1085 (234 overlapping bands)
Endmember 5: Vegetation (splib07a_Oak_QUDU_CA01-QUDU-1_bush_1_ASDFRa_AREF, usgs_splib07) -- sam=1.7603, margin=0.1107 (236 overlapping bands)
Endmember 6: Vegetation (splib07a_Grass-Smoothbrome_YNP-SB-1_AVIRISb_RTGC, usgs_splib07) -- sam=3.79253, margin=0.2401 (237 overlapping bands)
Writing endmember vector map → slovakia_endmembers_id…
Writing endmember spectra (csv) → slovakia_endmembers_id.csv…
Done: 6 endmember(s) extracted with NFINDR.

Genuinely sensible for this Slovakia forest scene: every endmember matches best against USGS's Vegetation chapter, not Minerals or Soils, despite all three being searched. The vector map's attribute table carries the identity and confidence fields directly:

v.db.select map=slovakia_endmembers_id columns=cat,match_title,match_record,match_score,match_overlap_bands,match_margin
cat|match_title|match_record|match_score|match_overlap_bands|match_margin
1|Vegetation|splib07a_LeafySpurge_Spurge-A2-Jun98_ASDFRa_AREF|5.184195|240|0.029587
2|Vegetation|splib07a_Marsh_SCAM50%..._CRMS121v32_ASDFRa_AREF|3.714981|234|0.274929
...

and the CSV additionally carries the full match_extra_metadata for the top match, e.g. for endmember 1:

{"documentation_format": "PLANT", "sample_id": "Spurge-A2-Jun98",
 "plant_type": "Invasive herbaceous perennial weed", "plant": "Leafy spurge",
 "latin_name": "Euphorbia esula L.", "collection_locality": "Golden, Colorado", ...}

Example 5: identify and plot, on a real EMIT scene (emit_dubai)

Using the emit_dubai project's own AlMarwanProjectSite saved region and its emit_20260623 cube (255 bands):

g.region region=AlMarwanProjectSite
i.hyper.endmembers input=emit_20260623 output=dubai_endmembers \
    -i -p -b n_endmembers=6 extraction_method=NFINDR \
    spec_source_database=usgs_splib07
Extracting 255 bands (350x706 pixels)…
WARNING: Excluding band at 1357.2nm from endmember extraction: 61.7% of
         valid pixels have an out-of-range (outside 0.0-1.0) reflectance
         value (max_invalid_fraction=0).
WARNING: Excluding band at 1416.8nm from endmember extraction: 28.1% of
         valid pixels have an out-of-range (outside 0.0-1.0) reflectance
         value (max_invalid_fraction=0).
WARNING: Excluding band at 1439.1nm from endmember extraction: 0.1% of
         valid pixels have an out-of-range (outside 0.0-1.0) reflectance
         value (max_invalid_fraction=0).
Extracting 6 endmembers using NFINDR…
Identifying endmembers against the shared spectral library (i.hyper.speclookup)…
Identified 6 of 6 endmember(s) (method=sam).
Endmember 1: Vegetation (splib07a_Rangeland_L02-045_S01%_G04%_ASDFRa_AREF, usgs_splib07) -- sam=4.0643, margin=0.06403 (247 overlapping bands)
Endmember 2: Organic Compounds (splib07a_Xanthine_SA-X0626_90K_NIC4aa_RREF, usgs_splib07) -- sam=11.6209, margin=0.5861 (152 overlapping bands)
Endmember 3: Liquids (splib07a_Water+Montmor_SWy-2+5.01g-l_ASDFRa_AREF, usgs_splib07) -- sam=12.8549, margin=3.96 (252 overlapping bands)
Endmember 4: Vegetation (splib07a_Marsh_sediment_DWV3-0511_dry_ASDFRa_AREF, usgs_splib07) -- sam=5.36816, margin=0.4425 (242 overlapping bands)
Endmember 5: Organic Compounds (splib07a_Peptidogylcan_SA-69554_ASDFRa_AREF, usgs_splib07) -- sam=5.96867, margin=0.3165 (252 overlapping bands)
Endmember 6: Minerals (splib07a_Vesuvianite_HS446.1B_Idocras_ASDFRa_AREF, usgs_splib07) -- sam=2.86591, margin=0.01122 (252 overlapping bands)
Wrote endmember spectra plot → /home/yann/grassdata/emit_dubai/dubai_endmembers_spectra.png
Wrote reference spectra plot → /home/yann/grassdata/emit_dubai/dubai_endmembers_reference_spectra.png
Writing endmember vector map → dubai_endmembers…
Done: 6 endmember(s) extracted with NFINDR.

No output_file/plot_dir given here, so both PNGs land in the project's own directory automatically ($GISDBASE/$LOCATION_NAME/ = ~/grassdata/emit_dubai/), named entirely by the module from the vector output name. dubai_endmembers_spectra.png shows the six extracted endmembers together, numbered; dubai_endmembers_reference_spectra.png shows their six identified USGS matches together, same numbers/colors, each labeled with its full identity and score.

This run also demonstrates why max_invalid_fraction's strict default (0) matters at the extraction stage, not just identification: with the two worst contaminated bands (and one marginal one, at just 0.14% of pixels) excluded before ATGP/N-FINDR ever run, different pixels get selected as endmembers than in an earlier run where those bands were left in — confirmed directly by comparing the raw ATGP pixel indices with and without the bands excluded (2 of 6 changed, including the very first max-norm pick). The endmember spectra plot is also visibly cleaner: no more multiple-hundred-percent reflectance spike near 1350-1450nm, since the pixels now selected were never chosen partly because of that artifact in the first place.

Example 6: sample a large drone cube instead of loading it

The C2RD Pourrieres mosaic is 8471x7284x408 at 0.03 m; the whole-region path measured 1 h 24 min of band extraction even at a decimated 0.3 m region, and would need about 100 GB of memory at the native one. Sampling 50000 whole spectra reads the same cube at full resolution in 367 s:

g.region raster_3d=mosaic_final
i.hyper.endmembers input=mosaic_final output=c2rd_endmembers \
    output_file=c2rd_endmembers.csv n_endmembers=20 \
    extraction_method=NFINDR sample_size=50000 random_seed=42 \
    max_invalid_fraction=0.005 reflectance_scale=10000 \
    spec_server=http://127.0.0.1:8977 spec_top_n=5 -i -b
Sampling 50000 random pixels of 7284x8471 (408 bands)…
Sampled 27752 valid of 50000 queried pixels (408 bands).
Reflectance scale detected: 10000 (stored value / 10000 = reflectance)
WARNING: Dropping 238 of 27752 valid pixels that are null in at least one
         band: the extraction algorithms cannot take a null. Bands with the
         most nulls: 1252.3nm (0.33%), 1247.2nm (0.33%), 1257.4nm (0.33%)
Extracting 20 endmembers using NFINDR…
Identified 20 of 20 endmember(s) (method=sam).

About 41% of that mosaic's bounding box is nodata, which is why 50000 queried pixels yield 27752 spectra. The run took 6 min 12 s in total, of which 367 s was the sampling itself.

Spatial variability from i.variance.opencl cross-checks the result, fed straight into homogeneity so that each endmember carries it:

i.variance.opencl input_3d=mosaic_final method=sam_mean size=5 \
    output=mosaic_sam5
i.hyper.endmembers input=mosaic_final output=c2rd_endmembers \
    n_endmembers=20 sample_size=50000 random_seed=42 \
    reflectance_scale=10000 homogeneity=mosaic_sam5

An endmember whose homogeneity is far above the region mean sits in a heterogeneous neighbourhood and is a mixed pixel or an edge, not a pure material.

SEE ALSO

i.hyper.speclookup, i.hyper.lib_ecosis, i.hyper.lib_usgs, i.hyper.lib_relab, i.hyper.spectroscopy, i.hyper.import, i.hyper.geology, i.pysptools.unmix, i.variance.opencl, r.what, r3.to.rast, r3.what, m.proj

AUTHOR

Spectral Feature Extraction and Interpretation Engine