i.sam.opencl segments a 3-band imagery group with the Segment
Anything Model (SAM) and writes the segments as a clean vector area map.
The whole network, image encoder and mask decoder, runs on an OpenCL
device (GPU or CPU), with weights read directly from the original PyTorch
checkpoint file: neither Python nor PyTorch is needed at run time.
The module implements SAM's automatic mask generation: the model is
prompted with a regular grid of points, and the masks it returns are kept
according to their predicted quality and stability, de-duplicated and
cleaned. The overlapping masks are then turned into a non-overlapping,
topologically clean vector map with one attribute row per segment.
The group (or its subgroup) must contain exactly three
raster maps, which are used as the red, green and blue channels in the
order they are listed in the group (see i.group). Each band is
stretched linearly between its lower and upper percentile to the
0-255 range SAM was trained on. Cells that are NULL in any band are not
segmented. Processing is done in the current computational region.
The weights are too large to be installed with the module. Without a
checkpoint file, the module looks for one in the weights folder:
weights_dir, by default the central folder
$SAM_MODELS_DIR/sam1, or ~/models/sam/sam1 when
the environment variable SAM_MODELS_DIR is not set. If the
model is not there, its checkpoint is downloaded into that folder
(created if needed) on first use and reused afterwards: the model chosen
with model, or ViT-H when the folder holds no checkpoint at all.
The download is checked against the published size, resumes where an
interrupted one stopped, and waits for another run downloading the same
file. The -n flag forbids downloading: a missing checkpoint is
then an error.
A checkpoint is recognised in the weights folder by
vit_h, vit_l or vit_b in its
.pth file name, so renamed copies such as
sam_vit_h_checkpoint.pth are found too. The best model present
is used (ViT-H, then ViT-L, then ViT-B), or the one chosen with
model; the module checks that the file really holds that model. If a
folder holds several files of the chosen model, the first in
alphabetical order is used, with a warning.
The region is processed in square crops. A region no larger than
tile_size cells is one crop, a larger region is split into tiles
of tile_size cells overlapping by tile_overlap cells. With
crop_layers greater than 0, each tile is additionally segmented in
zoomed-in sub-crops (4 in layer 1, 16 in layer 2, ...), prompted with
points_per_side divided by crop_points_downscale per layer.
Each crop is resized so that its longest side is 1024 pixels and passed
through the image encoder. The mask decoder is then prompted with a grid
of points_per_side x points_per_side points, batch
prompts at a time, returning three candidate masks per point. A mask is
kept if its predicted IoU exceeds iou_threshold and its stability
score (the IoU of the mask thresholded at plus and minus
stability_offset) is at least stability_threshold.
Duplicates whose bounding boxes overlap by more than
nms_threshold (IoU) are suppressed, keeping the best one. Masks
touching an inner tile or crop edge are discarded, since the neighbouring
tile sees them whole. Holes and islands smaller than minsize cells
are finally removed from each mask.
These steps and their defaults are those of the reference
SamAutomaticMaskGenerator; on identical input,
i.sam.opencl returns the same masks.
SAM masks overlap: a field and the parcels inside it can both be masks.
The overlap rule chooses which mask a cell belongs to: the
smallest covering mask (small, finest partition, the default),
the largest (large) or the one with the highest predicted IoU
(score). Masks larger than maxsize cells are discarded
first; this is the way to drop the single mask SAM often returns for the
whole background of a scene.
The resulting label raster is cleaned: every connected piece smaller than
minsize cells, including small unsegmented gaps, is merged into
the neighbouring piece it shares the longest border with. Boundaries are
traced along cell edges into shared topological boundaries, simplified
with the Douglas-Peucker algorithm using the simplify tolerance
(one cell by default; 0 keeps the raster staircase), and the topology is
cleaned (boundaries broken at intersections, duplicates removed,
pseudo-nodes merged, slivers smaller than one cell removed). Each segment
gets one centroid, placed at the cell farthest from its border.
Areas without a centroid are gaps that no mask covers. The optional
raster output holds the segment categories of the cleaned label
raster before simplification.
With device=auto, the first GPU is used, else the first
CPU device. When several OpenCL platforms expose the same device (for
example Mesa's Clover and rusticl drivers for AMD GPUs), the first
platform on which the kernels compile is used; platform restricts
the choice to platforms whose name contains the given text. Mesa's
rusticl only exposes a GPU when enabled in the environment, e.g.
RUSTICL_ENABLE=radeonsi. A CPU OpenCL implementation such as
PoCL gives the same result but is about two orders of magnitude slower.
The ViT-H weights take 2.4 GiB of device memory and the work buffers
about 1.3 GiB with the default batch of 32 prompts; a smaller
batch reduces memory use. Buffers are kept below the device's
maximum allocation size.
The profile chooses how buffers are allocated and which matrix
product kernels run, to suit the device:
cpu: a device working in host memory, such as PoCL. Buffers are
mapped instead of copied, and the matrix products use kernels without
local memory that stream rows with 8-wide vector loads, which PoCL
runs far better than the GPU kernels.
igpu: a GPU sharing host memory (integrated graphics, APU). Transfers
map the buffers instead of copying them; the driver places the
buffers (on Intel, buffers allocated in host memory slowed the
kernels by about 15 %).
dgpu: a GPU with its own memory on internal PCI Express. Buffers
live in device memory.
egpu: a GPU behind a slow link, such as a Thunderbolt or USB4
enclosure. Buffers live in device memory and large transfers go
through a pinned staging buffer. The weights stay on the device for
the whole run.
With profile=auto the profile is detected: cpu for a CPU
device, igpu for a GPU that reports unified memory, egpu for a GPU
whose PCI function Linux marks as removable (when the driver reports
its PCI address), dgpu otherwise. The chosen profile and the reason
are printed. A profile can be forced to compare them; every profile
gives the same results, only the speed differs.
precision=fp16 stores the weight matrices of the linear layers
(most of the model's memory) as IEEE half-precision numbers, read with
the core OpenCL vload_half functions, so no half-precision extension
is needed; the arithmetic stays in single precision. It halves their
memory and the bytes read by the matrix products, which matters on
devices limited by memory bandwidth (integrated GPUs, CPUs). Rounding
the weights to half precision changes the masks slightly; check the
result against precision=fp32 (the default) on a sample before
relying on it.
On an AMD Radeon Pro WX 7100 (Polaris, 16 GiB, Mesa rusticl), with the
ViT-H model, loading the model takes 1-2 s, the image encoder 3.8 s per
crop and the mask decoder 3.7 s per 1024 prompts (the default 32 x 32
grid). A single-crop region is thus segmented in about 8 s. The number
of crops, and so the run time, grows with the region size divided by
tile_size, and with crop_layers.
SAM sees every crop at 1024 x 1024 pixels. Objects should span at least
ten or so pixels at that scale: set the region resolution and
tile_size accordingly. A smaller tile_size zooms in (crops
are upsampled to 1024 pixels), which finds smaller objects at the cost of
more crops. For agricultural parcels in 2.5 to 10 m imagery, the defaults
work well.
When the environment variable I_SAM_OPENCL_DUMP names a
directory, the 8-bit crops, the encoder inputs, the image embeddings and
the list of masks are written there. The script
tests/reference_compare.py in the source tree compares them
with the PyTorch reference implementation.
I_SAM_OPENCL_PROFILE=1 prints the time spent in each OpenCL
kernel. Each line gives the event time and the wall-clock time (some drivers, such as rusticl on Intel iris, report wrong event times), and for the matrix products the achieved GFLOP/s, which shows whether a kernel is limited by compute or by memory bandwidth. I_SAM_OPENCL_DOWNLOAD_URL replaces the download
location (https://dl.fbaipublicfiles.com/segment_anything),
for a local mirror.
Delineate fields in a 2.5 m VNIR image with bands 1-3 grouped as RGB,
keeping only segments of at least 0.1 ha (160 cells) and a label raster
alongside the vector map:
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L.,
Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollar, P., Girshick, R.,
2023. Segment Anything. Proceedings of the IEEE/CVF International
Conference on Computer Vision (ICCV), 4015-4026.
arXiv:2304.02643