Document weird behavior for old GPUs giving strange "GET" errors

[anwai98]

Initiated by @lufre1:

(sam) [u12103@ggpu02 u12103]$ micro_sam.annotator_3d -i /mnt/ceph-ssd/workspaces/ws/nim00007/u12103-volume-em/4007_split/annotated_by_luca/block_1.h5 -k raw
[VGL] NOTICE: Replacing dlopen("/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/libGL.so.1") with dlopen("libvglfaker.so")
2025-11-04 10:00:27.377124458 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:2418 CreateInferencePybindStateModule] Init provider bridge failed.
Compute Image Embeddings 3D:   0%|                                                                                                                                                                                  | 0/128 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/bin/micro_sam.annotator_3d", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micro-sam/micro_sam/sam_annotator/annotator_3d.py", line 145, in main
    annotator_3d(
  File "/mnt/vast-nhr/home/freckmann15/u12103/micro-sam/micro_sam/sam_annotator/annotator_3d.py", line 99, in annotator_3d
    state.initialize_predictor(
  File "/mnt/vast-nhr/home/freckmann15/u12103/micro-sam/micro_sam/sam_annotator/_state.py", line 136, in initialize_predictor
    self.image_embeddings = util.precompute_image_embeddings(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micro-sam/micro_sam/util.py", line 1071, in precompute_image_embeddings
    embeddings = _compute_3d(input_, predictor, f, save_path, lazy_loading, pbar_init, pbar_update, batch_size)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micro-sam/micro_sam/util.py", line 874, in _compute_3d
    batched_embeddings, original_sizes, input_sizes = _compute_embeddings_batched(predictor, batched_images)
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micro-sam/micro_sam/util.py", line 646, in _compute_embeddings_batched
    features = predictor.model.image_encoder(batched_tensors)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/python3.12/site-packages/mobile_sam/modeling/image_encoder.py", line 107, in forward
    x = self.patch_embed(x)
        ^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/python3.12/site-packages/mobile_sam/modeling/image_encoder.py", line 392, in forward
    x = self.proj(x)
        ^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 548, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/vast-nhr/home/freckmann15/u12103/micromamba/envs/sam/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 543, in _conv_forward
    return F.conv2d(
           ^^^^^^^^^
RuntimeError: GET was unable to find an engine to execute this computation

In short, it's a V100 problem for a combination of CUDA12.9 and PyTorch v2.8.

We need to document this in FAQs for users to know about this!