GPU inference fails on Blackwell (RTX 5080, sm_120): CUDA_ERROR_INVALID_HANDLE during SavedModel execution

[axeljackal]

Summary

I'm trying to run InkSight inference on an RTX 5080 Laptop (Blackwell, sm_120) inside Docker.
The service starts correctly, model loads, and TensorFlow can run basic GPU ops.
However, real InkSight inference fails on GPU with runtime errors (notably CUDA_ERROR_INVALID_HANDLE), while CPU execution works reliably.

This looks like a model-graph/runtime compatibility issue on this hardware/software stack, not a Docker mounting or basic GPU visibility issue.

---

Environment

• GPU: NVIDIA GeForce RTX 5080 Laptop (compute capability 12.0 / sm_120)
• OS: Windows 11 (Docker Desktop)
• Container base: nvcr.io/nvidia/tensorflow:25.02-tf2-py3
• TensorFlow: 2.17.0+nv25.2
• Python: 3.12.3
• tensorflow-text: 2.18.1 (cp312 wheel)
• Model: Derendering/InkSight-Small-p (local mount)

Docker/runtime configuration:

• GPU device request enabled
• model mounted at /data/models/inksight
• shm_size: 1gb
• ulimits: memlock=-1, stack=67108864

---

Expected behavior

/vectorize runs on GPU and returns SVG output reliably.

Actual behavior

• /health is OK
• model loads
• basic TensorFlow GPU operations succeed
• InkSight inference graph fails on GPU with errors like:
  • CUDA_ERROR_INVALID_HANDLE
  • cuLaunchKernel ... failed
  • failures around Cast / StatefulPartitionedCall
• CPU inference works and returns valid SVG

---

Key error snippets

INTERNAL: 'cuLaunchKernel(...)' failed with 'CUDA_ERROR_INVALID_HANDLE'
...
Detected at node Cast / StatefulPartitionedCall

And when trying a naive runtime switch to CPU (without reloading model):

Trying to access resource ... located in device GPU:0 from device CPU:0

---

What I already tried

1) Building tensorflow-text 2.17 from source inside NGC (instead of wheel)

Attempted to avoid potential wheel mismatch, but hit multiple blockers:

• Bazel WORKSPACE/repo issues related to NGC TF metadata (@tf_wheel_version_suffix / prepare script path)
• linker/version-script failures in pybind targets
• BUILD dependency drift (Abseil log deps)
• occasional Bazel external download/network failures

2) Current pragmatic stack

• Keep NGC TF 2.17.0+nv25.2
• Install tensorflow-text==2.18.1 cp312 wheel
• Load model with tf.saved_model.load()

Result:

• stable service startup
• stable CPU inference
• unstable GPU inference on Blackwell

3) Fallback logic

• Direct with tf.device(/CPU:0) retry after GPU failure is not sufficient (device-bound variables)
• Reloading the SavedModel on CPU fixes fallback execution
• Current production workaround is CPU-preferred on sm_120 to avoid first-request GPU crash

---

Ask

Could you suggest the recommended way to make InkSight inference stable on Blackwell GPU (sm_120) for this stack, or confirm if this requires an upstream fix?

[axeljackal]

Update from additional TensorFlow-side debugging on March 13, 2026.

I expanded the matrix beyond the original NGC 2.17.0+nv25.2 stack and the key result is that the InkSight GPU failure on Blackwell is not limited to that exact TensorFlow build.

Newer TensorFlow-side results:

• official tensorflow==2.18.1 + tensorflow-text==2.18.1: model loads, GPU inference still fails with CUDA_ERROR_INVALID_HANDLE
• official tensorflow==2.20.0 + tensorflow-text==2.20.1: same failure
• pinning inputs to CPU did not fix it
• CUDA_MODULE_LOADING=EAGER did not fix it
• reloading the SavedModel on CPU after the GPU failure still works

Representative error on the newer matching TF/tf-text stacks:

INTERNAL: 'cuLaunchKernel(function, ..., reinterpret_cast<CUstream>(stream), params, nullptr)' failed with 'CUDA_ERROR_INVALID_HANDLE'
[[{{node Cast}}]]
[[StatefulPartitionedCall/SentenceTokenizer/...]]

So at this point the strongest read is still:

• InkSight itself/model artifact is valid
• GPU is visible and simple TF GPU ops work
• failure happens during real SavedModel graph execution on Blackwell (sm_120)
• this looks like a TensorFlow runtime / Blackwell support issue rather than an InkSight-only bug

One additional note: in a clean container, NGC TF 2.17.0+nv25.2 plus tensorflow-text==2.18.1 --no-deps is also not a clean supported combination; tensorflow_text import fails with an undefined symbol.

Current practical status:

• I still do not have a validated stable GPU TensorFlow path for InkSight on an RTX 5080
• the only stable supported path I confirmed here is CPU inference, including CPU reload after a GPU failure

Upstream TF issue with the expanded evidence:

• Blackwell (sm_120) GPU graph execution fails with CUDA_ERROR_INVALID_HANDLE while basic TF GPU ops succeed tensorflow/tensorflow#112272

I’m continuing to test additional combinations, but wanted to record that this still reproduces on newer official TF wheels too.

[axeljackal]

Follow-up from more Blackwell testing on an RTX 5080 Laptop GPU (sm_120).

I still do not have a stable path for end-to-end GPU serving_default inference, but I did find a practical TF-side workaround on:

• nvcr.io/nvidia/tensorflow:25.02-tf2-py3
• tensorflow==2.20.0
• tensorflow-text==2.20.1

The workable path is:

• CPU preproc_func
• GPU tpu_func
• CPU postprocess / detokenization
• then the existing downstream _text_to_tokens / _detokenize / _strokes_to_svg

On a real sample image, that split path produced decoded output text and a valid SVG on the 5080. So the numeric core model can run on GPU on Blackwell; the part that still breaks is the raw-input SavedModel __call__ / serving_default path.

Caveat: on the same sample image, the hybrid result was close to the pure CPU result but not byte-identical:

• both produced text_len=15149, token_count=1008, stroke_count=3
• CPU SVG length was 5741
• hybrid SVG length was 5718
• exact text/token/SVG equality was false

So I would call this a usable workaround, not a strict equivalence proof yet.

I also tested the model-card-recommended version band in a clean python:3.11 container:

• tensorflow[and-cuda]==2.17.1
• tensorflow-text==2.17.0

That was not viable on Blackwell here:

• CPU full path failed with UNIMPLEMENTED: unsupported operand type BF16 in op dot
• GPU core/hybrid path aborted with ptxas too old / XLA gemm fusion failure for CC 12.0

So at least in this environment, "use TF 2.15-2.17" is not enough to recover Blackwell GPU support.

Based on this, the most practical downstream direction may be to support a split execution path explicitly:

• CPU preprocessing and detokenization
• GPU core generation

If that separation were documented or exposed as a supported inference path, it would give Blackwell users a way to use GPU today while the upstream TF SavedModel issue is still open.

[axeljackal]

Another downstream update from March 14, 2026.

I implemented the hybrid Blackwell path locally in HandSolo and smoke-tested it on an RTX 5080 Laptop GPU (sm_120).

Local HandSolo result:

• base image: nvcr.io/nvidia/tensorflow:25.02-tf2-py3
• runtime: tensorflow==2.20.0, tensorflow-text==2.20.1
• service startup /health reported:
  • status=ok
  • model_loaded=true
  • inference_mode=gpu-hybrid-preferred
• /vectorize on a real sample image returned SVG successfully (svg_len=5400)

That means the split path is now practical in a real service:

• CPU preprocessing
• GPU core generation
• CPU postprocessing / SVG conversion

I also finished the extra fallback experiments:

• tensorflow[and-cuda]==2.17.1 + tensorflow-text==2.17.0 on python:3.11 is still not viable on Blackwell, even with XLA_FLAGS=--xla_gpu_autotune_level=0
• tf-nightly==2.22.0.dev20260313 + tensorflow-text-nightly==2.20.0.dev20250619 on python:3.11 is also not usable here (tensorflow-text-nightly import failure, and no usable GPU path)

So the best practical recommendation I have today for Blackwell users is:

• stop using end-to-end GPU serving_default
• use the split CPU-preproc / GPU-core / CPU-postprocess path
• target tensorflow==2.20.0 + tensorflow-text==2.20.1

From a downstream perspective, I think this is now the most realistic path to make InkSight usable on RTX 5080 while the upstream TensorFlow issue is still open.