HF BERT checkpoints exported by older transformers versions persist non-weight integer buffers in model.safetensors — e.g. embeddings.position_ids (I64 arange) in BAAI/bge-small-en-v1.5. SafeTensorsParametersLoader.load<T=FP32> fails the entire load on the first such tensor:
IllegalArgumentException: SafeTensors I64 tensor 'embeddings.position_ids' requires Int32 dtype (downcast), got FP32
at SafeTensorsParametersLoader.load(SafeTensorsParametersLoader.kt:124)
There is no way to skip tensors: the loader exposes no filter, and the exception fires inside load before the caller's callback can intervene.
Request: a tensorFilter: (name: String) -> Boolean = { true } constructor parameter (or: skip-with-warning for integer tensors when the requested dtype is a float type).
Downstream workaround: SKaiNET-transformers carries an interim FloatSafeTensorsLoader (built on the public StreamingSafeTensorsReader) for BERT embedding loads — to be dropped once this lands, same pattern as the permute-axes handler (#803).
🤖 Generated with Claude Code
HF BERT checkpoints exported by older
transformersversions persist non-weight integer buffers inmodel.safetensors— e.g.embeddings.position_ids(I64 arange) inBAAI/bge-small-en-v1.5.SafeTensorsParametersLoader.load<T=FP32>fails the entire load on the first such tensor:There is no way to skip tensors: the loader exposes no filter, and the exception fires inside
loadbefore the caller's callback can intervene.Request: a
tensorFilter: (name: String) -> Boolean = { true }constructor parameter (or: skip-with-warning for integer tensors when the requested dtype is a float type).Downstream workaround: SKaiNET-transformers carries an interim
FloatSafeTensorsLoader(built on the publicStreamingSafeTensorsReader) for BERT embedding loads — to be dropped once this lands, same pattern as the permute-axes handler (#803).🤖 Generated with Claude Code