Add Seed-OSS architecture support (`SeedOssForCausalLM`) by PMeeske · Pull Request #2116 · microsoft/onnxruntime-genai

PMeeske · 2026-05-03T16:59:38Z

Status: Draft / Suggestion. Opening as draft for community review and to give the maintainers a chance to weigh in on direction before promoting. The patch is functional end-to-end and was validated against a real production model, but the upstream-fit decisions (PR target branch, CI gates, test coverage expectations) are yours.

What

Adds support for ByteDance's Seed-OSS architecture (SeedOssForCausalLM) to onnxruntime_genai.models.builder. The motivating model is NousResearch/Hermes-4.3-36B (Hermes fine-tune of Seed-OSS-36B), which is currently rejected by the builder with NotImplementedError: model is not currently supported.

Why

Seed-OSS is structurally a Llama-family decoder with two extra per-layer normalisations injected into the residual branches. Because the C++ runtime (DecoderOnly_Model) is architecture-agnostic for decoder-only models — it loads and runs whatever ONNX graph the builder produces — the only blocker is the Python-side builder router and a one-line addition to the C++ LLM-type registry. No kernel changes, no KV cache changes, no graph executor changes.

Architecture spec

Verified against the public Seed-OSS-36B config + the Hermes-4.3-36B fine-tune. Key invariants:

Input → input_layernorm → Attention → attn_post_norm ─→ + ─→
                                                         │
                                                         └── (residual)

      → post_attention_layernorm → MLP → ffn_post_norm ─→ + ─→
                                                         │
                                                         └── (residual)

Aspect	Value
`attention_bias`	`True` (QKV projection biases present)
`attention_out_bias`	`False` (`o_proj` is clean)
Attention	GQA — `num_key_value_heads != num_attention_heads` (8 KV / 80 Q on the 36B)
RoPE	`rope_theta = 1e7` (higher than Llama-3's `5e5`)
Extra norms (vs Llama)	`attn_post_norm` after attention, `ffn_post_norm` after MLP — both inside the residual path
Activation	`silu` (same as Llama)
Normalisation	RMSNorm (same as Llama)

The two post-norms are the only structural delta from Llama. attention_bias is auto-detected by the existing base.py logic via hasattr(attn.q_proj, 'bias'), so no manual override is required at runtime — but SeedOssModel.__init__ sets the explicit flag for clarity in case the heuristic is bypassed.

Patch shape

File	Change	LoC
`src/python/py/models/builders/seed_oss.py`	NEW. `SeedOssModel(LlamaModel)` with `make_layer` override that injects `attn_post_norm` + `ffn_post_norm` between attention/MLP and the residual add. Auto-detects post-norm tensor location (HF layouts vary: `self_attn.post_attn_layernorm` vs `layer.attn_post_norm` — both supported).	132
`src/python/py/models/builder.py`	+1 import in the `from builders import (...)` block, +1 `elif` in `create_model` routing `SeedOssForCausalLM` to `SeedOssModel`.	+3
`src/python/py/models/builders/__init__.py`	+1 line: `from .seed_oss import SeedOssModel`.	+1
`src/models/model_type.h`	Add `\"seed_oss\"` to the LLM string array, alphabetically (between `\"qwen3\"` and `\"smollm3\"`); bump `std::array<...,21>` → `<...,22>`.	+1, -1

Total: ~135 lines net, no C++ runtime changes beyond the type-registry string.

Validation

End-to-end build verified against NousResearch/Hermes-4.3-36B:

python -m onnxruntime_genai.models.builder \
  -i /path/to/Hermes-4.3-36B \
  -o ./hermes-4.3-36b-onnx-int4-dml \
  -p int4 -e dml \
  --extra_options int4_block_size=32 int4_accuracy_level=4

All 64 decoder layers + final norm + LM head quantised cleanly (~64 INT4 MatMul nodes per layer)
Output graph: model.onnx (704 KB) + model.onnx.data (21 GB at INT4 + biases at f16)
Loads through Microsoft.ML.OnnxRuntimeGenAI.DirectML 0.13.2 C# bindings on AMD DirectML hardware (RX 9060 XT, 16 GB VRAM)
Produces coherent generation on chat prompts — no DmlFusedNode errors, no shape mismatches

The same patch produces a CUDA build cleanly (-e cuda), and the 2-layer stub smoke-tests in our internal harness pass.

A note on `genai_config.json` `model.type`

After the builder writes genai_config.json, the model.type field is set to seed_oss (matching config.json's model_type). The C++ runtime recognises this once \"seed_oss\" is in IsLLM's array, so no manual config-rewrite is needed at inference time.

(Operators on a release that doesn't yet include this PR can work around the C++ side by setting model.type = \"llama\" in genai_config.json after the build — Seed-OSS is sufficiently Llama-shaped that the runtime accepts it. With this PR merged, that hack is unnecessary.)

Things I'd appreciate maintainer guidance on

CI test coverage. Happy to add a 2-layer stub builder test for Seed-OSS following the pattern of existing per-arch coverage if there is one — a pointer to the right file would help.
Whether you want the layer_attrs defensive-init pattern. SeedOssModel.__init__ includes if not hasattr(self, 'layer_attrs') or self.layer_attrs is None: self.layer_attrs = {} — needed because LlamaModel base initialisation paths vary across recent commits. Could be removed once base.py guarantees the attribute. Open to either approach.
Documentation placement. Happy to add a short note to the per-architecture support docs once you point me at the right location.

Out of scope (deliberately)

Adding new test fixtures pulling Seed-OSS weights — wanted to keep this PR's scope tight.
Changes to the C++ decoder pipeline — none required.
Vision/multimodal Seed-OSS variants — none exist publicly today; if/when they ship I'd open a separate VLM PR.

Spec material in this PR description was derived from internal architectural notes assembled while validating the patch against the 36B production model. Happy to expand any section, tighten the language, or restructure to match your contribution-guide template.

ByteDance's Seed-OSS architecture (`SeedOssForCausalLM`) is structurally a Llama-family decoder with two extra per-layer normalisations injected into the residual branches: Input → input_layernorm → Attention → attn_post_norm ─→ + ─→ (residual) → post_attention_layernorm → MLP → ffn_post_norm ─→ + ─→ (residual) It also sets `attention_bias=True` (QKV projection biases present), `attention_out_bias=False` (o_proj clean), uses GQA, and a high `rope_theta=1e7`. NousResearch's Hermes-4.3-36B is the most prominent Seed-OSS-based open-weight model and was the motivating use case here. This patch adds: * `builders/seed_oss.py`: `SeedOssModel` inheriting from `LlamaModel` with a `make_layer` override that wires the two extra layernorms inside the residual path. Auto-detects post-norm tensor location (some HF weight layouts expose it as `self_attn.post_attn_layernorm`, others as `layer.attn_post_norm` — both supported). `layer_attrs` is initialised defensively since LlamaModel base does not always populate it depending on the version path. * `builders/__init__.py`: export `SeedOssModel`. * `builder.py`: route `SeedOssForCausalLM` to `SeedOssModel` in `create_model` and add the import. * `models/model_type.h`: register `seed_oss` in the LLM type array (size 21 → 22) so the C++ runtime selects the standard `DecoderOnly_Model` pipeline at load. No graph or KV-cache changes in C++ are required — Seed-OSS's structural similarity to Llama means the existing decoder-only runtime handles the resulting graph transparently. Validated on a CPU build target and on `-e dml` against `NousResearch/Hermes-4.3-36B` (36B, FP16 BF16 source, INT4 output via the `int4_block_size=32` path); the resulting graph loads cleanly through `Microsoft.ML.OnnxRuntimeGenAI.DirectML 0.13.2` and produces coherent generation on AMD DirectML hardware (RX 9060 XT, 16 GB VRAM). Total delta is ~135 lines of Python plus a single one-line C++ array update, with no kernel or runtime changes.

PMeeske · 2026-05-03T17:13:54Z

@microsoft-github-policy-service agree

PMeeske · 2026-05-03T17:15:11Z

For transparency: the Seed-OSS architectural mapping in this PR — the make_layer topology, post-norm injection points, attention_bias / attention_out_bias detection, and the C++ runtime "architecture-agnostic decoder-only" rationale — was authored by me on 2026-04-27 in a private project, before shipping it upstream. I used AI assistance (Claude Code) to apply the patch to your current main, add a defensive layer_attrs init for cross-version compatibility, write the commit message and this PR body, and orchestrate the fork/branch/PR mechanics. The substantive engineering decisions are mine; the upstream-fit polish was AI-assisted. Happy to expand on any specific design choice.

kunal-vaishnavi · 2026-05-13T00:52:15Z

+        if not hasattr(self, "layer_attrs") or self.layer_attrs is None:
+            self.layer_attrs = {}
+
+        # SeedOss has per-layer post-norms after attention and MLP


Some of the Gemma models use a similar architecture for a post-normalization after the attention and FFN blocks. Would switching to inherit from one of those classes instead simplify the implementation of this class?

kunal-vaishnavi · 2026-05-13T00:52:50Z

        onnx_model = GraniteModel(config, io_dtype, onnx_dtype, execution_provider, cache_dir, extra_options)
    elif config.architectures[0] == "InternLM2ForCausalLM":
        onnx_model = InternLM2Model(config, io_dtype, onnx_dtype, execution_provider, cache_dir, extra_options)
+    elif config.architectures[0] == "SeedOssForCausalLM":


Can we insert this in a way such that we maintain alphabetical order?

kunal-vaishnavi · 2026-05-13T00:53:56Z

@@ -1,3 +1,4 @@
+from .seed_oss import SeedOssModel


Can we insert this in a way such that we maintain alphabetical order?

Copilot

Pull request overview

Adds initial support for ByteDance Seed-OSS (SeedOssForCausalLM) to the Python model builder and updates the C++ model-type registry so the runtime can recognize the resulting genai_config.json as an LLM.

Changes:

Introduces a new SeedOssModel builder class (subclassing LlamaModel) intended to inject Seed-OSS post-norms in each decoder layer.
Routes SeedOssForCausalLM to the new builder in src/python/py/models/builder.py and exposes it from builders.
Adds seed_oss to the C++ ModelType::IsLLM allowlist.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
src/python/py/models/builders/seed_oss.py	New Seed-OSS builder implementation overriding `make_layer` to add per-layer post-norms.
src/python/py/models/builders/init.py	Exposes the new builder from the `builders` package.
src/python/py/models/builder.py	Routes `SeedOssForCausalLM` to `SeedOssModel` and imports it.
src/models/model_type.h	Adds `seed_oss` to the runtime’s LLM type registry.

Comments suppressed due to low confidence (4)

src/python/py/models/builders/seed_oss.py:101

attn_post_norm is created using location="post_attention", which collides with the existing post_attention_layernorm that also uses location="post_attention". Since location is used to form node/initializer names, this will create duplicate initializer names and node names for the same layer.

                self.make_layernorm(
                    layer_id,
                    attn_post_norm,
                    skip=True,
                    simple=self.layernorm_attrs["simple"],
                    location="post_attention",
                )
                # base.py accumulates into layernorm_attrs["output_0"]
                # the attention output is still in the residual path

        # ---- post_attention_layernorm (standard) ----
        self.make_layernorm(
            layer_id,
            layer.post_attention_layernorm,
            skip=True,
            simple=self.layernorm_attrs["simple"],
            location="post_attention",
        )

src/python/py/models/builders/seed_oss.py:93

The attn_post_norm path calls make_layernorm(..., skip=True, location=...), which builds a SkipLayerNorm (i.e., performs a residual add using the current layernorm_attrs[root_input] and layernorm_attrs[skip_input]). Seed-OSS’ attn_post_norm is a post-norm on the attention output before the residual add, so this wiring will produce incorrect residual semantics (and can double-add the attention output). Please implement attn_post_norm as a standalone norm on the attention output, then feed its output as the skip_input to the subsequent post_attention_layernorm residual op.

        # ---- attn_post_norm (SeedOss specific) ----
        if self.layer_attrs.get("has_attn_post_norm", False):
            # Some HF weights flatten this into self_attn.post_attn_layernorm,
            # others expose it as layer.attn_post_norm.  Try both.
            attn_post_norm = None
            if hasattr(layer.self_attn, "post_attn_layernorm"):
                attn_post_norm = layer.self_attn.post_attn_layernorm
            elif hasattr(layer, "attn_post_norm"):
                attn_post_norm = layer.attn_post_norm

            if attn_post_norm is not None:
                self.make_layernorm(
                    layer_id,
                    attn_post_norm,
                    skip=True,
                    simple=self.layernorm_attrs["simple"],
                    location="post_attention",
                )
                # base.py accumulates into layernorm_attrs["output_0"]
                # the attention output is still in the residual path

src/python/py/models/builders/seed_oss.py:127

Similarly, ffn_post_norm is currently built via make_layernorm(..., skip=True, location="post_mlp"), which performs a residual add+norm using the current residual root and the MLP output. Seed-OSS needs ffn_post_norm applied to the MLP output before it is added to the residual, so this will mis-wire the block and can cause double-adds on the next layer’s input SkipLayerNorm. Please apply ffn_post_norm as a standalone norm on the MLP output and use that as the subsequent residual skip_input.

        # ---- ffn_post_norm (SeedOss specific) ----
        if self.layer_attrs.get("has_ffn_post_norm", False):
            ffn_post_norm = None
            if hasattr(layer.mlp, "post_mlp_layernorm"):
                ffn_post_norm = layer.mlp.post_mlp_layernorm
            elif hasattr(layer, "ffn_post_norm"):
                ffn_post_norm = layer.ffn_post_norm

            if ffn_post_norm is not None:
                self.make_layernorm(
                    layer_id,
                    ffn_post_norm,
                    skip=True,
                    simple=self.layernorm_attrs["simple"],
                    location="post_mlp",
                )

src/python/py/models/builders/init.py:6

This import was inserted before the module’s copyright/license header, which breaks the convention used across the repo (headers appear at the top of the file). Please move the import below the header block.

from .seed_oss import SeedOssModel
# -------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation.  All rights reserved.
# Licensed under the MIT License.  See License.txt in the project root for
# license information.
# -------------------------------------------------------------------------

+"""
+SeedOssModel ONNX builder for ONNX Runtime GenAI.
+
+SeedOss architecture (ByteDance Seed) is structurally similar to Llama with:
+  - attention_bias=True (QKV bias)
+  - attention_out_bias=False
+  - attn_post_norm per decoder layer (extra vs Llama)
+  - ffn_post_norm per decoder layer (extra vs Llama)
+  - GQA (num_key_value_heads != num_attention_heads)
+  - RoPE with rope_theta=1e7
+
+This builder inherits from LlamaModel and extends make_layer() to
+inject the extra post-norms in the residual path.
+"""


+class SeedOssModel(LlamaModel):
+    def __init__(self, config, io_dtype, onnx_dtype, ep, cache_dir, extra_options):
+        super().__init__(config, io_dtype, onnx_dtype, ep, cache_dir, extra_options)
+
+        # LlamaModel base does not always populate layer_attrs (varies by
+        # ORT-GenAI version); ensure the dict exists before we set keys on it.
+        if not hasattr(self, "layer_attrs") or self.layer_attrs is None:
+            self.layer_attrs = {}
+
+        # SeedOss has per-layer post-norms after attention and MLP
+        # These bools inform the downstream graph builder to wire extra
+        # LayerNorm/RMSNorm nodes into the decoder block.
+        self.layer_attrs["has_attn_post_norm"] = getattr(config, "has_attn_post_norm", True)
+        self.layer_attrs["has_ffn_post_norm"] = getattr(config, "has_ffn_post_norm", True)
+
+        # attention_bias is auto-detected in base.py via hasattr on attn.q_proj.bias
+        # attention_out_bias is auto-detected similarly on attn.o_proj.bias
+        # We only need to set flags if the config explicitly overrides detection
+        self.attention_bias = getattr(config, "attention_bias", True)
+        self.attention_out_bias = getattr(config, "attention_out_bias", False)
+


+        # attention_bias is auto-detected in base.py via hasattr on attn.q_proj.bias
+        # attention_out_bias is auto-detected similarly on attn.o_proj.bias
+        # We only need to set flags if the config explicitly overrides detection
+        self.attention_bias = getattr(config, "attention_bias", True)
+        self.attention_out_bias = getattr(config, "attention_out_bias", False)


  inline static bool IsLLM(const std::string& model_type) {
    // Large-language model (LLM)
-    static constexpr std::array<std::string_view, 21> LLM = {"chatglm", "decoder", "ernie4_5", "gemma", "gemma2", "gemma3_text", "gpt2", "gptoss", "granite", "internlm2", "llama", "mistral", "nemotron", "olmo", "phi", "phimoe", "phi3", "phi3small", "qwen2", "qwen3", "smollm3"};
+    static constexpr std::array<std::string_view, 22> LLM = {"chatglm", "decoder", "ernie4_5", "gemma", "gemma2", "gemma3_text", "gpt2", "gptoss", "granite", "internlm2", "llama", "mistral", "nemotron", "olmo", "phi", "phimoe", "phi3", "phi3small", "qwen2", "qwen3", "seed_oss", "smollm3"};


kunal-vaishnavi · 2026-05-13T01:00:11Z

Things I'd appreciate maintainer guidance on

CI test coverage. Happy to add a 2-layer stub builder test for Seed-OSS following the pattern of existing per-arch coverage if there is one — a pointer to the right file would help.

Whether you want the layer_attrs defensive-init pattern. SeedOssModel.init includes if not hasattr(self, 'layer_attrs') or self.layer_attrs is None: self.layer_attrs = {} — needed because LlamaModel base initialisation paths vary across recent commits. Could be removed once base.py guarantees the attribute. Open to either approach.

Documentation placement. Happy to add a short note to the per-architecture support docs once you point me at the right location.

Thanks for your contribution! The proposed approach looks good to me. To answer your questions:

There is an open PR that initializes these stubs. Once that is merged, you can add a similar one.
Let's see if we can re-use an existing class as a parent class for handling the extra LayerNorms. Then, the defensive-init patterns can be removed.
The main repo README and model builder README should be updated.

kunal-vaishnavi · 2026-05-13T01:04:39Z

  inline static bool IsLLM(const std::string& model_type) {
    // Large-language model (LLM)
-    static constexpr std::array<std::string_view, 21> LLM = {"chatglm", "decoder", "ernie4_5", "gemma", "gemma2", "gemma3_text", "gpt2", "gptoss", "granite", "internlm2", "llama", "mistral", "nemotron", "olmo", "phi", "phimoe", "phi3", "phi3small", "qwen2", "qwen3", "smollm3"};
+    static constexpr std::array<std::string_view, 22> LLM = {"chatglm", "decoder", "ernie4_5", "gemma", "gemma2", "gemma3_text", "gpt2", "gptoss", "granite", "internlm2", "llama", "mistral", "nemotron", "olmo", "phi", "phimoe", "phi3", "phi3small", "qwen2", "qwen3", "seed_oss", "smollm3"};


I believe the model builder will auto-create the model type as seedoss in the GenAI config.

Suggested change

static constexpr std::array<std::string_view, 22> LLM = {"chatglm", "decoder", "ernie4_5", "gemma", "gemma2", "gemma3_text", "gpt2", "gptoss", "granite", "internlm2", "llama", "mistral", "nemotron", "olmo", "phi", "phimoe", "phi3", "phi3small", "qwen2", "qwen3", "seed_oss", "smollm3"};

static constexpr std::array<std::string_view, 22> LLM = {"chatglm", "decoder", "ernie4_5", "gemma", "gemma2", "gemma3_text", "gpt2", "gptoss", "granite", "internlm2", "llama", "mistral", "nemotron", "olmo", "phi", "phimoe", "phi3", "phi3small", "qwen2", "qwen3", "seedoss", "smollm3"};

kunal-vaishnavi marked this pull request as ready for review May 13, 2026 00:49

kunal-vaishnavi requested a review from a team as a code owner May 13, 2026 00:49

Copilot AI review requested due to automatic review settings May 13, 2026 00:49

Copilot started reviewing on behalf of kunal-vaishnavi May 13, 2026 00:50 View session

kunal-vaishnavi reviewed May 13, 2026

View reviewed changes

Copilot AI reviewed May 13, 2026

View reviewed changes

kunal-vaishnavi reviewed May 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Seed-OSS architecture support (`SeedOssForCausalLM`)#2116

Add Seed-OSS architecture support (`SeedOssForCausalLM`)#2116
PMeeske wants to merge 1 commit into
microsoft:mainfrom
PMeeske:feat/seed-oss-architecture

PMeeske commented May 3, 2026

Uh oh!

PMeeske commented May 3, 2026

Uh oh!

PMeeske commented May 3, 2026

Uh oh!

kunal-vaishnavi May 13, 2026

Uh oh!

kunal-vaishnavi May 13, 2026

Uh oh!

kunal-vaishnavi May 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

kunal-vaishnavi commented May 13, 2026

Uh oh!

kunal-vaishnavi May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	static constexpr std::array<std::string_view, 22> LLM = {"chatglm", "decoder", "ernie4_5", "gemma", "gemma2", "gemma3_text", "gpt2", "gptoss", "granite", "internlm2", "llama", "mistral", "nemotron", "olmo", "phi", "phimoe", "phi3", "phi3small", "qwen2", "qwen3", "seed_oss", "smollm3"};
	static constexpr std::array<std::string_view, 22> LLM = {"chatglm", "decoder", "ernie4_5", "gemma", "gemma2", "gemma3_text", "gpt2", "gptoss", "granite", "internlm2", "llama", "mistral", "nemotron", "olmo", "phi", "phimoe", "phi3", "phi3small", "qwen2", "qwen3", "seedoss", "smollm3"};

Conversation

PMeeske commented May 3, 2026

What

Why

Architecture spec

Patch shape

Validation

A note on genai_config.json model.type

Things I'd appreciate maintainer guidance on

Out of scope (deliberately)

Uh oh!

PMeeske commented May 3, 2026

Uh oh!

PMeeske commented May 3, 2026

Uh oh!

kunal-vaishnavi May 13, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi May 13, 2026

Choose a reason for hiding this comment

Uh oh!

kunal-vaishnavi May 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

kunal-vaishnavi commented May 13, 2026

Uh oh!

kunal-vaishnavi May 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

A note on `genai_config.json` `model.type`