Skip to content
This repository was archived by the owner on Jan 21, 2025. It is now read-only.
This repository was archived by the owner on Jan 21, 2025. It is now read-only.

[MOE-transformer] How do you build static graph of MOE-Model? #330

Open
Open
[MOE-transformer] How do you build static graph of MOE-Model?#330
@imyzx2017

Description

@imyzx2017
imyzx2017
opened on Jun 25, 2021

Thank you for your great work, Here I'm curious about MOE-Transformer's static graph construction.

Q: When there is 1024 experts, switch gating method is used, you need to build 1024 different graph for FFN_Module? What's the detail of graph construction?

(Is it related to #40 ？)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions