Skip to content

Does it support Wordpiece tokenizer? #71

Open
Open
Does it support Wordpiece tokenizer?#71
@yangxudong

Description

@yangxudong
yangxudong
opened on May 19, 2025

Does it support Wordpiece tokenizer?

if it does, may I know some examples that create a Wordpiece tokenizer

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions