As today tokcpp uses a preview of sentencepiece 0.2: https://github.com/google/sentencepiece/tree/f2219b53e24ff5deee4cacdc2d0ca3074e529a07 Google released sentencepiece 0.2 in feb 2024: https://github.com/google/sentencepiece/releases/tag/v0.2.0 Would you mind if I upgrade tokcpp to use that 0.2 final version of sentencepiece?
As today tokcpp uses a preview of sentencepiece 0.2:
https://github.com/google/sentencepiece/tree/f2219b53e24ff5deee4cacdc2d0ca3074e529a07
Google released sentencepiece 0.2 in feb 2024:
https://github.com/google/sentencepiece/releases/tag/v0.2.0
Would you mind if I upgrade tokcpp to use that 0.2 final version of sentencepiece?