Tokenisation

[turchmo]

Does the sentence pairs need to be tokenised before using the Invitation Model?

[amirkamran]

This is not a requirement but proper tokenization can produce better results. It is a data selection tool, if you want to add it e.g. in the SMT pipeline, it should be run after the preprocessing of data and then you can take n number of top sentences to pass to the alignment step.