BPE Tokenizer Trainer
Train a Byte Pair Encoding tokenizer on your text and see how it works step by step.
Target Vocab Size:
Train BPE
Clear
Training Text
The quick brown fox jumps over the lazy dog. The dog was very lazy and the fox was very quick. This is a sample text for training a BPE tokenizer. We need repeated patterns to see meaningful merges.
Merge Steps
Final Vocabulary
Tokenize New Text
The quick brown fox
Tokenize