Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.

Learning BPE embeddings by first learning a segmentation model and then training word2vec

Byte Pair Encoding (BPE)

Split tokens into word pieces

Detect whether the text is AI-generated by training a new tokenizer and combining it with tree classification models or by training language models on a large dataset of human & AI-generated texts.

WordPiece Tokenizer for BERT models.

go-bert-tokenizer

A framework for generating subword vocabulary from a tensorflow dataset and building custom BERT tokenizer models.