# ULM-32k SlimPajama-3M
ULM tokeniser with vocabulary size 32768, trained on the first 3 million examples in [SlimPajama-627B](https://huggingface.co/datasets/cerebras/SlimPajama-627B).

## Tokeniser details
ULM trainer implementation: 
  - Back-end: [SentencePiece](https://github.com/google/sentencepiece)'s `SentencePieceTrainer`.
  - Front-end: [TkTkT](https://github.com/bauwenst/TkTkT)'s [`KudoPieceTrainer`](https://github.com/bauwenst/TkTkT/blob/341ae85980a5a9a2d60dbdc88645f8828b5c3a06/src/tktkt/models/kudopiece/vocabularisation.py#L40) 

Preprocessor: 
  - During training: TkTkT's [`SentencePiecePreprocessor`](https://github.com/bauwenst/TkTkT/blob/341ae85980a5a9a2d60dbdc88645f8828b5c3a06/src/tktkt/preparation/instances.py#L181)
  - During inference: TkTkT's [`ModernEnglishPreprocessor`](https://github.com/bauwenst/TkTkT/blob/341ae85980a5a9a2d60dbdc88645f8828b5c3a06/src/tktkt/preparation/instances.py#L105)
    1. NFKC normalisation
    2. Punctuation splitter, whitespace splitter, English contraction splitter
    3. GPT-2's pseudo-byte mapping
    4. Start-of-word marker `Ġ`
    5. Digit and hyphen isolation

## Training details
**Time:** 3h40m
  - Preprocessing and counting the 3M corpus: 2h45m
  - ULM algorithm: 55m

**Memory:** 257 GiB peak usage (i.e. about 80 GiB RAM per million sentences).

**Data sizes:**
  - Examples considered: 3 000 000
  - Examples used: 2 609 893 (390 107 examples dropped for being > 8192 characters).
  - Characters counted: 6 685 212 190
  - Unique words after whitespace splitting: 9 254 839