TextGen / added_tokens.json
YeBhoneLin10's picture
Upload tokenizer
5f6a40f verified
raw
history blame contribute delete
75 Bytes
{
"<BOS>": 50257,
"<EOS>": 50258,
"<PAD>": 50260,
"<SEP>": 50259
}