|
--- |
|
license: mit |
|
language: |
|
- en |
|
- de |
|
- es |
|
- fr |
|
- hi |
|
- it |
|
- ja |
|
- ko |
|
- pl |
|
- pt |
|
- ru |
|
- tr |
|
- zh |
|
pipeline_tag: text-to-speech |
|
library: bark |
|
tags: |
|
- bark |
|
- audio |
|
- text-to-speech |
|
inference: true |
|
--- |
|
|
|
# Safetensors files for [Bark](https://huggingface.co/suno/bark) |
|
This repository hosts safetensors files for Suno's Bark. They are directly converted by loading the model, and using `safetensors.torch.save_model`. |
|
These safetensors files only contain the model, and the config has been split to another file with the name `model_name.json`. |
|
|
|
All original model files have been converted and had their configs extracted into json format. |
|
For example, for model `text`, the safetensors file is `text.safetensors`, and the config file is `text.json`. |
|
|
|
# Bark (From [original readme](https://huggingface.co/suno/bark), modified to exclude information not relevant to this repo) |
|
|
|
Bark is a transformer-based text-to-audio model created by [Suno](https://www.suno.ai). |
|
Bark can generate highly realistic, multilingual speech as well as other audio - including music, |
|
background noise and simple sound effects. The model can also produce nonverbal |
|
communications like laughing, sighing and crying. To support the research community, |
|
we are providing access to pretrained model checkpoints ready for inference. |
|
|
|
The original github repo and model card can be found [here](https://github.com/suno-ai/bark). |
|
|
|
This model is meant for research purposes only. |
|
The model output is not censored and the authors do not endorse the opinions in the generated content. |
|
Use at your own risk. |
|
|
|
Two checkpoints are released: |
|
- [small](https://huggingface.co/suno/bark-small) |
|
- [large](https://huggingface.co/suno/bark) |
|
|
|
## Model Details |
|
|
|
|
|
The following is additional information about the models released here. |
|
|
|
Bark is a series of three transformer models that turn text into audio. |
|
|
|
### Text to semantic tokens |
|
- Input: text, tokenized with [BERT tokenizer from Hugging Face](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertTokenizer) |
|
- Output: semantic tokens that encode the audio to be generated |
|
|
|
### Semantic to coarse tokens |
|
- Input: semantic tokens |
|
- Output: tokens from the first two codebooks of the [EnCodec Codec](https://github.com/facebookresearch/encodec) from facebook |
|
|
|
### Coarse to fine tokens |
|
- Input: the first two codebooks from EnCodec |
|
- Output: 8 codebooks from EnCodec |
|
|
|
### Architecture |
|
| Model | Parameters | Attention | Output Vocab size | |
|
|:-------------------------:|:----------:|------------|:-----------------:| |
|
| Text to semantic tokens | 80/300 M | Causal | 10,000 | |
|
| Semantic to coarse tokens | 80/300 M | Causal | 2x 1,024 | |
|
| Coarse to fine tokens | 80/300 M | Non-causal | 6x 1,024 | |
|
|
|
|
|
### Release date |
|
April 2023 |
|
|
|
## Broader Implications |
|
We anticipate that this model's text to audio capabilities can be used to improve accessbility tools in a variety of languages. |
|
|
|
While we hope that this release will enable users to express their creativity and build applications that are a force |
|
for good, we acknowledge that any text to audio model has the potential for dual use. While it is not straightforward |
|
to voice clone known people with Bark, it can still be used for nefarious purposes. To further reduce the chances of unintended use of Bark, |
|
we also release a simple classifier to detect Bark-generated audio with high accuracy (see notebooks section of the main repository). |
