PyTorch
English
roberta
toxic comments classification
leondz commited on
Commit
64f2796
·
verified ·
1 Parent(s): 8e2652f

Upload 7 files

Browse files
README.md CHANGED
@@ -1,3 +1,61 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - toxic comments classification
6
+ licenses:
7
+ - cc-by-nc-sa
8
+ license: openrail++
9
+ base_model:
10
+ - FacebookAI/roberta-large
11
+ datasets:
12
+ - google/jigsaw_toxicity_pred
13
+ ---
14
+
15
+ ## Toxicity Classification Model
16
+
17
+ This model is trained for toxicity classification task. The dataset used for training is the merge of the English parts of the three datasets by **Jigsaw** ([Jigsaw 2018](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge), [Jigsaw 2019](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification), [Jigsaw 2020](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification)), containing around 2 million examples. We split it into two parts and fine-tune a RoBERTa model ([RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692)) on it. The classifiers perform closely on the test set of the first Jigsaw competition, reaching the **AUC-ROC** of 0.98 and **F1-score** of 0.76.
18
+
19
+ ## How to use
20
+ ```python
21
+ import torch
22
+ from transformers import RobertaTokenizer, RobertaForSequenceClassification
23
+
24
+ tokenizer = RobertaTokenizer.from_pretrained('s-nlp/roberta_toxicity_classifier')
25
+ model = RobertaForSequenceClassification.from_pretrained('s-nlp/roberta_toxicity_classifier')
26
+
27
+ batch = tokenizer.encode("You are amazing!", return_tensors="pt")
28
+
29
+ output = model(batch)
30
+ # idx 0 for neutral, idx 1 for toxic
31
+ ```
32
+
33
+ ## Citation
34
+
35
+ To acknowledge our work, please, use the corresponding citation:
36
+
37
+ ```
38
+ @inproceedings{logacheva-etal-2022-paradetox,
39
+ title = "{P}ara{D}etox: Detoxification with Parallel Data",
40
+ author = "Logacheva, Varvara and
41
+ Dementieva, Daryna and
42
+ Ustyantsev, Sergey and
43
+ Moskovskiy, Daniil and
44
+ Dale, David and
45
+ Krotova, Irina and
46
+ Semenov, Nikita and
47
+ Panchenko, Alexander",
48
+ booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
49
+ month = may,
50
+ year = "2022",
51
+ address = "Dublin, Ireland",
52
+ publisher = "Association for Computational Linguistics",
53
+ url = "https://aclanthology.org/2022.acl-long.469",
54
+ pages = "6804--6818",
55
+ abstract = "We present a novel pipeline for the collection of parallel data for the detoxification task. We collect non-toxic paraphrases for over 10,000 English toxic sentences. We also show that this pipeline can be used to distill a large existing corpus of paraphrases to get toxic-neutral sentence pairs. We release two parallel corpora which can be used for the training of detoxification models. To the best of our knowledge, these are the first parallel datasets for this task.We describe our pipeline in detail to make it fast to set up for a new language or domain, thus contributing to faster and easier development of new parallel resources.We train several detoxification models on the collected data and compare them with several baselines and state-of-the-art unsupervised approaches. We conduct both automatic and manual evaluations. All models trained on parallel data outperform the state-of-the-art unsupervised models by a large margin. This suggests that our novel datasets can boost the performance of detoxification systems.",
56
+ }
57
+ ```
58
+
59
+ ## Licensing Information
60
+
61
+ This model is licensed under the OpenRAIL++ License, which supports the development of various technologies—both industrial and academic—that serve the public good.
config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "RobertaForSequenceClassification"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.1,
6
+ "bos_token_id": 0,
7
+ "eos_token_id": 2,
8
+ "gradient_checkpointing": false,
9
+ "hidden_act": "gelu",
10
+ "hidden_dropout_prob": 0.1,
11
+ "hidden_size": 768,
12
+ "id2label": {
13
+ "0": "neutral",
14
+ "1": "toxic"
15
+ },
16
+ "initializer_range": 0.02,
17
+ "intermediate_size": 3072,
18
+ "label2id": {
19
+ "neutral": 0,
20
+ "toxic": 1
21
+ },
22
+ "layer_norm_eps": 1e-05,
23
+ "max_position_embeddings": 514,
24
+ "model_type": "roberta",
25
+ "num_attention_heads": 12,
26
+ "num_hidden_layers": 12,
27
+ "pad_token_id": 1,
28
+ "position_embedding_type": "absolute",
29
+ "problem_type": "single_label_classification",
30
+ "torch_dtype": "float32",
31
+ "transformers_version": "4.8.2",
32
+ "type_vocab_size": 1,
33
+ "vocab_size": 50265
34
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9e5a8b26fd679feca2ee99448a67ef2f0af8f6aade197a8f28bd3088766aa3e
3
+ size 134
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "pad_token": "<pad>", "cls_token": "<s>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": false}}
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"model_max_length": 512}
vocab.json ADDED
The diff for this file is too large to render. See raw diff