deepseek-ai

/

DeepSeek-R1

like 5.24k

Follow

DeepSeek 20.9k

我使用的闭源数据集，因此数据相关部分就省略代替。微调模型使用的是 distill qwen 7b。短暂训练后出现模型无限输出（重复一小段话，然后出现乱码，多为问号和句号），不出 eos 的情况。

训练采用 trl.SFTTrainer，数据集预处理后采用 trl 可接受的 chat template 输入格式，即 {"prompt": "xxx", "completion": "xxx"}，trl 会使用 tokenizer.apply_chat_template 预处理。数据不含 thinking 信息，仅有 prompt 和 answer。检查了训练集在 apply chat template 之后包含 eos token，tokenize 之后也存在 151643。

样例代码：

MODEL_NAME = "./data/DeepSeek-R1-Distill-Qwen-7B"

args = SFTConfig(
    output_dir=OUTPUT_DIR,
    do_train=True,
    logging_first_step=True,
    logging_dir=LOG_DIR,
    logging_steps=100,
    save_strategy=IntervalStrategy.EPOCH,
    save_steps=1,
    num_train_epochs=TRAIN_EPOCHS,
    optim=OptimizerNames.ADAMW_TORCH,
    per_device_train_batch_size=BATCH_SIZE,
    gradient_accumulation_steps=GRADIENT_ACCUMULATION_STEPS,
    learning_rate=LEARNING_RATE,
    weight_decay=WEIGHT_DECAY,
    warmup_ratio=WARMUP_RATIO,
    max_seq_length=1024,
)

with args.main_process_first(local=False, desc="loading tokenizer"):
    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

with args.main_process_first(local=False, desc="loading model"):
    model = Qwen2ForCausalLM.from_pretrained(MODEL_NAME, torch_dtype="auto", device_map="auto")

def preprocess_dataset(batch: dict[str, list[str]]):
    return {"prompt": batch["input"], "completion": batch["target"]}

with args.main_process_first(local=False, desc="loading dataset"):
    train_dataset = Dataset()  # load dataset here
    # preprocess
    train_dataset = train_dataset.map(
        preprocess_dataset, batched=True, remove_columns=train_dataset.column_names
    )

trainer = SFTTrainer(
    model,
    args=args,
    processing_class=tokenizer,
    train_dataset=train_dataset,
)

if __name__ == "__main__":
    print("Start training...")  # noqa: T201
    trainer.train()