DataCollatorForLanguageModeling uses the unmasked labels

[shuiyueche]

In finetune.py, the Line 226 uses the tranformers.DataCollatorForLanguageModeling. This data collator actually ignores the "labels" field in line 185 and uses "input_ids" as the new labels in the trainer.train() function. You can see this behavior in the Line 745 of transformers.data_collator

Compared against the implementation of stanford_alpaca, its train.py uses the masked "labels" as the line 122~123 shows (This is the same purpose as your "labels" in line 185). But since the code write its own dataset and collator. The "labels" is not overwritten later in the training.

I wonder if the use of tranformers.DataCollatorForLanguageModeling in finetune.py is a bug. Does it serve the purpose here?