NLP中的迁移学习：利用预训练模型进行文本分类

蟒

 

 

 

 

1

from transformers import TextDataset

2

def tokenize(batch):

3

    return tokenizer(batch['text'], padding=True, truncation=True)

4

train_dataset = TextDataset("path/to/train_data", tokenize)

5

val_dataset = TextDataset("path/to/val_data", tokenize)