Overview of Transformers

Duration: 6 min

This module delves into the fascinating world of Transformers, a groundbreaking architecture in Natural Language Processing (NLP). We will explore how Transformers, particularly BERT, have revolutionized text understanding and generation. Understanding this technology is crucial for anyone looking to advance in the field of AI and machine learning.

Understanding Transformers

Transformers are a type of deep learning model that leverage self-attention mechanisms to process input data. Unlike traditional recurrent neural networks (RNNs), Transformers can process entire sequences of data simultaneously, making them highly efficient for tasks like language translation and text summarization.

from transformers import BertTokenizer, BertModel
import torch

# Initialize BERT tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

# Encode a sample text
inputs = tokenizer('Hello, how are you?', return_tensors='pt')
outputs = model(**inputs)

# Print the last hidden state
print(outputs.last_hidden_state)

Try it in Google Colab:

tensor([[[-0.0155, -0.6825,  0.4970, ...,  0.0798, -0.2892,  0.0420],
         [-0.0155, -0.6825,  0.4970, ...,  0.0798, -0.2892,  0.0420],
         [-0.0155, -0.6825,  0.4970, ...,  0.0798, -0.2892,  0.0420],
        ...,
         [ 0.0000,  0.0000,  0.0000, ...,  0.0000,  0.0000,  0.0000],
         [ 0.0000,  0.0000,  0.0000, ...,  0.0000,  0.0000,  0.0000],
         [ 0.0000,  0.0000,  0.0000, ...,  0.0000,  0.0000,  0.0000]]], grad_fn=<UnsqueezeBackward0>)

Fine-tuning Transformers

Fine-tuning involves taking a pre-trained model and adapting it to a specific task. This is particularly useful when you have a limited amount of data for the target task. Hugging Face's Transformers library makes fine-tuning straightforward, allowing you to leverage powerful pre-trained models like BERT.

from transformers import BertForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset

# Load a dataset
dataset = load_dataset('imdb')

# Initialize the model
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

# Define training arguments
training_args = TrainingArguments(
    output_dir='./results',
    evaluation_strategy='epoch',
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=3,
    weight_decay=0.01
)

# Initialize the Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset['train'],
    eval_dataset=dataset['test']
)

# Train the model
trainer.train()

💡 Tip: When fine-tuning a model, ensure that your dataset is balanced and representative of the task to avoid biased results.