Understanding BERT

Duration: 8 min

This module delves into the intricacies of BERT, a groundbreaking model in the field of Natural Language Processing. We will explore its architecture, understand its significance, and learn how to leverage it for various NLP tasks. Understanding BERT is crucial for anyone looking to stay at the forefront of NLP research and applications.

BERT Architecture

BERT, or Bidirectional Encoder Representations from Transformers, is a method of pre-training language representations that can be fine-tuned to a wide range of downstream tasks. It uses a bidirectional training approach, allowing it to understand context from both directions, unlike traditional unidirectional models.

from transformers import BertTokenizer, BertModel
import torch

# Initialize BERT tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

# Tokenize input text
inputs = tokenizer("Hello, how are you?", return_tensors='pt')

# Get model outputs
outputs = model(**inputs)

# Print the last hidden states
print(outputs.last_hidden_state)

Try it in Google Colab:

tensor([[[-0.0162, -0.6005,  0.2985, ...,  0.0796,  0.2293,  0.1186],
         [-0.0703, -0.4794,  0.3431, ...,  0.1182,  0.2698,  0.1436],
         [-0.0162, -0.6005,  0.2985, ...,  0.0796,  0.2293,  0.1186],
        ...,
         [ 0.0000,  0.0000,  0.0000, ...,  0.0000,  0.0000,  0.0000],
         [ 0.0000,  0.0000,  0.0000, ...,  0.0000,  0.0000,  0.0000],
         [ 0.0000,  0.0000,  0.0000, ...,  0.0000,  0.0000,  0.0000]]], device='cuda:0')

Fine-tuning BERT

Fine-tuning BERT involves taking a pre-trained BERT model and training it further on a specific task, such as sentiment analysis or named entity recognition. This process allows the model to adapt to the nuances of the new task while leveraging the knowledge it gained during pre-training.

from transformers import BertForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset

# Load a dataset
dataset = load_dataset('glue','mrpc')

# Initialize BERT model for sequence classification
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

# Define training arguments
training_args = TrainingArguments(
    output_dir='./results',
    evaluation_strategy = 'epoch',
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=64,
    num_train_epochs=3,
    weight_decay=0.01,
)

# Initialize Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset['train'],
    eval_dataset=dataset['validation']
)

# Train the model
trainer.train()

💡 Tip: When fine-tuning BERT, ensure that your dataset is balanced and representative of the task to avoid bias in the model's predictions.