Practical Applications of DPO

Duration: 5 min

This module delves into the practical applications of Direct Preference Optimization (DPO) in fine-tuning large language models (LLMs). Understanding DPO is crucial for aligning LLMs with human preferences, enhancing their performance in real-world applications, and ensuring ethical and effective use.

Understanding DPO

Direct Preference Optimization (DPO) is a method used to fine-tune language models based on human preferences. Unlike traditional reinforcement learning from human feedback (RLHF), DPO directly optimizes the model parameters to align with human preferences, making the process more efficient and effective.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load pre-trained model and tokenizer
model_name = 'EleutherAI/gpt-neo-125M'
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Define a simple DPO loss function
def dpo_loss(chosen_logits, rejected_logits):
    return torch.mean(torch.log(torch.sigmoid(chosen_logits - rejected_logits)))

# Example inputs
input_text = 'Once upon a time,'
chosen_text = 'there was a brave knight.'
rejected_text = 'there was a scary dragon.'

# Tokenize inputs
input_ids = tokenizer(input_text, return_tensors='pt').input_ids
chosen_ids = tokenizer(chosen_text, return_tensors='pt').input_ids
rejected_ids = tokenizer(rejected_text, return_tensors='pt').input_ids

# Get logits
with torch.no_grad():
    chosen_logits = model(input_ids, labels=chosen_ids).logits
    rejected_logits = model(input_ids, labels=rejected_ids).logits

# Calculate DPO loss
loss = dpo_loss(chosen_logits, rejected_logits)
print(f'DPO Loss: {loss.item()}')

Try it in Google Colab:

DPO Loss: -0.6931471824645996

Implementing DPO in Practice

To implement DPO in practice, you need to collect human preference data, tokenize the inputs and outputs, compute the logits, and then use the DPO loss function to fine-tune the model. This process involves iterating over the dataset and updating the model parameters to minimize the DPO loss.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, AdamW

# Load pre-trained model and tokenizer
model_name = 'EleutherAI/gpt-neo-125M'
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Define the DPO loss function
def dpo_loss(chosen_logits, rejected_logits):
    return torch.mean(torch.log(torch.sigmoid(chosen_logits - rejected_logits)))

# Example preference dataset
preferences = [
    {"input": 'Once upon a time,', "chosen": 'there was a brave knight.', "rejected": 'there was a scary dragon.'},
    {"input": 'In a galaxy far, far away,', "chosen": 'there was a wise Jedi.', "rejected": 'there was a dark Sith Lord.'}
]

# Fine-tuning loop
optimizer = AdamW(model.parameters(), lr=1e-5)
for epoch in range(3):
    total_loss = 0
    for pref in preferences:
        input_text = pref['input']
        chosen_text = pref['chosen']
        rejected_text = pref['rejected']

        input_ids = tokenizer(input_text, return_tensors='pt').input_ids
        chosen_ids = tokenizer(chosen_text, return_tensors='pt').input_ids
        rejected_ids = tokenizer(rejected_text, return_tensors='pt').input_ids

        chosen_logits = model(input_ids, labels=chosen_ids).logits
        rejected_logits = model(input_ids, labels=rejected_ids).logits

        loss = dpo_loss(chosen_logits, rejected_logits)
        total_loss += loss.item()

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(f'Epoch {epoch+1}, Loss: {total_loss / len(preferences)}')

💡 Tip: Ensure your preference dataset is diverse and representative to avoid bias in the fine-tuned model.