SageMaker Model Monitoring

Duration: 5 min

This module covers Amazon SageMaker Model Monitoring, a crucial component for maintaining the performance and reliability of machine learning models in production. It explains how to set up and configure model monitoring, interpret monitoring results, and take corrective actions when necessary. Understanding this module is essential for ensuring that your ML models continue to deliver accurate predictions over time.

Setting Up Model Monitoring

Amazon SageMaker Model Monitoring allows you to monitor the quality of machine learning models in production. You can set up monitoring schedules to automatically detect deviations in model performance. This involves defining what metrics to monitor, how often to run the monitoring jobs, and what thresholds to use for alerting.

import boto3
from sagemaker.model_monitor import DefaultModelMonitor
from sagemaker.model_monitor.dataset_format import DatasetFormat

# Initialize the boto3 client
sagemaker_client = boto3.client('sagemaker')

# Define the model monitor
monitor = DefaultModelMonitor(
    role='AmazonSageMaker-ExecutionRole',
    instance_count=1,
    instance_type='ml.m5.large',
    volume_size_in_gb=20,
    max_runtime_in_seconds=3600,
)

# Set up the monitoring schedule
monitor.create_monitoring_schedule(
    monitor_schedule_name='example-monitoring-schedule',
    endpoint_input='example-endpoint',
    output_s3_uri='s3://example-bucket/model-monitor-output',
    statistics='s3://example-bucket/baseline-statistics',
    constraints='s3://example-bucket/constraints',
    schedule_cron_expression='0 0 * * *',
    dataset_format=DatasetFormat.csv(header=True)
)

Try it in Google Colab:

{'MonitoringScheduleArn': 'arn:aws:sagemaker:us-west-2:123456789012:monitoring-schedule/example-monitoring-schedule'}

Interpreting Monitoring Results

Once the monitoring schedule is set up, SageMaker will periodically run monitoring jobs and store the results in the specified S3 bucket. You can interpret these results to understand if your model is drifting or if there are any anomalies in the data. This involves reviewing the constraint violation reports and taking appropriate actions if the model performance degrades.

import boto3
import pandas as pd

# Initialize the boto3 client
s3_client = boto3.client('s3')

# Download the monitoring results
response = s3_client.get_object(Bucket='example-bucket', Key='model-monitor-output/example-monitoring-schedule/output.csv')
csv_content = response['Body'].read().decode('utf-8')

# Load the results into a pandas DataFrame
df = pd.read_csv(pd.compat.StringIO(csv_content))

# Display the results
print(df.head())

💡 Tip: Always ensure that your monitoring schedule aligns with your data update frequency to catch drifts as early as possible.

Key Concepts

Check Your Understanding