Model Persistence and Deployment

Duration: 45 min

Training a model takes time and compute. Once trained, you need to save it for later use. This module covers serializing scikit-learn models, pipelines, and managing versions. We'll cover joblib (recommended), pickle, ONNX export, and deployment patterns.

Why Model Persistence?

Reproducibility: Same model, same predictions

Avoid retraining: Expensive computations saved

Versioning: Track model changes over time

Deployment: Move models to production systems

Collaboration: Share trained models with teammates

Joblib: The Scikit-Learn Way

Joblib is optimized for NumPy-heavy objects like scikit-learn models. It's faster and more reliable than pickle.

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import joblib
Train a model
X, y = load_iris(return_X_y=True)
model = RandomForestClassifier(n_estimators=10, random_state=42)
model.fit(X, y)
Save to disk
joblib.dump(model, 'iris_model.pkl')
Load from disk
loaded_model = joblib.load('iris_model.pkl')
Verify it works
predictions = loaded_model.predict(X[:5])
print(f"Predictions: {predictions}")

Joblib is the standard for scikit-learn models.

Saving Complex Pipelines

Pipelines with multiple steps serialize seamlessly:

from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
Complex pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('feature_selection', SelectKBest(f_classif, k=3)),
    ('model', LogisticRegression(max_iter=200, random_state=42))
])
Train
pipeline.fit(X, y)
Save
joblib.dump(pipeline, 'iris_pipeline.pkl')
Load and use
loaded_pipeline = joblib.load('iris_pipeline.pkl')
new_predictions = loaded_pipeline.predict(X[:5])
print(f"Pipeline predictions: {new_predictions}")

The entire pipeline, including all hyperparameters and learned coefficients, is saved.

Pickle: Built-In Serialization

Python's pickle module works with scikit-learn but is slower and less reliable than joblib for NumPy objects:

import pickle
Save with pickle
with open('model.pickle', 'wb') as f:
    pickle.dump(model, f)
Load with pickle
with open('model.pickle', 'rb') as f:
    loaded = pickle.load(f)

For scikit-learn, prefer joblib. Pickle is useful for other Python objects.

Version Tracking

Store model version and scikit-learn version alongside the model:

import joblib
import sklearn
Create metadata
metadata = {
    'model_type': 'RandomForestClassifier',
    'sklearn_version': sklearn.__version__,
    'model_version': '1.0.0',
    'training_date': '2024-01-15',
    'train_accuracy': 0.95,
    'test_accuracy': 0.92,
    'hyperparameters': {
        'n_estimators': 10,
        'max_depth': None,
        'random_state': 42
    }
}
Save model and metadata together
joblib.dump({'model': model, 'metadata': metadata}, 'iris_v1.0.0.pkl')
Load and check
saved_data = joblib.load('iris_v1.0.0.pkl')
print(f"Model version: {saved_data['metadata']['model_version']}")
print(f"Sklearn version used: {saved_data['metadata']['sklearn_version']}")
print(f"Test accuracy: {saved_data['metadata']['test_accuracy']}")

Version tracking prevents using incompatible models.

ONNX Export for Production

ONNX (Open Neural Network Exchange) is a standard format for models. Export scikit-learn to ONNX for interoperability:

Requires: pip install skl2onnx onnxruntime
try:
    from skl2onnx import convert_sklearn
    from skl2onnx.common.data_types import FloatTensorType
    import onnx
    import onnxruntime as rt
    
    # Train simple model
    from sklearn.linear_model import LogisticRegression
    model = LogisticRegression(random_state=42)
    model.fit(X, y)
    
    # Convert to ONNX
    initial_type = [('float_input', FloatTensorType([None, X.shape[1]]))]
    onnx_model = convert_sklearn(model, initial_types=initial_type)
    
    # Save ONNX model
    with open('iris_model.onnx', 'wb') as f:
        f.write(onnx_model.SerializeToString())
    
    # Load and use ONNX model
    sess = rt.InferenceSession('iris_model.onnx')
    input_name = sess.get_inputs()[0].name
    onnx_predictions = sess.run(None, {input_name: X[:5].astype('float32')})
    print(f"ONNX predictions: {onnx_predictions[0]}")
    
except ImportError:
    print("Install ONNX tools: pip install skl2onnx onnxruntime")

ONNX enables using scikit-learn models in Java, C++, JavaScript, and other languages.

Handling Model Versioning in Production

Pattern for managing multiple model versions:

import os
from datetime import datetime
class ModelRegistry:
    def __init__(self, model_dir='models'):
        self.model_dir = model_dir
        os.makedirs(model_dir, exist_ok=True)
    
    def save_model(self, model, model_name, version=None):
        if version is None:
            version = datetime.now().strftime('%Y%m%d_%H%M%S')
        
        path = os.path.join(self.model_dir, f'{model_name}_v{version}.pkl')
        joblib.dump(model, path)
        return path
    
    def load_latest_model(self, model_name):
        files = [f for f in os.listdir(self.model_dir) if f.startswith(model_name)]
        if not files:
            raise FileNotFoundError(f"No model found for {model_name}")
        
        latest = sorted(files)[-1]
        path = os.path.join(self.model_dir, latest)
        return joblib.load(path)
    
    def load_model_version(self, model_name, version):
        path = os.path.join(self.model_dir, f'{model_name}_v{version}.pkl')
        return joblib.load(path)
Usage
registry = ModelRegistry()
registry.save_model(model, 'iris_rf', version='1.0.0')
loaded = registry.load_latest_model('iris_rf')

This pattern tracks multiple versions and makes rollbacks easy.

Understanding Serialization Formats

Joblib and pickle are Python-specific. If you need to use models in production systems written in other languages (Java, C++, JavaScript), this becomes a limitation. ONNX solves this by providing a language-agnostic format. A model exported to ONNX can run in any framework supporting ONNX: TensorFlow, PyTorch, Core ML, TensorFlow Lite, and many others.

ONNX represents models as computation graphs with standardized operators. Converting to ONNX requires representing your sklearn operations as ONNX operators. Not all sklearn models are supported in ONNX yet, but most common ones are: linear models, tree-based models, SVMs, and clustering algorithms.

Environment and Dependencies

A model trained in environment A might not load in environment B if the sklearn version differs. Breaking changes between major versions (1.x to 2.x) can render saved models incompatible. This is why version tracking matters enormously.

Document your environment:

scikit-learn==1.3.2
numpy==1.24.3
scipy==1.11.1

Use a requirements.txt or environment.yml file. When deploying, recreate this exact environment to ensure models behave identically to development.

Handling Model Updates and A/B Testing

In production, you need to transition from an old model to a new one without downtime. The ModelRegistry pattern enables this: deploy the new model, route a percentage of traffic to it, monitor performance, gradually increase traffic, then fully cutover.

Store models with timestamps or semantic versions. This enables rollback if a new model performs poorly in production. Query the registry for the "current_production" version, allowing instant rollbacks by changing a configuration flag.

Performance Implications

Model size directly affects deployment speed. A 1GB model takes seconds to load; a 10MB model loads instantly. Quantization (reducing precision from float64 to float32 or int8) reduces model size with minimal accuracy loss. Use dtype parameter in joblib to control precision.

In high-throughput systems, loading and prediction time matter. Batch predictions are faster than individual predictions. If your system receives 1000 requests per second, collecting them into batches before prediction can reduce latency and increase throughput dramatically.

Version Compatibility Matrix

When you train a model, scikit-learn version, Python version, and library versions all matter. Different versions might implement algorithms slightly differently (different random seeds, optimization improvements, bug fixes). This shouldn't matter for reproducibility within the same version, but when upgrading, test thoroughly.

Create a compatibility matrix:

Model Version | sklearn | Python | Test Accuracy
v1.0          | 1.3.2   | 3.11   | 0.952
v1.1          | 1.4.0   | 3.11   | 0.954

Model Serving Infrastructure

The choice of serialization format affects your infrastructure. If using joblib in a Python web service (Flask, FastAPI), load the model once at startup, then call predict() for each request. If using ONNX, consider ONNX Runtime (optimized inference engine) or deploying via TensorFlow Serving, which handles model versioning, A/B testing, and canary deployments automatically.

Handling Model Versioning in Production

Random forests are large; limit depth and trees
compact_model = RandomForestClassifier(
    n_estimators=5,      # Fewer trees
    max_depth=5,         # Shallower trees
    random_state=42
)
compact_model.fit(X, y)
Compare sizes
import os
joblib.dump(model, 'large_model.pkl')
joblib.dump(compact_model, 'compact_model.pkl')
large_size = os.path.getsize('large_model.pkl')
compact_size = os.path.getsize('compact_model.pkl')
Model loading overhead (deserializing from disk) can be significant for large models. A 100MB model takes ~100ms to load. Serverless functions initializing on every cold start see this overhead. Strategies:
1. Keep models in memory across requests (works for webservers)
2. Pre-warm models in serverless (scheduled requests to maintain warm instances)
3. Use smaller models (faster loading)
4. Use faster serialization (ONNX Runtime is 2-3x faster than joblib for some models)
Profiling production deployments reveals these bottlenecks. Log model loading time, inference time, and end-to-end request latency to identify optimization targets.
Monitoring Model Performance Drift
Production models degrade over time as data distribution changes. Implement monitoring:

python

Track prediction statistics

prediction_stats = { 'timestamp': datetime.now(), 'mean_prediction': predictions.mean(), 'std_prediction': predictions.std(), 'n_predictions': len(predictions) }

Compare with baseline

baseline_mean = 0.65 # From training if abs(prediction_stats['mean_prediction'] - baseline_mean) > 0.1: print("WARNING: Mean prediction shifted significantly") # Retrain or investigate

When performance drifts, retrain the model on recent data. Automate this: trigger retraining when metrics drop below thresholds or after specific time intervals. A/B Testing and Canary Deployments Never deploy directly to all users. Use: 1. Canary: 5% traffic to new model, 95% to old. If new performs better for 24 hours, increase to 50%, then 100%. 2. A/B test: 50/50 split for statistical significance testing. 3. Shadow: Run new model in parallel but don't use predictions (detect issues without affecting users). This reduces risk. If the new model has bugs, you catch them with minimal user impact.

print(f"Large model: {large_size / 1024:.1f} KB") print(f"Compact model: {compact_size / 1024:.1f} KB") print(f"Reduction: {(1 - compact_size/large_size)*100:.1f}%")

Smaller models deploy faster and use less memory.

Model Compression

For deployment, reduce model size:

Random forests are large; limit depth and trees
compact_model = RandomForestClassifier(
    n_estimators=5,      # Fewer trees
    max_depth=5,         # Shallower trees
    random_state=42
)
compact_model.fit(X, y)
Compare sizes
import os
joblib.dump(model, 'large_model.pkl')
joblib.dump(compact_model, 'compact_model.pkl')large_size = os.path.getsize('large_model.pkl')
compact_size = os.path.getsize('compact_model.pkl')
print(f"Large model: {large_size / 1024:.1f} KB")
print(f"Compact model: {compact_size / 1024:.1f} KB")
print(f"Reduction: {(1 - compact_size/large_size)*100:.1f}%")

Smaller models deploy faster and use less memory. There's a trade-off: trading 1-2% accuracy for 10x smaller model often makes sense in production. A model that loads in 100ms vs 10 seconds creates vastly different user experiences. Test this trade-off with your deployment constraints.

Deployment Patterns

Local inference: Models loaded in Python/Java/Node processes. Lowest latency but per-instance resource usage.

Microservice: Dedicated model serving service (TensorFlow Serving, Seldon, KServe). Scales independently, supports versioning and A/B testing.

Serverless: AWS Lambda, Google Cloud Functions with models. Scales automatically but cold start latency can be problematic.

Edge: Models deployed on mobile or IoT devices. Requires extreme compression (distillation, quantization).

Choose based on latency requirements, throughput, and infrastructure constraints.

Understanding Version Compatibility

A model trained with sklearn 1.0, Python 3.9, numpy 1.19 might produce slightly different predictions in sklearn 1.2, Python 3.11, numpy 1.23. These differences are usually small but cumulative in complex pipelines. For critical applications, maintain version locks.

Model Serving Infrastructure

High-throughput systems benefit from batch prediction: collecting multiple requests, predicting on batches (faster than individual predictions), then returning results. This requires queuing and careful latency management.

Cold Start and Model Loading Overhead

1. Why is joblib preferred over pickle for scikit-learn? A) It's always faster B) It handles NumPy objects better and is more reliable ✓ C) It produces smaller files D) It works with ONNX

2. What should be stored with a serialized model? A) Training data B) Feature names and metadata ✓ C) Cross-validation results D) Raw source code

3. What does ONNX enable? A) Faster model training B) Using models across different programming languages and frameworks ✓ C) Automatic hyperparameter tuning D) Data versioning

4. In the ModelRegistry pattern, why load the "latest" model? A) It's always more accurate B) It ensures you use the most recent version in production ✓ C) It prevents memory leaks D) It's faster than loading specific versions

5. When should you compress a model? A) Always B) Only for neural networks C) When deploying to resource-constrained environments ✓ D) Never; accuracy always matters

6. What information is crucial to store with version tracking? A) Only model type B) Sklearn version, hyperparameters, and training/test metrics ✓ C) Only training date D) Model predictions on all training data

Quiz

1. What is the main advantage of joblib over pickle for scikit-learn models? A) It's always faster B) It handles NumPy objects more efficiently and reliably ✓ C) It produces smaller files D) It works with ONNX automatically

2. Why should you store version metadata with a serialized model? A) For documentation purposes only B) To enable rollback and ensure compatibility across environments ✓ C) To improve model accuracy D) It's optional but nice-to-have

3. What does ONNX enable for machine learning models? A) Faster training B) Automatic hyperparameter tuning C) Using models across different programming languages and frameworks ✓ D) Better data visualization

4. In the ModelRegistry pattern, what is the primary benefit of version tracking? A) It makes code shorter B) It enables easy rollback and production management ✓ C) It improves model performance D) It reduces memory usage

5. When should you compress a model for deployment? A) Always; smaller is always better B) When deploying to resource-constrained environments ✓ C) Only for neural networks D) Never; accuracy always takes priority

6. What information should be tracked with every saved model? A) Just the sklearn version B) Sklearn version, hyperparameters, training date, and performance metrics ✓ C) Only performance metrics D) Nothing beyond the model itself