S3 & Storage

Duration: 55 min

Amazon Simple Storage Service (S3) is AWS's object storage service. It's highly scalable, durable, and cost-effective for storing any amount of data. S3 is used for backups, data lakes, static website hosting, and application data. This module covers buckets, objects, versioning, and lifecycle policies.

S3 Buckets and Objects

An S3 bucket is a container for objects. Bucket names must be globally unique across all AWS accounts. Objects are files stored in buckets, identified by a key (path).

S3 provides 99.999999999% (11 nines) durability and 99.99% availability. Data is automatically replicated across multiple availability zones.

Storage Classes

S3 offers different storage classes for different access patterns and cost optimization:

S3 Standard is for frequently accessed data. Highest cost per GB but no retrieval fees.

S3 Intelligent-Tiering automatically moves objects between access tiers based on usage patterns.

S3 Standard-IA (Infrequent Access) is cheaper but has retrieval fees. Use for data accessed less than once per month.

S3 Glacier is for long-term archival. Very cheap but retrieval takes hours.

S3 Glacier Deep Archive is the cheapest option for data rarely accessed.

Versioning and Lifecycle Policies

Versioning keeps multiple versions of objects. When you upload a new version, the old version is retained. This protects against accidental deletion and allows rollback.

Lifecycle policies automatically transition objects between storage classes or delete them based on age. For example, move objects to Glacier after 90 days.

Hands-On: Create and Manage S3 Bucket

Create a bucket:

aws s3 mb s3://my-unique-bucket-name

Upload a file:

aws s3 cp myfile.txt s3://my-unique-bucket-name/

List bucket contents:

aws s3 ls s3://my-unique-bucket-name/

Enable versioning:

aws s3api put-bucket-versioning --bucket my-unique-bucket-name \
  --versioning-configuration Status=Enabled

Create a lifecycle policy:

aws s3api put-bucket-lifecycle-configuration --bucket my-unique-bucket-name \
  --lifecycle-configuration '{
    "Rules": [
      {
        "Id": "archive-rule",
        "Status": "Enabled",
        "Transitions": [
          {
            "Days": 90,
            "StorageClass": "GLACIER"
          }
        ]
      }
    ]
  }'

Python Boto3 Example

import boto3

s3 = boto3.client('s3')

# Create bucket
s3.create_bucket(Bucket='my-unique-bucket-name')

# Upload file
s3.upload_file('myfile.txt', 'my-unique-bucket-name', 'myfile.txt')

# List objects
response = s3.list_objects_v2(Bucket='my-unique-bucket-name')
for obj in response.get('Contents', []):
    print(f"Key: {obj['Key']}, Size: {obj['Size']}")

# Download file
s3.download_file('my-unique-bucket-name', 'myfile.txt', 'downloaded.txt')

# Delete object
s3.delete_object(Bucket='my-unique-bucket-name', Key='myfile.txt')

Terraform Example

resource "aws_s3_bucket" "data" {
  bucket = "my-unique-bucket-name"
}

resource "aws_s3_bucket_versioning" "data" {
  bucket = aws_s3_bucket.data.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "data" {
  bucket = aws_s3_bucket.data.id

  rule {
    id     = "archive-rule"
    status = "Enabled"

    transition {
      days          = 90
      storage_class = "GLACIER"
    }
  }
}

resource "aws_s3_object" "file" {
  bucket = aws_s3_bucket.data.id
  key    = "myfile.txt"
  source = "myfile.txt"
}

S3 Access Control

S3 buckets are private by default. Use bucket policies to grant access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-unique-bucket-name/*"
    }
  ]
}

Quiz 1

Quiz 2

Quiz 3

Quiz 4

Quiz 5