Linux & Shell Scripting for DevOps

Duration: 120 min

Linux is the foundation of modern DevOps. Most servers, containers, and cloud infrastructure run Linux. Shell scripting enables automation of repetitive tasks, system administration, and deployment workflows. Mastering bash and Linux fundamentals is essential for any DevOps engineer.

Linux Fundamentals for DevOps

DevOps engineers need to be comfortable with Linux command-line tools and system administration. Key areas include:

• File system navigation: cd, ls, find, locate
• File operations: cp, mv, rm, chmod, chown
• Text processing: grep, sed, awk, cut, sort
• Process management: ps, top, kill, systemctl
• Networking: ifconfig, netstat, ss, curl, wget
• Package management: apt, yum, dnf depending on distribution

Bash Scripting Essentials

Bash scripts automate infrastructure tasks. A well-written script is idempotent, handles errors gracefully, and logs its actions.

#!/bin/bash
set -euo pipefail  # Exit on error, undefined vars, pipe failures

# Logging function
log() {
  echo "[$(date +'%Y-%m-%d %H:%M:%S')] $*" | tee -a /var/log/script.log
}

# Error handling
trap 'log "Error on line $LINENO"' ERR

log "Starting deployment script"

# Check prerequisites
if ! command -v docker &> /dev/null; then
  log "ERROR: Docker not installed"
  exit 1
fi

# Main logic
docker pull myapp:latest
docker run -d --name myapp myapp:latest

log "Deployment completed successfully"

Cron Jobs for Scheduled Tasks

Cron allows scheduling scripts to run at specific times. The crontab format is: minute hour day month weekday command

# View current crontab
crontab -l

# Edit crontab
crontab -e

# Example crontab entries
0 2 * * * /usr/local/bin/backup.sh              # Daily at 2 AM
*/15 * * * * /usr/local/bin/health-check.sh     # Every 15 minutes
0 0 * * 0 /usr/local/bin/weekly-report.sh       # Weekly on Sunday at midnight

Systemd Services

Systemd manages services on modern Linux systems. DevOps engineers often create custom systemd units for applications.

# Create a systemd service file
cat > /etc/systemd/system/myapp.service << 'EOF'
[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
User=appuser
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/start.sh
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

# Enable and start the service
systemctl daemon-reload
systemctl enable myapp
systemctl start myapp
systemctl status myapp

Advanced Bash Patterns

Error Handling and Validation

#!/bin/bash
set -euo pipefail

# Function with error handling
deploy_app() {
  local app_name=$1
  local version=$2
  
  if [[ -z "$app_name" || -z "$version" ]]; then
    echo "Usage: deploy_app <app_name> <version>"
    return 1
  fi
  
  echo "Deploying $app_name:$version"
  # Deployment logic here
}

# Validate input
if [[ $# -lt 2 ]]; then
  echo "Error: Missing arguments"
  exit 1
fi

deploy_app "$@"

Parallel Execution

#!/bin/bash

# Run multiple tasks in parallel
run_tests() {
  local test_dirs=("unit" "integration" "e2e")
  
  for dir in "${test_dirs[@]}"; do
    (
      cd "tests/$dir"
      ./run.sh
    ) &
  done
  
  # Wait for all background jobs
  wait
  echo "All tests completed"
}

run_tests

AWS CLI Scripting

The AWS CLI is essential for DevOps automation on AWS.

#!/bin/bash

# List all EC2 instances
aws ec2 describe-instances \
  --query 'Reservations[].Instances[].{ID:InstanceId,Type:InstanceType,State:State.Name}' \
  --output table

# Create an S3 bucket with versioning
aws s3api create-bucket \
  --bucket my-backup-bucket \
  --region us-east-1

aws s3api put-bucket-versioning \
  --bucket my-backup-bucket \
  --versioning-configuration Status=Enabled

# Get CloudWatch metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-1234567890abcdef0 \
  --start-time 2024-01-01T00:00:00Z \
  --end-time 2024-01-02T00:00:00Z \
  --period 3600 \
  --statistics Average

Text Processing with grep, sed, awk

These tools are powerful for parsing logs and configuration files.

# grep: Find lines matching a pattern
grep "ERROR" /var/log/app.log
grep -i "warning" /var/log/app.log  # Case-insensitive
grep -c "ERROR" /var/log/app.log    # Count matches

# sed: Stream editor for text transformation
sed 's/old/new/g' config.txt        # Replace all occurrences
sed -i.bak 's/old/new/g' config.txt # In-place edit with backup

# awk: Text processing and data extraction
awk '{print $1, $3}' data.txt       # Print columns 1 and 3
awk -F: '{print $1}' /etc/passwd    # Use colon as delimiter
awk '$2 > 100 {print $1}' data.txt  # Conditional printing

Monitoring and Logging

DevOps scripts should produce clear logs for troubleshooting.

#!/bin/bash

LOG_FILE="/var/log/deployment.log"
ERROR_LOG="/var/log/deployment-errors.log"

log_info() {
  echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') - $*" | tee -a "$LOG_FILE"
}

log_error() {
  echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') - $*" | tee -a "$ERROR_LOG" >&2
}

# Usage
log_info "Starting deployment"
if ! docker pull myapp:latest; then
  log_error "Failed to pull Docker image"
  exit 1
fi
log_info "Deployment completed"