BLOG POSTS

MangoHost Blog / How to Automatically Update Docker Container Images with Watchtower on Ubuntu 24

How to Automatically Update Docker Container Images with Watchtower on Ubuntu 24

Managing Docker containers manually is about as fun as watching paint dry, especially when you’re dealing with production environments that need regular updates. Watchtower automates the entire process of updating Docker container images, monitoring your running containers for newer versions and seamlessly replacing them without breaking your deployment flow. This guide walks you through setting up Watchtower on Ubuntu 24, configuring automatic updates, handling rollbacks, and avoiding the common gotchas that can turn your smooth automation into a debugging nightmare.

How Watchtower Works Under the Hood

Watchtower operates as a Docker container itself, which is pretty clever when you think about it. It connects to the Docker daemon via the Docker socket and periodically polls container registries to check for updated images. When it finds a newer version, Watchtower pulls the new image, gracefully stops the existing container, and starts a replacement with the same configuration.

The polling mechanism uses image digests and tags to determine if updates are available. By default, Watchtower checks every 24 hours, but you can configure this interval based on your needs. It maintains the original container’s environment variables, port mappings, volumes, and network settings, making the update process transparent to your applications.

Here’s what happens during a typical update cycle:

Watchtower queries the registry for the latest image digest
Compares the digest with the currently running container’s image
Downloads the new image if a difference is detected
Stops the old container gracefully (sends SIGTERM, waits for cleanup)
Creates and starts a new container with identical configuration
Optionally removes the old image to save disk space

Step-by-Step Setup Guide for Ubuntu 24

First, make sure Docker is properly installed and your user has appropriate permissions. If you haven’t set this up yet, here’s the quick version:

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER
newgrp docker

Now let’s deploy Watchtower. The basic setup requires mounting the Docker socket so Watchtower can communicate with the Docker daemon:

docker run -d \
  --name watchtower \
  --restart unless-stopped \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower

This gives you the default behavior: checking all containers every 24 hours. For production environments, you’ll want more control. Here’s a more comprehensive configuration:

docker run -d \
  --name watchtower \
  --restart unless-stopped \
  -e WATCHTOWER_POLL_INTERVAL=3600 \
  -e WATCHTOWER_CLEANUP=true \
  -e WATCHTOWER_INCLUDE_RESTARTING=true \
  -e WATCHTOWER_NOTIFICATIONS_LEVEL=info \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower

For even better control, create a docker-compose.yml file:

version: '3.8'
services:
  watchtower:
    image: containrrr/watchtower
    container_name: watchtower
    restart: unless-stopped
    environment:
      - WATCHTOWER_POLL_INTERVAL=3600
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_INCLUDE_RESTARTING=true
      - WATCHTOWER_NOTIFICATIONS_LEVEL=info
      - WATCHTOWER_TIMEOUT=30s
      - WATCHTOWER_ROLLING_RESTART=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: --interval 3600 --cleanup

Deploy it with:

docker-compose up -d

Configuration Options and Fine-Tuning

Watchtower offers extensive configuration through environment variables and command-line arguments. Here are the most useful ones for real-world deployments:

Environment Variable	Description	Default Value	Use Case
WATCHTOWER_POLL_INTERVAL	Check interval in seconds	86400 (24h)	Frequent updates for dev, longer for prod
WATCHTOWER_CLEANUP	Remove old images after update	false	Prevent disk space issues
WATCHTOWER_TIMEOUT	Stop timeout for containers	10s	Graceful shutdown for databases
WATCHTOWER_ROLLING_RESTART	Update containers one by one	false	Maintain service availability
WATCHTOWER_INCLUDE_STOPPED	Update stopped containers	false	Keep all images current

You can also target specific containers using labels. Add this label to containers you want Watchtower to monitor:

docker run -d \
  --label com.centurylinklabs.watchtower.enable=true \
  --name myapp \
  nginx:latest

Then configure Watchtower to only watch labeled containers:

docker run -d \
  --name watchtower \
  --restart unless-stopped \
  -e WATCHTOWER_LABEL_ENABLE=true \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower

Real-World Use Cases and Examples

Here are some practical scenarios where Watchtower shines:

Development Environment Auto-Updates: Set up a short polling interval for development containers that need to stay current with the latest builds:

version: '3.8'
services:
  app:
    image: myregistry/myapp:latest
    labels:
      - com.centurylinklabs.watchtower.enable=true
    ports:
      - "3000:3000"
  
  watchtower:
    image: containrrr/watchtower
    environment:
      - WATCHTOWER_POLL_INTERVAL=300  # 5 minutes
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_LABEL_ENABLE=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Production Security Updates: Configure Watchtower to only update containers with security patches, using specific tag patterns:

docker run -d \
  --name watchtower-security \
  --restart unless-stopped \
  -e WATCHTOWER_POLL_INTERVAL=7200 \
  -e WATCHTOWER_CLEANUP=true \
  -e WATCHTOWER_ROLLING_RESTART=true \
  -e WATCHTOWER_timeout=60s \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower \
  --include-restarting \
  container1 container2 container3

Notification Integration: Set up Slack notifications for production updates:

docker run -d \
  --name watchtower \
  --restart unless-stopped \
  -e WATCHTOWER_NOTIFICATIONS=slack \
  -e WATCHTOWER_NOTIFICATION_SLACK_HOOK_URL=https://hooks.slack.com/services/... \
  -e WATCHTOWER_NOTIFICATION_SLACK_IDENTIFIER=watchtower-prod \
  -e WATCHTOWER_NOTIFICATION_SLACK_CHANNEL=#deployments \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower

Comparison with Alternative Solutions

Watchtower isn’t the only game in town for container updates. Here’s how it stacks up against other options:

Solution	Complexity	Features	Best For	Limitations
Watchtower	Low	Automatic updates, notifications, filtering	Simple deployments, dev environments	No rollback strategy, limited orchestration
Portainer	Medium	GUI management, manual updates, stacks	Manual control, visual management	No automatic updates without webhooks
Kubernetes	High	Rolling updates, rollbacks, health checks	Production clusters, complex applications	Steep learning curve, resource overhead
Custom CI/CD	High	Full customization, testing integration	Complex workflows, enterprise environments	Requires significant setup and maintenance

For most single-server deployments and development environments, Watchtower provides the best balance of simplicity and functionality. It’s particularly useful when you’re running Docker Compose applications and want automatic updates without the complexity of a full orchestration platform.

Best Practices and Security Considerations

Running Watchtower safely requires following several important practices:

Registry Authentication: For private registries, you’ll need to provide credentials. The cleanest approach uses a config.json file:

mkdir -p ~/.docker
echo '{
  "auths": {
    "your-registry.com": {
      "auth": "base64-encoded-credentials"
    }
  }
}' > ~/.docker/config.json

docker run -d \
  --name watchtower \
  --restart unless-stopped \
  -v ~/.docker/config.json:/config.json \
  -e DOCKER_CONFIG=/config.json \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower

Resource Limits: Always set resource constraints to prevent Watchtower from consuming excessive resources during image pulls:

docker run -d \
  --name watchtower \
  --restart unless-stopped \
  --memory=512m \
  --cpus=0.5 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower

Backup Strategy: Before implementing automatic updates, ensure you have a solid backup and rollback strategy:

#!/bin/bash
# Pre-update backup script
BACKUP_DIR="/backups/$(date +%Y%m%d_%H%M%S)"
mkdir -p $BACKUP_DIR

# Export container configurations
docker ps --format "table {{.Names}}\t{{.Image}}" > $BACKUP_DIR/running_containers.txt

# Backup volumes
docker run --rm -v myapp_data:/data -v $BACKUP_DIR:/backup alpine tar czf /backup/myapp_data.tar.gz -C /data .

echo "Backup completed: $BACKUP_DIR"

Health Check Integration: Use health checks to ensure containers are actually running properly after updates:

version: '3.8'
services:
  app:
    image: myapp:latest
    labels:
      - com.centurylinklabs.watchtower.enable=true
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    ports:
      - "3000:3000"

Common Issues and Troubleshooting

Here are the most frequent problems you’ll encounter and their solutions:

Permission Issues: If Watchtower can’t access the Docker socket, you’ll see permission errors. Fix this by ensuring proper socket permissions:

# Check current permissions
ls -la /var/run/docker.sock

# Fix permissions if needed
sudo chmod 666 /var/run/docker.sock

# Better: add user to docker group
sudo usermod -aG docker $USER

Memory Issues During Updates: Large image pulls can cause out-of-memory errors. Monitor disk space and set up cleanup:

# Check disk usage
docker system df

# Clean up unused images
docker system prune -a

# Set up automatic cleanup in Watchtower
-e WATCHTOWER_CLEANUP=true

Update Failures: When containers fail to start after updates, check the logs:

# Check Watchtower logs
docker logs watchtower --tail 50

# Check the updated container logs
docker logs container_name --tail 50

# Roll back manually if needed
docker stop container_name
docker run -d --name container_name previous_image:tag

Registry Rate Limits: Docker Hub and other registries impose rate limits. If you’re hitting these, implement caching or use authentication:

# Use Docker Hub authentication to increase limits
docker login

# Or set up a local registry cache
docker run -d -p 5000:5000 --name registry-cache \
  -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
  registry:2

Network Connectivity Issues: Watchtower needs internet access to check for updates. Test connectivity and configure proxy settings if needed:

# Test registry connectivity
docker run --rm alpine/curl curl -I https://registry-1.docker.io/v2/

# Configure HTTP proxy for Watchtower
-e HTTP_PROXY=http://proxy.company.com:8080 \
-e HTTPS_PROXY=http://proxy.company.com:8080

Advanced Configuration and Monitoring

For production environments, you’ll want more sophisticated monitoring and control. Here’s how to set up comprehensive logging and monitoring:

version: '3.8'
services:
  watchtower:
    image: containrrr/watchtower
    container_name: watchtower
    restart: unless-stopped
    environment:
      - WATCHTOWER_POLL_INTERVAL=3600
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_INCLUDE_RESTARTING=true
      - WATCHTOWER_NOTIFICATIONS=slack
      - WATCHTOWER_NOTIFICATION_SLACK_HOOK_URL=${SLACK_WEBHOOK}
      - WATCHTOWER_NOTIFICATION_SLACK_CHANNEL=#infrastructure
      - WATCHTOWER_DEBUG=true
      - WATCHTOWER_TRACE=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./watchtower-logs:/logs
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

You can also integrate Watchtower with monitoring tools like Prometheus by parsing its logs or using webhook notifications:

# Webhook notification example
-e WATCHTOWER_NOTIFICATIONS=webhook \
-e WATCHTOWER_NOTIFICATION_WEBHOOK_URL=http://monitoring.local/webhook/watchtower \
-e WATCHTOWER_NOTIFICATION_WEBHOOK_TEMPLATE='{"text":"Container {{.Name}} updated from {{.OldImage}} to {{.NewImage}}"}'

For more advanced use cases, check out the official Watchtower documentation at https://containrrr.dev/watchtower/ and the Docker documentation at https://docs.docker.com/ for additional configuration options and best practices.

Remember that automation is great, but it’s not a substitute for proper testing and deployment practices. Always test updates in a staging environment first, maintain proper backups, and have a rollback plan ready. Watchtower works best as part of a comprehensive deployment strategy, not as a fire-and-forget solution.

This article incorporates information and material from various online sources. We acknowledge and appreciate the work of all original authors, publishers, and websites. While every effort has been made to appropriately credit the source material, any unintentional oversight or omission does not constitute a copyright infringement. All trademarks, logos, and images mentioned are the property of their respective owners. If you believe that any content used in this article infringes upon your copyright, please contact us immediately for review and prompt action.

This article is intended for informational and educational purposes only and does not infringe on the rights of the copyright owners. If any copyrighted material has been used without proper credit or in violation of copyright laws, it is unintentional and we will rectify it promptly upon notification. Please note that the republishing, redistribution, or reproduction of part or all of the contents in any form is prohibited without express written permission from the author and website owner. For permissions or further inquiries, please contact us.