BLOG POSTS

MangoHost Blog / How to Set Up a Continuous Deployment Pipeline with GitLab on Ubuntu

How to Set Up a Continuous Deployment Pipeline with GitLab on Ubuntu

Continuous deployment pipelines have become the backbone of modern software development, allowing teams to automatically build, test, and deploy applications whenever code changes are pushed to repositories. Setting up a continuous deployment pipeline with GitLab on Ubuntu combines the power of GitLab’s integrated CI/CD features with the stability and flexibility of Ubuntu servers, creating a robust automation workflow that reduces manual intervention and accelerates software delivery. This guide will walk you through the complete process of establishing a GitLab CI/CD pipeline on Ubuntu, from initial server setup and GitLab Runner installation to creating sophisticated deployment workflows that handle everything from code testing to production deployments.

How GitLab CI/CD Pipelines Work

GitLab’s CI/CD system operates on a runner-based architecture where GitLab Runners are lightweight agent processes that execute pipeline jobs defined in your project’s .gitlab-ci.yml file. When code changes are pushed to your GitLab repository, the GitLab server automatically triggers pipeline execution based on predefined conditions and stages.

The pipeline workflow typically follows these stages:

GitLab detects repository changes and reads the .gitlab-ci.yml configuration file
Available GitLab Runners pick up queued jobs based on tags and availability
Runners execute jobs in isolated environments (containers, VMs, or shell executors)
Results are reported back to GitLab, which determines whether to proceed to subsequent stages
Successful pipelines trigger deployment actions to staging or production environments

GitLab Runners can be configured as shared runners (available to all projects) or specific runners (dedicated to particular projects or groups). For production deployments, specific runners provide better security isolation and resource control.

Step-by-Step Implementation Guide

Prerequisites and Server Preparation

Before setting up your GitLab CI/CD pipeline, ensure your Ubuntu server meets the minimum requirements. A VPS with at least 2GB RAM and 2 CPU cores is recommended for basic deployments, though production workloads may require more substantial resources like those available with dedicated servers.

Start by updating your Ubuntu system and installing essential dependencies:

sudo apt update && sudo apt upgrade -y
sudo apt install curl wget git software-properties-common apt-transport-https ca-certificates gnupg lsb-release -y

Installing GitLab Runner

GitLab provides official repositories for Ubuntu installations. Add the GitLab repository and install the runner:

# Add GitLab's official repository
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

# Install GitLab Runner
sudo apt install gitlab-runner -y

# Verify installation
gitlab-runner --version

Registering GitLab Runner

To connect your runner to GitLab, you’ll need the registration token from your GitLab project or group settings. Navigate to Settings > CI/CD > Runners in your GitLab interface to find this token.

# Register the runner interactively
sudo gitlab-runner register

# Non-interactive registration example
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "YOUR_REGISTRATION_TOKEN" \
  --executor "docker" \
  --docker-image alpine:latest \
  --description "ubuntu-docker-runner" \
  --tag-list "docker,ubuntu,deployment" \
  --run-untagged="true" \
  --locked="false" \
  --access-level="not_protected"

Docker Configuration for Container-Based Jobs

For modern CI/CD workflows, Docker executor provides excellent isolation and reproducibility. Install Docker and configure it properly:

# Install Docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io -y

# Add gitlab-runner user to docker group
sudo usermod -aG docker gitlab-runner

# Start and enable Docker service
sudo systemctl start docker
sudo systemctl enable docker

Creating Your First Pipeline Configuration

Create a .gitlab-ci.yml file in your project’s root directory. Here’s a comprehensive example for a Node.js application with testing and deployment stages:

stages:
  - build
  - test
  - security
  - deploy

variables:
  NODE_VERSION: "18"
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: "/certs"

# Define reusable templates
.node_template: &node_template
  image: node:${NODE_VERSION}-alpine
  cache:
    paths:
      - node_modules/
  before_script:
    - npm ci --cache .npm --prefer-offline

# Build stage
build_job:
  <<: *node_template
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour
  only:
    - main
    - develop
    - merge_requests

# Testing stages
unit_tests:
  <<: *node_template
  stage: test
  script:
    - npm run test:unit
    - npm run test:coverage
  coverage: '/Lines\s*:\s*(\d+\.\d+)%/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
    paths:
      - coverage/
    expire_in: 30 days

integration_tests:
  <<: *node_template
  stage: test
  services:
    - postgres:13-alpine
    - redis:6-alpine
  variables:
    DATABASE_URL: "postgresql://postgres:postgres@postgres:5432/test_db"
    REDIS_URL: "redis://redis:6379"
  script:
    - npm run test:integration
  only:
    - main
    - develop

# Security scanning
security_scan:
  stage: security
  image: node:${NODE_VERSION}-alpine
  script:
    - npm audit --audit-level high
    - npm install -g snyk
    - snyk test --severity-threshold=high
  allow_failure: true
  only:
    - main
    - merge_requests

# Deployment stages
deploy_staging:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache rsync openssh
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
  script:
    - rsync -avz --delete dist/ $STAGING_USER@$STAGING_HOST:$STAGING_PATH
    - ssh $STAGING_USER@$STAGING_HOST "cd $STAGING_PATH && pm2 restart all"
  environment:
    name: staging
    url: https://staging.yourapp.com
  only:
    - develop
  when: manual

deploy_production:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache rsync openssh
    - eval $(ssh-agent -s)
    - echo "$PROD_SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo "$PROD_SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
  script:
    - rsync -avz --delete dist/ $PROD_USER@$PROD_HOST:$PROD_PATH
    - ssh $PROD_USER@$PROD_HOST "cd $PROD_PATH && pm2 restart all"
  environment:
    name: production
    url: https://yourapp.com
  only:
    - main
  when: manual

Real-World Examples and Use Cases

Multi-Environment Deployment Strategy

A common production scenario involves deploying to multiple environments with different configurations. Here’s an advanced pipeline configuration that handles development, staging, and production deployments with environment-specific variables:

# Advanced multi-environment deployment
.deploy_template: &deploy_template
  image: alpine:latest
  before_script:
    - apk add --no-cache curl jq
    - echo "Deploying to $CI_ENVIRONMENT_NAME environment"
  script:
    - |
      # Health check before deployment
      if curl -f "$HEALTH_CHECK_URL"; then
        echo "Current deployment is healthy, proceeding..."
      else
        echo "Health check failed, investigating..."
      fi
    
    # Blue-green deployment logic
    - |
      CURRENT_COLOR=$(curl -s "$DEPLOYMENT_API/current" | jq -r '.color')
      NEW_COLOR=$([ "$CURRENT_COLOR" = "blue" ] && echo "green" || echo "blue")
      echo "Switching from $CURRENT_COLOR to $NEW_COLOR"
    
    # Deploy to inactive slot
    - curl -X POST "$DEPLOYMENT_API/deploy" -d "{\"color\":\"$NEW_COLOR\",\"version\":\"$CI_COMMIT_SHA\"}"
    
    # Switch traffic after health verification
    - sleep 30  # Allow deployment to stabilize
    - curl -X POST "$DEPLOYMENT_API/switch" -d "{\"active_color\":\"$NEW_COLOR\"}"

deploy_development:
  <<: *deploy_template
  stage: deploy
  environment:
    name: development
    url: https://dev.yourapp.com
  variables:
    HEALTH_CHECK_URL: "https://dev.yourapp.com/health"
    DEPLOYMENT_API: "https://dev-deploy-api.yourapp.com"
  only:
    - develop

deploy_production:
  <<: *deploy_template
  stage: deploy
  environment:
    name: production
    url: https://yourapp.com
  variables:
    HEALTH_CHECK_URL: "https://yourapp.com/health"
    DEPLOYMENT_API: "https://deploy-api.yourapp.com"
  only:
    - main
  when: manual

Microservices Deployment Pipeline

For microservices architectures, you often need to build and deploy multiple services from a single repository. This configuration handles selective builds based on changed files:

# Microservices pipeline with selective builds
variables:
  DOCKER_REGISTRY: "registry.gitlab.com/yourgroup/yourproject"

.build_service: &build_service
  stage: build
  image: docker:20.10.16
  services:
    - docker:20.10.16-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - |
      if git diff --name-only $CI_COMMIT_BEFORE_SHA $CI_COMMIT_SHA | grep -q "^${SERVICE_PATH}/"; then
        echo "Changes detected in ${SERVICE_NAME}, building..."
        cd ${SERVICE_PATH}
        docker build -t ${DOCKER_REGISTRY}/${SERVICE_NAME}:${CI_COMMIT_SHA} .
        docker push ${DOCKER_REGISTRY}/${SERVICE_NAME}:${CI_COMMIT_SHA}
        docker tag ${DOCKER_REGISTRY}/${SERVICE_NAME}:${CI_COMMIT_SHA} ${DOCKER_REGISTRY}/${SERVICE_NAME}:latest
        docker push ${DOCKER_REGISTRY}/${SERVICE_NAME}:latest
      else
        echo "No changes detected in ${SERVICE_NAME}, skipping build"
      fi

build_user_service:
  <<: *build_service
  variables:
    SERVICE_NAME: "user-service"
    SERVICE_PATH: "services/user-service"

build_order_service:
  <<: *build_service
  variables:
    SERVICE_NAME: "order-service"
    SERVICE_PATH: "services/order-service"

build_notification_service:
  <<: *build_service
  variables:
    SERVICE_NAME: "notification-service"
    SERVICE_PATH: "services/notification-service"

Comparison with Alternative CI/CD Solutions

Feature	GitLab CI/CD	GitHub Actions	Jenkins	Travis CI
Self-hosted options	Yes (Community/Enterprise)	Limited (Enterprise only)	Yes (Open source)	Enterprise only
Configuration format	YAML (.gitlab-ci.yml)	YAML (workflows)	Groovy/Declarative	YAML (.travis.yml)
Built-in Docker support	Excellent	Good	Plugin required	Good
Parallel job execution	Yes (unlimited self-hosted)	Yes (usage limits apply)	Yes (depends on setup)	Limited on free tier
Security scanning	Built-in (SAST/DAST)	Third-party integrations	Plugin ecosystem	Third-party integrations
Learning curve	Moderate	Easy	Steep	Easy

Performance Comparison

Based on real-world usage across different project sizes:

Metric	GitLab CI/CD	GitHub Actions	Jenkins
Average job startup time	15-30 seconds	10-20 seconds	5-15 seconds
Pipeline execution overhead	Low	Low	Variable
Resource efficiency	High (Docker executor)	High	Depends on configuration
Concurrent pipeline limit	Unlimited (self-hosted)	20 (free), more paid	Unlimited

Best Practices and Common Pitfalls

Security Best Practices

Security should be a primary consideration when setting up CI/CD pipelines. Implement these practices to protect your deployment infrastructure:

Store sensitive information like API keys, passwords, and certificates in GitLab's CI/CD variables with masking enabled
Use protected variables for production deployments that only run on protected branches
Implement proper SSH key management with separate keys for different environments
Enable runner registration tokens rotation and use project-specific runners for sensitive deployments
Regularly audit and rotate deployment credentials
Implement network isolation between runners and production systems where possible

# Secure variable usage example
deploy_production:
  stage: deploy
  script:
    - echo "$DEPLOY_KEY" | base64 -d > deploy_key.pem
    - chmod 600 deploy_key.pem
    - ssh -i deploy_key.pem -o StrictHostKeyChecking=no $DEPLOY_USER@$DEPLOY_HOST "bash -s" < deploy_script.sh
    - rm -f deploy_key.pem  # Clean up sensitive files
  variables:
    # These should be set as protected variables in GitLab
    DEPLOY_USER: "$PRODUCTION_DEPLOY_USER"
    DEPLOY_HOST: "$PRODUCTION_DEPLOY_HOST"
  only:
    variables:
      - $CI_COMMIT_REF_NAME == "main"

Performance Optimization Strategies

Optimize your pipeline performance with these proven techniques:

Implement aggressive caching for dependencies, build artifacts, and Docker layers
Use parallel job execution for independent tasks like testing and security scanning
Optimize Docker images by using multi-stage builds and Alpine-based images
Configure appropriate artifact retention policies to save storage space
Use GitLab's cache:fallback_keys feature for more flexible cache retrieval

# Advanced caching configuration
variables:
  CACHE_VERSION: "v1"
  NPM_CONFIG_CACHE: "$CI_PROJECT_DIR/.npm"

.cache_template: &cache_config
  cache:
    key:
      files:
        - package-lock.json
      prefix: "$CACHE_VERSION"
    paths:
      - node_modules/
      - .npm/
    policy: pull-push
    fallback_keys:
      - "$CACHE_VERSION-default"

build_job:
  <<: *cache_config
  stage: build
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run build

Common Pitfalls and Troubleshooting

Avoid these frequent mistakes that can cause pipeline failures:

Runner resource exhaustion: Monitor runner resource usage and configure appropriate concurrency limits
Cache corruption: Implement cache validation and use cache:policy appropriately
Environment variable conflicts: Use descriptive variable names and proper scoping
Docker socket permissions: Ensure gitlab-runner user has proper Docker access
Network connectivity issues: Configure proper DNS and firewall rules for runners

When troubleshooting pipeline issues, enable debug logging:

# Enable debug logging for troubleshooting
variables:
  CI_DEBUG_TRACE: "true"  # Enable detailed pipeline logging
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""  # Disable TLS for debugging

debug_job:
  stage: build
  script:
    - env | sort  # Display all environment variables
    - df -h       # Check disk space
    - free -m     # Check memory usage
    - docker info # Docker system information
  when: manual

Monitoring and Maintenance

Establish monitoring for your CI/CD infrastructure:

Set up GitLab's built-in pipeline monitoring and failure notifications
Monitor runner performance and availability using GitLab's admin interface
Implement health checks in deployment scripts to catch issues early
Regular maintenance includes updating GitLab Runner versions and cleaning up old artifacts
Monitor disk space usage on runner machines, especially when using shell executors

# Runner maintenance script
#!/bin/bash
# runner_maintenance.sh

echo "Starting GitLab Runner maintenance..."

# Clean up old Docker images
docker system prune -f --volumes

# Update GitLab Runner
sudo apt update
sudo apt upgrade gitlab-runner -y

# Restart runner service
sudo gitlab-runner restart

# Check runner status
sudo gitlab-runner status

echo "Maintenance completed. Current runner version:"
gitlab-runner --version

Advanced Integration Possibilities

GitLab CI/CD integrates well with various tools and services commonly used in modern development workflows. Consider these integration opportunities:

Slack/Teams notifications: Integrate pipeline status updates with team communication tools
Jira integration: Automatically update issue status based on deployment success
Monitoring tools: Send deployment events to Datadog, New Relic, or Prometheus
Database migrations: Integrate database schema updates into deployment pipelines
Feature flag management: Coordinate feature releases with services like LaunchDarkly

The flexibility of GitLab's CI/CD system, combined with Ubuntu's stability and extensive package ecosystem, creates a powerful foundation for automated software delivery. Whether you're managing simple web applications or complex microservices architectures, this setup provides the scalability and reliability needed for production environments. Regular monitoring, security updates, and performance optimization will ensure your pipeline continues to serve your development team effectively as your projects grow and evolve.

For additional information on GitLab CI/CD best practices and advanced configurations, refer to the official GitLab CI/CD documentation and the GitLab Runner documentation.

This article incorporates information and material from various online sources. We acknowledge and appreciate the work of all original authors, publishers, and websites. While every effort has been made to appropriately credit the source material, any unintentional oversight or omission does not constitute a copyright infringement. All trademarks, logos, and images mentioned are the property of their respective owners. If you believe that any content used in this article infringes upon your copyright, please contact us immediately for review and prompt action.

This article is intended for informational and educational purposes only and does not infringe on the rights of the copyright owners. If any copyrighted material has been used without proper credit or in violation of copyright laws, it is unintentional and we will rectify it promptly upon notification. Please note that the republishing, redistribution, or reproduction of part or all of the contents in any form is prohibited without express written permission from the author and website owner. For permissions or further inquiries, please contact us.