Skip to main content

Jenkins Architecture & Setup

Understanding Jenkins architecture and setting up enterprise-grade Jenkins instances for scalable CI/CD operations.

Jenkins Architecture Overview​

Jenkins follows a distributed architecture model that enables scalable and flexible CI/CD operations across multiple machines and environments.

Core Architecture Components​

Master Node Responsibilities​

1. Central Coordination​

  • Build Scheduling: Manages build queue and execution order
  • Job Management: Creates, configures, and manages build jobs
  • Resource Allocation: Distributes builds across available agents
  • Status Monitoring: Tracks build status and agent health

2. Configuration Management​

  • System Configuration: Global Jenkins settings and preferences
  • Plugin Management: Installation, updates, and configuration of plugins
  • Security Settings: User authentication, authorization, and access control
  • Environment Variables: Global and job-specific environment variables

3. User Interface​

  • Web Dashboard: Main user interface for job management and monitoring
  • Build History: Complete build logs, artifacts, and reports
  • User Management: User accounts, roles, and permissions
  • System Information: Jenkins health, performance metrics, and diagnostics

Agent Node Responsibilities​

1. Build Execution​

  • Job Execution: Runs assigned build jobs and pipelines
  • Environment Isolation: Provides isolated execution environments
  • Resource Utilization: Manages CPU, memory, and disk resources
  • Tool Access: Provides access to build tools and dependencies

2. Environment Management​

  • Operating System: Various OS support (Linux, Windows, macOS)
  • Build Tools: Pre-installed development tools and compilers
  • Dependencies: Software libraries and runtime environments
  • Network Access: Connectivity to external systems and repositories

Master-Slave Architecture​

Traditional Master-Slave Setup​

Agent Types and Configurations​

1. Static Agents​

# Static agent configuration
agent:
  type: "static"
  name: "linux-builder"
  os: "ubuntu-20.04"
  tools:
    - java-11
    - maven-3.8
    - docker-20.10
  labels:
    - "linux"
    - "maven"
    - "docker"

2. Dynamic Agents​

# Dynamic agent configuration
agent:
  type: "dynamic"
  cloud: "aws"
  instance_type: "t3.medium"
  ami: "ami-12345678"
  auto_scaling: true
  min_instances: 1
  max_instances: 10
  labels:
    - "aws"
    - "dynamic"

3. Container Agents​

# Container agent configuration
agent:
  type: "container"
  image: "jenkins/inbound-agent:latest"
  resources:
    memory: "2Gi"
    cpu: "1000m"
  volumes:
    - "/var/run/docker.sock:/var/run/docker.sock"
  labels:
    - "docker"
    - "container"

Installation Methods​

Traditional Installation​

Linux Installation​

# Ubuntu/Debian installation
wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb https://pkg.jenkins.io/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

# Start Jenkins service
sudo systemctl start jenkins
sudo systemctl enable jenkins

# Check status
sudo systemctl status jenkins

CentOS/RHEL Installation​

# CentOS/RHEL installation
sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key
sudo yum install jenkins

# Start Jenkins service
sudo systemctl start jenkins
sudo systemctl enable jenkins

Windows Installation​

# Download Jenkins Windows installer
Invoke-WebRequest -Uri "https://get.jenkins.io/war-stable/latest/jenkins.war" -OutFile "jenkins.war"

# Install as Windows Service
java -jar jenkins.war --httpPort=8080 --serviceName=Jenkins

# Start service
net start Jenkins

Containerized Installation​

Docker Installation​

# Dockerfile for Jenkins
FROM jenkins/jenkins:lts

# Switch to root user
USER root

# Install Docker CLI
RUN apt-get update && apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

RUN curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

RUN echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
    $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

RUN apt-get update && apt-get install -y docker-ce-cli

# Install additional tools
RUN apt-get install -y \
    git \
    maven \
    nodejs \
    npm

# Switch back to jenkins user
USER jenkins

# Install plugins
RUN jenkins-plugin-cli --plugins \
    workflow-aggregator \
    git \
    blueocean \
    docker-workflow \
    kubernetes \
    credentials-binding \
    ssh-slaves \
    matrix-auth \
    pam-auth \
    ldap \
    email-ext \
    mailer

Docker Compose Setup​

version: '3.8'

services:
  jenkins:
    build: .
    ports:
      - "8080:8080"
      - "50000:50000"
    volumes:
      - jenkins_home:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - JAVA_OPTS=-Djenkins.install.runSetupWizard=false
    networks:
      - jenkins-network

  jenkins-agent:
    image: jenkins/inbound-agent:latest
    depends_on:
      - jenkins
    environment:
      - JENKINS_URL=http://jenkins:8080
      - JENKINS_SECRET=your-agent-secret
      - JENKINS_AGENT_NAME=agent1
      - JENKINS_AGENT_WORKDIR=/home/jenkins/agent
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - jenkins-network

volumes:
  jenkins_home:

networks:
  jenkins-network:
    driver: bridge

Kubernetes Installation​

Jenkins Master Deployment​

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-master
  namespace: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-master
  template:
    metadata:
      labels:
        app: jenkins-master
    spec:
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts
        ports:
        - containerPort: 8080
        - containerPort: 50000
        env:
        - name: JAVA_OPTS
          value: "-Djenkins.install.runSetupWizard=false"
        - name: JENKINS_OPTS
          value: "--httpPort=8080"
        volumeMounts:
        - name: jenkins-home
          mountPath: /var/jenkins_home
        - name: jenkins-config
          mountPath: /usr/share/jenkins/ref/plugins
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 90
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins-pvc
      - name: jenkins-config
        configMap:
          name: jenkins-plugins
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: jenkins
spec:
  selector:
    app: jenkins-master
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  - name: agent
    port: 50000
    targetPort: 50000
  type: LoadBalancer

Jenkins Agent Deployment​

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-agent
  namespace: jenkins
spec:
  replicas: 3
  selector:
    matchLabels:
      app: jenkins-agent
  template:
    metadata:
      labels:
        app: jenkins-agent
    spec:
      containers:
      - name: jenkins-agent
        image: jenkins/inbound-agent:latest
        env:
        - name: JENKINS_URL
          value: "http://jenkins-service:8080"
        - name: JENKINS_SECRET
          valueFrom:
            secretKeyRef:
              name: jenkins-secret
              key: agent-secret
        - name: JENKINS_AGENT_NAME
          value: "k8s-agent"
        - name: JENKINS_AGENT_WORKDIR
          value: "/home/jenkins/agent"
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

Plugin Ecosystem Management​

Essential Plugins​

1. Core Workflow Plugins​

# Essential workflow plugins
jenkins-plugin-cli --plugins \
    workflow-aggregator \
    pipeline-stage-view \
    build-timeout \
    credentials-binding \
    timestamper \
    ws-cleanup \
    ant \
    gradle \
    maven-plugin \
    pipeline-maven

2. Version Control Plugins​

# Version control plugins
jenkins-plugin-cli --plugins \
    git \
    github \
    gitlab-plugin \
    bitbucket \
    subversion \
    mercurial

3. Build and Test Plugins​

# Build and test plugins
jenkins-plugin-cli --plugins \
    junit \
    testng-results \
    cobertura \
    jacoco \
    sonar \
    checkstyle \
    pmd \
    findbugs \
    warnings-ng

4. Deployment Plugins​

# Deployment plugins
jenkins-plugin-cli --plugins \
    deployit-plugin \
    deploy \
    docker-plugin \
    docker-workflow \
    kubernetes \
    kubernetes-cli \
    ansible \
    terraform

5. Notification Plugins​

# Notification plugins
jenkins-plugin-cli --plugins \
    email-ext \
    mailer \
    slack \
    telegram-notifications \
    mattermost \
    teams \
    hipchat

Plugin Management Best Practices​

1. Plugin Version Control​

// plugins.txt for version control
workflow-aggregator:2.6
git:4.7.1
blueocean:1.24.3
docker-workflow:1.26
kubernetes:1.29.2
credentials-binding:1.25
ssh-slaves:1.31.0
matrix-auth:2.6.2
pam-auth:1.6
ldap:2.7
email-ext:2.83
mailer:1.34

2. Plugin Security Management​

# Check for plugin vulnerabilities
curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"pluginName": "plugin-name"}' \
  http://jenkins:8080/updateCenter/checkPlugin

# Update vulnerable plugins
jenkins-plugin-cli --plugins \
    --available-updates \
    --output-updates

3. Plugin Performance Monitoring​

// Monitor plugin performance
import jenkins.model.Jenkins
import hudson.PluginWrapper

Jenkins.instance.pluginManager.plugins.each { plugin ->
    if (plugin.isEnabled()) {
        println "${plugin.getShortName()}: ${plugin.getVersion()}"
        // Check plugin health and performance
    }
}

Security Configuration​

Authentication and Authorization​

1. Matrix-Based Security​

// Configure matrix-based security
import jenkins.model.Jenkins
import hudson.security.*
import hudson.security.csrf.DefaultCrumbIssuer

def instance = Jenkins.getInstance()

// Enable matrix-based security
def matrixRealm = new FullControlOnceLoggedInAuthorizationStrategy()
matrixRealm.setAllowAnonymousRead(false)

def strategy = new MatrixAuthorizationStrategy()
strategy.add(Jenkins.ADMINISTER, "admin")
strategy.add(Jenkins.READ, "authenticated")
strategy.add(Jenkins.BUILD, "developers")
strategy.add(Jenkins.READ, "viewers")

instance.setAuthorizationStrategy(strategy)

// Enable CSRF protection
instance.setCrumbIssuer(new DefaultCrumbIssuer(true))

instance.save()

2. LDAP Integration​

// LDAP configuration
import hudson.security.LDAPSecurityRealm
import hudson.security.SecurityRealm

def ldapRealm = new LDAPSecurityRealm(
    "ldap://ldap.company.com:389",
    "DC=company,DC=com",
    "uid={0},ou=people,dc=company,dc=com",
    "ou=groups,dc=company,dc=com",
    "cn=jenkins,ou=service,dc=company,dc=com",
    "jenkins-password"
)

Jenkins.instance.setSecurityRealm(ldapRealm)
Jenkins.instance.save()

3. Role-Based Access Control​

// Role-based access control
import com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy
import com.michelin.cio.hudson.plugins.rolestrategy.Role

def strategy = new RoleBasedAuthorizationStrategy()

// Define global roles
def adminRole = new Role("admin", "Administrators")
def developerRole = new Role("developer", "Developers")
def viewerRole = new Role("viewer", "Viewers")

// Assign permissions to roles
strategy.addRole("globalRoles", adminRole)
strategy.addRole("globalRoles", developerRole)
strategy.addRole("globalRoles", viewerRole)

// Assign users to roles
strategy.assignRole("globalRoles", "admin", "admin-user")
strategy.assignRole("globalRoles", "developer", "dev-user1,dev-user2")
strategy.assignRole("globalRoles", "viewer", "viewer-user")

Jenkins.instance.setAuthorizationStrategy(strategy)
Jenkins.instance.save()

Security Hardening​

1. Network Security​

# Configure firewall rules
ufw allow 8080/tcp
ufw allow 50000/tcp
ufw deny 22/tcp
ufw enable

# Configure SSL/TLS
keytool -genkey -keyalg RSA -alias jenkins -keystore jenkins.jks -keysize 2048

# Update Jenkins configuration
echo "JENKINS_OPTS=\"--httpPort=-1 --httpsPort=8443 --httpsKeyStore=jenkins.jks --httpsKeyStorePassword=password\"" >> /etc/default/jenkins

2. System Security​

# Create dedicated Jenkins user
sudo useradd -r -m -s /bin/bash jenkins

# Set up proper file permissions
sudo chown -R jenkins:jenkins /var/lib/jenkins
sudo chmod 755 /var/lib/jenkins

# Configure log rotation
sudo tee /etc/logrotate.d/jenkins << EOF
/var/log/jenkins/jenkins.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 644 jenkins jenkins
}
EOF

3. Backup and Recovery​

#!/bin/bash
# Jenkins backup script

BACKUP_DIR="/backup/jenkins"
JENKINS_HOME="/var/lib/jenkins"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p $BACKUP_DIR

# Stop Jenkins
sudo systemctl stop jenkins

# Create backup
tar -czf $BACKUP_DIR/jenkins_backup_$DATE.tar.gz -C $JENKINS_HOME .

# Start Jenkins
sudo systemctl start jenkins

# Clean old backups (keep last 30 days)
find $BACKUP_DIR -name "jenkins_backup_*.tar.gz" -mtime +30 -delete

Performance Optimization​

System Tuning​

1. JVM Optimization​

# JVM tuning for Jenkins
export JAVA_OPTS="-Xmx4g -Xms2g -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+OptimizeStringConcat"
export JENKINS_OPTS="--httpPort=8080 --prefix=/jenkins"

2. Build Queue Optimization​

// Configure build queue
import hudson.model.Node
import hudson.slaves.NodeProperty
import hudson.slaves.EnvironmentVariablesNodeProperty

// Set executor count based on CPU cores
def nodes = Jenkins.instance.nodes
nodes.each { node ->
    def executors = Runtime.getRuntime().availableProcessors()
    node.setNumExecutors(executors)
}

3. Artifact Management​

// Configure artifact retention
import hudson.model.Build
import hudson.model.Project

Jenkins.instance.items.each { item ->
    if (item instanceof Project) {
        item.setBuildDiscarder(new LogRotator(30, 100, 30, 100))
    }
}

Monitoring and Alerting​

1. System Monitoring​

# Jenkins health check script
#!/bin/bash

JENKINS_URL="http://localhost:8080"
JENKINS_USER="admin"
JENKINS_TOKEN="your-api-token"

# Check Jenkins status
response=$(curl -s -o /dev/null -w "%{http_code}" -u $JENKINS_USER:$JENKINS_TOKEN $JENKINS_URL/api/json)

if [ $response -eq 200 ]; then
    echo "Jenkins is healthy"
else
    echo "Jenkins is unhealthy (HTTP $response)"
    # Send alert
    curl -X POST -H 'Content-type: application/json' \
        --data '{"text":"Jenkins is down!"}' \
        $SLACK_WEBHOOK_URL
fi

2. Performance Metrics​

// Collect performance metrics
import hudson.model.Computer
import hudson.model.Node

def metrics = [:]
def totalExecutors = 0
def busyExecutors = 0

Jenkins.instance.computers.each { computer ->
    totalExecutors += computer.getNumExecutors()
    busyExecutors += computer.countBusy()
}

metrics.put("totalExecutors", totalExecutors)
metrics.put("busyExecutors", busyExecutors)
metrics.put("utilization", (busyExecutors / totalExecutors) * 100)

println "Jenkins Performance Metrics: ${metrics}"

Key Takeaways​

Jenkins Architecture Best Practices​

  1. Scalable Design: Use distributed architecture for large-scale operations
  2. Resource Management: Properly configure agents and resource allocation
  3. Security First: Implement comprehensive security measures
  4. Monitoring: Continuous monitoring of system health and performance
  5. Backup Strategy: Regular backups and disaster recovery planning

Installation Considerations​

  • Container vs Traditional: Choose based on infrastructure and requirements
  • Plugin Management: Version control and security monitoring
  • Performance Tuning: JVM optimization and system configuration
  • High Availability: Consider clustering for critical environments
  • Compliance: Ensure security and regulatory compliance

Enterprise Features​

  • Scalability: Horizontal scaling with multiple agents
  • Security: Enterprise-grade authentication and authorization
  • Integration: Extensive plugin ecosystem and API support
  • Monitoring: Comprehensive monitoring and alerting capabilities
  • Automation: Advanced automation and orchestration features

Next Steps: Ready to build pipelines? Continue to Section 3.2: Pipeline as Code to learn how to create sophisticated Jenkins pipelines.

Understanding Jenkins architecture is crucial for building scalable, enterprise-grade CI/CD systems. In the next section, we'll explore how to create powerful pipelines using Pipeline as Code.