Preamble: Core Pipeline Concepts
Hosted vs. Self-Hosted Agents
The fundamental choice of who manages the build machines. This decision impacts speed, cost, security, and maintenance overhead.
Hosted (Microsoft/GitHub-Managed)
Pros:
No maintenance, a fresh VM for every job, wide range of pre-installed software.
Cons:
Slower startup, network restrictions, can be expensive at scale.
Self-Hosted
Pros:
Faster feedback, cheaper at scale, full control, direct access to private networks.
Cons:
You own the maintenance, patching, security, and scaling.
Dependency Caching
The easiest way to get a massive speed boost. Caching dependencies (NuGet, npm, etc.) between runs is critical for fast feedback loops.
Azure Pipelines (`cache` task):
- task: Cache@2
inputs:
key: 'nuget | "$(Agent.OS)" | **/packages.lock.json'
path: '$(NUGET_PACKAGES)'
GitHub Actions (`actions/cache`):
- uses: actions/cache@v4
with:
path: ~/.nuget/packages
key: ${{ runner.os }}-nuget-${{ hashFiles('**/packages.lock.json') }}
Deployment Strategies
How to release code without causing outages. Use intelligent strategies to limit the blast radius of a bad deployment.
- Immutable Infrastructure: Don't modify running servers. Replace them with fresh ones built from versioned artifacts to eliminate configuration drift.
- Blue/Green: Deploy to a parallel production environment ("Green"). Once verified, switch traffic from "Blue" to "Green" for instant rollback.
- Canary: Deploy to a small subset of users, monitor for errors, then gradually roll out to everyone.
Phase 1: Source Control Management (SCM)
Critical Practice: Branch Protection
Everything starts with code. The SCM is your single source of truth. Protecting the `main` branch is the first and most important quality gate.
Use rules to enforce that no code merges into `main` without meeting specific criteria. This is non-negotiable for stable, high-quality software.
Mandatory Rules (Azure Repos & GitHub):
- Build Validation / Status Checks: The pull request code must build successfully and pass all CI checks.
- Reviewer Requirements: At least one other person must approve the code changes. No self-merges.
- Work Item / Conversation Linking: Ensure every change is traceable to a requirement or that all review comments are resolved.
Phase 2: Continuous Integration (CI)
The "Pipeline as Code"
Modern pipelines are defined in YAML files (`azure-pipelines.yml` or `.github/workflows/*.yml`) committed to the repository. This provides versioning, peer review, and repeatability. Avoid classic UI editors.
"Shift-Left" Security
Security is not a separate step; it's a series of automated gates within CI. Find flaws early before they reach production.
Key Security Gates:
- SAST (Static Application Security Testing): Tools like SonarQube or GitHub CodeQL scan your source code for vulnerabilities and code quality issues.
- SCA (Software Composition Analysis): Tools like Dependabot or Snyk scan your open-source dependencies for known vulnerabilities.
Publishing Artifacts: The Docker Example
The final output of a successful CI run is a versioned, deployable artifact. Most commonly, this is a Docker image pushed to a container registry.
Azure Pipelines (`azure-pipelines.yml`)
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- task: Docker@2
inputs:
containerRegistry: 'MyACRConnection'
repository: 'my-app'
command: 'buildAndPush'
Dockerfile: '**/Dockerfile'
tags: '$(Build.BuildId)'
GitHub Actions (`.github/workflows/ci.yml`)
on:
push:
branches: [ "main" ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Log in to Registry
uses: azure/docker-login@v1
with:
login-server: myregistry.azurecr.io
username: ${{ secrets.ACR_USERNAME }}
password: ${{ secrets.ACR_PASSWORD }}
- name: Build and push image
run: |
docker build . -t my.acr.io/app:${{ github.sha }}
docker push my.acr.io/app:${{ github.sha }}
Phase 3: Continuous Delivery (CD)
IaC & The Approval Flow
Use Infrastructure as Code (IaC) to define environments. OpenTofu (Linux Foundation, MPL 2.0) is the open-source community standard; Terraform (owned by IBM since Feb 2025, BSL since Aug 2023) remains widely deployed but is no longer open source. The key to safe CD is a plan → approve → apply flow with a manual approval gate before any infrastructure is changed.
The Safety-Critical Flow:
-
Plan:
The pipeline runs
tofu plan(OpenTofu) orterraform planand saves the plan file as an artifact. - Approve: A human reviewer inspects the plan. Both Azure (Release Gates) and GitHub (Environments) have features to pause the pipeline and require manual sign-off.
-
Apply:
Only after approval does the pipeline run
tofu apply(OpenTofu) orterraform applyusing the saved plan.
Application Deployment
Getting the code running in the provisioned environment. This phase handles deploying the application itself and managing dependent changes, like database schemas.
To Kubernetes:
Use
Helm
, the package manager for Kubernetes (v4 is the current stable release as of 2026). The helm upgrade --install command is the standard for deploying or updating an application.
For Databases:
This is a classic point of failure. Migration scripts (using tools like EF Core Migrations or Flyway) must be applied before the application code is deployed to ensure compatibility.
Phase 4: Monitoring & Observability
The Three Pillars of Observability
Once deployed, you must be able to understand the application's health. If you're not monitoring your app, you're flying blind. Instrument your code with a tool like Application Insights or Prometheus.
You need all three to get a complete picture of your system's behavior:
- Logs: Detailed, timestamped records of discrete events. (e.g., "User X failed to log in")
- Metrics: Aggregated numerical data over time. (e.g., "5% error rate over the last 10 minutes")
- Traces: A detailed view of a single request's journey through all services in your application.
Appendix: Platform Choices
The Old Guard: Jenkins
The infinitely customizable, plugin-driven workhorse. If you can think of it, you can build it. But you own all its complexity, maintenance, and `Jenkinsfile` Groovy spaghetti.
The All-in-One: GitLab
A powerful competitor with a "single application" philosophy. Its CI/CD is extremely mature and tightly integrated via `.gitlab-ci.yml`, with many security features built-in.