Preamble: Core Pipeline Concepts
Hosted vs. Self-Hosted Agents
The fundamental choice of who manages the build machines. This decision impacts speed, cost, security, and maintenance overhead.
Hosted (Microsoft/GitHub-Managed)
Pros: No maintenance, a fresh VM for every job, wide range of pre-installed software.
Cons: Slower startup, network restrictions, can be expensive at scale.
Self-Hosted
Pros: Faster feedback, cheaper at scale, full control, direct access to private networks.
Cons: You own the maintenance, patching, security, and scaling.
Dependency Caching
The easiest way to get a massive speed boost. Caching dependencies (NuGet, npm, etc.) between runs is critical for fast feedback loops.
Azure Pipelines (`cache` task):
- task: Cache@2
inputs:
key: 'nuget | "$(Agent.OS)" | **/packages.lock.json'
path: '$(NUGET_PACKAGES)'
GitHub Actions (`actions/cache`):
- uses: actions/cache@v4
with:
path: ~/.nuget/packages
key: ${{ runner.os }}-nuget-${{ hashFiles('**/packages.lock.json') }}
Deployment Strategies
How to release code without causing outages. Use intelligent strategies to limit the blast radius of a bad deployment.
- Immutable Infrastructure: Don't modify running servers. Replace them with fresh ones built from versioned artifacts to eliminate configuration drift.
- Blue/Green: Deploy to a parallel production environment ("Green"). Once verified, switch traffic from "Blue" to "Green" for instant rollback.
- Canary: Deploy to a small subset of users, monitor for errors, then gradually roll out to everyone.
Phase 1: Source Control Management (SCM)
Critical Practice: Branch Protection
Everything starts with code. The SCM is your single source of truth. Protecting the `main` branch is the first and most important quality gate.
Use rules to enforce that no code merges into `main` without meeting specific criteria. This is non-negotiable for stable, high-quality software.
Mandatory Rules (Azure Repos & GitHub):
- Build Validation / Status Checks: The pull request code must build successfully and pass all CI checks.
- Reviewer Requirements: At least one other person must approve the code changes. No self-merges.
- Work Item / Conversation Linking: Ensure every change is traceable to a requirement or that all review comments are resolved.
Phase 2: Continuous Integration (CI)
The "Pipeline as Code"
Modern pipelines are defined in YAML files (`azure-pipelines.yml` or `.github/workflows/*.yml`) committed to the repository. This provides versioning, peer review, and repeatability. Avoid classic UI editors.
"Shift-Left" Security
Security is not a separate step; it's a series of automated gates within CI. Find flaws early before they reach production.
Key Security Gates:
- SAST (Static Application Security Testing): Tools like SonarQube or GitHub CodeQL scan your source code for vulnerabilities and code quality issues.
- SCA (Software Composition Analysis): Tools like Dependabot or Snyk scan your open-source dependencies for known vulnerabilities.
Publishing Artifacts: The Docker Example
The final output of a successful CI run is a versioned, deployable artifact. Most commonly, this is a Docker image pushed to a container registry.
Azure Pipelines (`azure-pipelines.yml`)
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- task: Docker@2
inputs:
containerRegistry: 'MyACRConnection'
repository: 'my-app'
command: 'buildAndPush'
Dockerfile: '**/Dockerfile'
tags: '$(Build.BuildId)'
GitHub Actions (`.github/workflows/ci.yml`)
on:
push:
branches: [ "main" ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Log in to Registry
uses: azure/docker-login@v1
with:
login-server: myregistry.azurecr.io
username: ${{ secrets.ACR_USERNAME }}
password: ${{ secrets.ACR_PASSWORD }}
- name: Build and push image
run: |
docker build . -t my.acr.io/app:${{ github.sha }}
docker push my.acr.io/app:${{ github.sha }}
Phase 3: Continuous Delivery (CD)
IaC & The Approval Flow
Use Infrastructure as Code (IaC) with Terraform to define environments. The key to safe CD is a `plan -> approve -> apply` flow with a manual approval gate before any infrastructure is changed.
The Safety-Critical Flow:
- Plan: The pipeline runs `terraform plan` and saves the plan file as an artifact.
- Approve: A human reviewer inspects the plan. Both Azure (Release Gates) and GitHub (Environments) have features to pause the pipeline and require manual sign-off.
- Apply: Only after approval does the pipeline run `terraform apply` using the saved plan.
Application Deployment
Getting the code running in the provisioned environment. This phase handles deploying the application itself and managing dependent changes, like database schemas.
To Kubernetes:
Use Helm, the package manager for Kubernetes. The `helm upgrade --install` command is the standard for deploying or updating an application.
For Databases:
This is a classic point of failure. Migration scripts (using tools like EF Core Migrations or Flyway) must be applied before the application code is deployed to ensure compatibility.
Phase 4: Monitoring & Observability
The Three Pillars of Observability
Once deployed, you must be able to understand the application's health. If you're not monitoring your app, you're flying blind. Instrument your code with a tool like Application Insights or Prometheus.
You need all three to get a complete picture of your system's behavior:
- Logs: Detailed, timestamped records of discrete events. (e.g., "User X failed to log in")
- Metrics: Aggregated numerical data over time. (e.g., "5% error rate over the last 10 minutes")
- Traces: A detailed view of a single request's journey through all services in your application.
Appendix: Platform Choices
The Old Guard: Jenkins
The infinitely customizable, plugin-driven workhorse. If you can think of it, you can build it. But you own all its complexity, maintenance, and `Jenkinsfile` Groovy spaghetti.
The All-in-One: GitLab
A powerful competitor with a "single application" philosophy. Its CI/CD is extremely mature and tightly integrated via `.gitlab-ci.yml`, with many security features built-in.