Lesson 13 advanced

GitLab CI Best Practices

The habits that keep pipelines fast, safe and maintainable: a ten-minute feedback budget, fail-fast ordering, caching and needs, pinned images and includes, least-privilege secrets and runners, readable YAML with components, measuring pipeline health, and the complete Orbit pipeline that ties the course together.

You now know every piece: stages and needs, artifacts and caches, rules, variables, runners, Docker builds, tests, scanners, environments, OIDC deploys and components. This last lesson collects the habits that separate a pipeline people trust from one they work around, then shows the complete Orbit pipeline built over the course.

1. Treat feedback time as a budget

A merge request pipeline is a promise to the developer: you will know within N minutes. Set N (ten is a good target) and defend it.

  • Fail fast. Lint and type checks start with needs: [] and finish in seconds. A missing semicolon should not wait for a Docker build.
  • **Shape the graph with needs.** Every job starts as soon as its real inputs exist.
  • Cache downloads, keyed on lock files, never on branch names alone.
  • Parallelise slow suites with parallel and test sharding.
  • Skip what did not change on merge requests with rules:changes; run everything on main.
  • Move slow suites after the merge or to a nightly schedule when they do not need to block.
  • Cancel outdated pipelines with interruptible and auto_cancel.

2. Make every run reproducible

The same commit should produce the same result today and in six months.

  • Pin images to versions, and to digests for anything that deploys: node:24-alpine@sha256:….
  • Pin includes and components to tags or SHAs.
  • Install from lock files (npm ci, pip install -r requirements.lock, go mod download).
  • Build once, deploy many. Build the image in one job, tag it with the commit, and deploy that exact image to staging and production. Never rebuild for production.
  • Keep scripts runnable locally. Put real logic in ci/*.sh scripts that work on a laptop too; keep YAML for wiring.

3. Keep the YAML readable

Pipelines are code that everyone reads and nobody owns, unless you make them easy to own.

  • Put shared setup in default: and hidden jobs used with extends.
  • Split long files with include: local by concern: ci/build.yml, ci/test.yml, ci/deploy.yml.
  • Share across repositories with versioned CI/CD components, not copy-paste.
  • Name jobs by what they do (deploy-api-staging), not by number (job3).
  • Give pipelines readable names with workflow:name.
  • Add a short comment above anything surprising. The next person to read it is you in a year.
  • Validate in the pipeline editor or with glab ci lint before pushing.

4. Least privilege everywhere

  • Protected branches and tags for main and releases; nobody pushes directly.
  • Protected, masked and hidden variables, scoped to environments.
  • OIDC for cloud access, one role per environment, trust policies that check project and ref.
  • Protected runners for deploys; no privileged mode on shared runners.
  • Job token allowlist limited to projects that need it.
  • CODEOWNERS on .gitlab-ci.yml and ci/ so pipeline changes get a platform review.
  • Scanners for secrets, code, dependencies and images, with high findings blocking merges.

5. Make deploys boring

  • Every deploy job has an environment so it is tracked and can be rolled back.
  • Deploys use resource_group and interruptible: false.
  • Deploy jobs wait for health (ecs wait, kubectl rollout status, a smoke test) instead of passing on “request accepted”.
  • Database migrations are backward compatible for one release, so rollback stays possible.
  • Production is a manual job or an approval until monitoring and rollback have earned your trust.

6. Keep costs and storage in check

  • Set expire_in on artifacts; keep long-lived results in a registry instead.
  • Turn on a container registry cleanup policy.
  • Use small runner sizes for small jobs; reserve big machines for builds that need them.
  • Watch compute minute usage under the group’s Usage quotas page.
  • Add timeout to jobs that call external systems so a hang does not burn an hour.

7. Measure the pipeline like a product

Open Analyze → CI/CD analytics regularly and track:

MetricHealthy sign
Merge request pipeline durationStable and under your budget
Success rate on mainAbove 90%; red main is fixed within the hour
Most failed jobsNot the same flaky job week after week
Deployment frequencySteady or rising (DORA metrics, from environments)
Lead time for changesShrinking as the pipeline gets faster

Pick the worst number each month and fix it. Pipeline health decays quietly if nobody watches it.

Anti-patterns to avoid

Anti-patternWhy it hurtsDo this instead
image: node:latestBuilds change without any commitPin versions and digests
Long-lived cloud keys in variablesOne leak exposes productionOIDC with short-lived credentials
retry: 2 on test jobsHides flaky testsQuarantine and fix them
Rebuilding the image for productionProduction runs untested bytesBuild once, promote the same tag
Deploying latestRollback is impossibleImmutable commit or version tags
One giant 800-line fileNobody dares to change itIncludes and components
allow_failure: true everywhereRed jobs become wallpaperOnly for genuinely optional checks
only / exceptLimited, legacy syntaxrules
Printing the environment for debuggingLeaks secrets into logsPrint specific non-secret variables

The complete Orbit pipeline

Here is the main .gitlab-ci.yml, with details moved into included files exactly as the course built them:

# Orbit: build, test, scan, ship.
workflow:
  name: "$CI_PIPELINE_SOURCE: $CI_COMMIT_REF_NAME"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH
    - if: $CI_COMMIT_TAG
    - if: $CI_PIPELINE_SOURCE == "schedule"
  auto_cancel:
    on_new_commit: interruptible

stages: [build, test, package, deploy, verify]

default:
  image: node:24-alpine
  interruptible: true
  retry:
    max: 2
    when: [runner_system_failure, stuck_or_timeout_failure]

include:
  - local: ci/build.yml          # build-api, build-web
  - local: ci/test.yml           # lint, unit-tests, integration-tests
  - local: ci/image.yml          # build-image, push-ecr
  - local: ci/deploy.yml         # review apps, staging, production
  - component: $CI_SERVER_FQDN/components/secret-detection/secret-detection@~latest
  - component: $CI_SERVER_FQDN/components/sast/sast@~latest
  - template: Jobs/Container-Scanning.gitlab-ci.yml

container_scanning:
  needs: [build-image]
  variables:
    CS_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

Replace ~latest with fixed component versions once you have tested them. And the flow it produces:

merge request:
  lint ─────────────────────────────┐
  build-api ─▶ unit-tests (x2)       │
            ─▶ integration-tests     ├─▶ build-image ─▶ container_scanning
  build-web ─▶ web-tests             │              ─▶ deploy-review ─▶ e2e-tests
  secret_detection, semgrep-sast ───┘

main:
  ...same checks... ─▶ push-ecr ─▶ deploy-staging ─▶ trigger-e2e
                                                ─▶ deploy-production (manual, protected)

Every piece maps to a lesson:

PieceLesson
Stages, needs, artifacts, cache3
Variables, protected secrets, job token4
Runners and tags5
workflow:rules, auto_cancel, manual production6
Environments, review apps, resource_group7
Docker image with an immutable tag8
JUnit, coverage, parallel tests9
Secret detection, SAST, container scanning10
OIDC deploy to ECR, ECS, S3 and CloudFront11
Includes, components, multi-project triggers12

A checklist for any new pipeline

  1. Lint and unit tests run on every merge request and finish in minutes.
  2. main is protected and its pipeline must pass before merging.
  3. Images and includes are pinned.
  4. Artifacts expire; caches are keyed on lock files.
  5. The build happens once; deploys promote the same artifact.
  6. Secrets are protected and masked, or replaced by OIDC.
  7. Deploy jobs have environments, resource groups and health checks.
  8. Security scanners run and high findings block the merge.
  9. Pipeline changes are reviewed by their owners.
  10. Someone looks at CI/CD analytics every month.

Exercise

The production deploy job runs docker build again so the image includes production settings. What is the problem?

  1. Nothing, as long as the Dockerfile is the same
  2. It uses more compute minutes but is otherwise fine
  3. Production gets a new, untested image; configuration should come from environment variables and the tested image should be promoted unchanged
  4. Docker cannot build images in the deploy stage
Show answer

Answer: C. A rebuild can pick up different base layers or dependencies than the image you tested on staging. Build once, inject environment-specific settings at runtime and deploy the same immutable tag everywhere.

Exercise

Your merge request pipeline takes 25 minutes. Which change gives developers faster feedback without dropping any checks?

  1. Mark every job allow_failure: true
  2. Start lint with needs: [], cache dependencies, add needs between jobs and split the slowest suite with parallel
  3. Run all jobs on one bigger runner sequentially
  4. Remove the test stage and rely on staging
Show answer

Answer: B. Fail-fast ordering, caching, a needs graph and parallel test splitting shorten the critical path while keeping every check. The other options either hide failures or remove safety.

Key takeaways

  • Treat feedback time as a budget and defend it with fail-fast ordering, needs, caching and parallelism.
  • Reproducibility comes from pinned images and includes, lock files and building once.
  • Readable pipelines use defaults, extends, includes and components instead of copy-paste.
  • Least privilege applies to branches, variables, runners, tokens and cloud roles.
  • Deploys should be tracked, serialised, health-checked and easy to roll back.
  • Measure pipeline health and fix the worst number regularly.

FAQ

Where should I start if my current pipeline is a mess?

Measure first: duration, success rate and the most failing jobs. Then fix in this order: secrets and protected branches, a red main, the slowest step on the critical path, and finally YAML structure.

How do I convince my team to keep main green?

Make red visible (pipeline status in chat, a badge on the README), make the fix fast (small merges, clear logs), and agree that a red main beats any feature work. Merge trains on GitLab Premium also stop broken combinations from landing.

Should every project have the same pipeline?

The same building blocks, not the same file. Shared components give consistency for build, test, scan and deploy, while each project wires them for its own needs.

What should I learn after this course?

GitOps with Argo CD or Flux for Kubernetes deploys, progressive delivery (canaries and feature flags), and infrastructure pipelines for Terraform. The Terraform Tutorial covers the last one, including a full CI pipeline for infrastructure.

How do these practices translate to GitHub Actions?

Almost directly: jobs and needs, caches and artifacts, environments with protection rules, OIDC to AWS with id-token: write, reusable workflows instead of components. The habits matter more than the keywords.

↑ ↓ navigate Enter open Esc close