Lesson 03 beginner

Pipeline Structure, Stages and Jobs

How GitLab turns .gitlab-ci.yml into a pipeline: stages and their order, jobs and defaults, needs for a faster DAG, artifacts versus cache, extends and !reference to avoid repetition, allow_failure, retry, timeout and when a job runs.

Your Orbit pipeline works, but every job reinstalls dependencies from scratch, the test stage waits for every build job even when it only needs one, and the same lines repeat in every job. This lesson covers the building blocks that fix all three: stages and needs, artifacts and cache, and the keywords that keep a growing file readable.

Stages set the order

stages lists the columns of your pipeline from left to right:

stages:
  - build
  - test
  - package
  - deploy

The rules are simple:

  • Jobs in the same stage run in parallel (as long as runners are free).
  • A stage starts only when every job in the previous stage succeeded.
  • If any job fails, later stages are skipped and the pipeline fails.
  • A job without stage: lands in test.

If you leave out stages entirely, GitLab uses the defaults .pre, build, test, deploy, .post. The .pre and .post stages always exist and are always first and last, which makes them handy for setup checks and notifications.

needs: stop waiting for unrelated jobs

Orbit now has a front end too. With stages alone, the API tests wait for the slow web build to finish even though they never use it:

build:   build-api (1 min)    build-web (6 min)
test:    api-tests waits 6 minutes for build-web to finish

needs lets a job start as soon as the jobs it actually depends on are done, ignoring the rest of the previous stage:

stages: [build, test]

build-api:
  stage: build
  script: npm ci && npm run build
  artifacts:
    paths: [dist/]

build-web:
  stage: build
  script: cd web && npm ci && npm run build
  artifacts:
    paths: [web/dist/]

api-tests:
  stage: test
  needs: [build-api]
  script: npm ci && npm test

web-tests:
  stage: test
  needs: [build-web]
  script: cd web && npm ci && npm test

Now api-tests starts the moment build-api passes. The pipeline becomes a directed acyclic graph (DAG) instead of rigid columns. Switch the pipeline view to Job dependencies to see the arrows.

needs also controls artifacts: a job with needs downloads artifacts only from the jobs it lists. Use needs: [] for a job that depends on nothing and can start immediately, like a quick linter.

Stages still matter with needs. They group jobs visually and still decide the order for jobs that have no needs. A job can only need jobs from its own or earlier stages.

Artifacts pass results forward

An artifact is a file or folder a job uploads to GitLab when it finishes. Later jobs download it and people can browse or download it from the job page.

build-api:
  stage: build
  script:
    - npm ci
    - npm run build
  artifacts:
    name: "orbit-api-$CI_COMMIT_SHORT_SHA"
    paths:
      - dist/
    exclude:
      - dist/**/*.map
    expire_in: 7 days
    when: on_success
OptionWhat it does
pathsFiles and folders to upload, relative to the project directory
excludePatterns to leave out
expire_inHow long GitLab keeps them; the latest artifacts of a branch are kept by default
whenon_success (default), on_failure or always; always is useful for test reports and logs
reportsSpecial files GitLab understands, like JUnit test results (lesson 9)

Cache speeds up repeated work

A cache is also a set of files saved by a job, but with a different purpose: speed, not correctness. It keeps things you could download again, such as npm packages, between pipelines.

default:
  image: node:24-alpine
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
  before_script:
    - npm ci --cache .npm --prefer-offline

How this works: the cache key is computed from package-lock.json. While the lock file does not change, every job restores the same .npm/ folder and npm ci installs from it in seconds instead of downloading the world. When someone adds a dependency, the key changes and a fresh cache is built.

We cache npm’s download folder (.npm/) rather than node_modules/: npm ci deletes node_modules anyway, while the download cache survives and is safe to share.

Artifacts versus cache

This is the distinction that confuses everyone at first:

ArtifactsCache
PurposePass results between jobs and keep themSpeed up inputs you could recreate
Exampledist/, test reports, a built binarynpm, pip, Maven or Go module downloads
Guaranteed to exist?Yes, later jobs fail if it is missingNo, a job must work without it
StoredOn the GitLab serverOn the runner or in object storage configured for the runner
ScopeOne pipeline (and downloadable afterwards)Shared across pipelines with the same key

Rule of thumb: if the next job needs the file to be correct, it is an artifact. If the job would just be slower without it, it is a cache.

default: set things once

Keywords under default apply to every job that does not override them: image, before_script, after_script, cache, services, retry, timeout, tags, interruptible and a few more.

default:
  image: node:24-alpine
  interruptible: true
  retry:
    max: 2
    when:
      - runner_system_failure
      - stuck_or_timeout_failure

A job that sets its own image replaces the default. The two values are not merged.

extends: reuse a job template

Jobs whose name starts with a dot are hidden: GitLab never runs them, but other jobs can inherit from them with extends.

.node-job:
  image: node:24-alpine
  cache:
    key:
      files: [package-lock.json]
    paths: [.npm/]
  before_script:
    - npm ci --cache .npm --prefer-offline

lint:
  extends: .node-job
  stage: test
  script: npm run lint

unit-tests:
  extends: .node-job
  stage: test
  script: npm test

extends merges hashes deeply, so a job can override a single key and keep the rest. Arrays such as script are replaced, not appended.

!reference: borrow a single section

When you only want a piece of another job, for example its before_script, use the !reference tag:

.setup:
  script:
    - apk add --no-cache curl jq

smoke-test:
  stage: test
  script:
    - !reference [.setup, script]
    - curl -fsS "$STAGING_URL/health" | jq .

YAML anchors (&name and <<: *name) also work, but only inside one file. extends and !reference work across included files too, which matters once your configuration grows (lesson 12).

Controlling how a job behaves

docs-link-check:
  stage: test
  script: npm run check-links
  allow_failure: true      # a red job shows a warning, the pipeline continues
  timeout: 10 minutes      # kill it if it hangs
  retry: 1                 # try once more on any failure

cleanup-preview:
  stage: deploy
  script: ./scripts/cleanup.sh
  when: always             # run even if earlier jobs failed

deploy-production:
  stage: deploy
  script: ./scripts/deploy.sh production
  when: manual             # wait for a person to click play
when valueThe job runs…
on_successWhen all earlier jobs passed (default)
on_failureOnly when an earlier job failed, good for alerts
alwaysNo matter what happened before
manualWhen someone clicks it in the UI or API
delayedAfter start_in time, for example start_in: 30 minutes
neverNot at all (useful inside rules, lesson 6)

Use retry for infrastructure flakiness, never to hide flaky tests. A test that passes on the second try is still broken.

The Orbit pipeline so far

stages: [build, test]

default:
  image: node:24-alpine
  interruptible: true

.node-job:
  cache:
    key:
      files: [package-lock.json]
    paths: [.npm/]
  before_script:
    - npm ci --cache .npm --prefer-offline

build-api:
  extends: .node-job
  stage: build
  script: npm run build
  artifacts:
    paths: [dist/]
    expire_in: 7 days

lint:
  extends: .node-job
  stage: test
  needs: []
  script: npm run lint

unit-tests:
  extends: .node-job
  stage: test
  needs: [build-api]
  script: npm test

lint now starts immediately, unit-tests starts as soon as the build is done and dependencies come from the cache.

Exercise

A job needs the compiled dist/ folder from the build job to run its tests. Where should dist/ be saved?

  1. As an artifact of the build job
  2. In the cache, keyed on the branch name
  3. In a CI/CD variable
  4. Nowhere, the test job builds it again
Show answer

Answer: A. The test job cannot work without dist/, so it must be guaranteed to exist: that is an artifact. The cache is best-effort and may be missing, so it is only for things you can recreate.

Exercise

A pipeline has build-api and build-web in the build stage. api-tests has needs: [build-api]. build-web takes 6 minutes. When does api-tests start?

  1. After every job in the build stage finishes
  2. As soon as build-api succeeds, without waiting for build-web
  3. At the same time as build-api
  4. Only if build-web also succeeds
Show answer

Answer: B. needs turns the pipeline into a graph: a job waits only for the jobs it lists. That is the main tool to shorten a pipeline without changing any job.

Key takeaways

  • Stages run in order; jobs inside a stage run in parallel.
  • needs lets a job start as soon as its real dependencies finish, and limits which artifacts it downloads.
  • Artifacts carry results that later jobs require; caches speed up downloads and may be missing.
  • default, hidden jobs with extends and !reference keep the file short and consistent.
  • allow_failure, when, retry and timeout control how each job behaves.

FAQ

Should I still use stages if I use needs everywhere?

Yes. Stages keep the pipeline view readable and give an order to jobs that have no needs. Many teams use a few broad stages and needs for the fine-grained dependencies.

Why is my cache never restored?

Common causes: the key changed (for example a key based on a file that changes every commit), jobs run on different runners without shared cache storage, or the cached path is outside the project directory. Caches can only contain paths inside the project.

Can a job use artifacts from a previous pipeline?

Not directly through stages. You can download them through the API or with needs:pipeline and needs:project (lesson 12). For build outputs you reuse often, publishing to a registry is usually cleaner.

What is the difference between extends and YAML anchors?

Anchors are plain YAML and only work inside one file. extends is GitLab’s own inheritance, merges nested keys and works across included files. Prefer extends and !reference.

Do jobs in the same stage always run at the same time?

Only if enough runners are free. On a busy runner, parallel jobs queue. They are allowed to run at the same time, not guaranteed to.

↑ ↓ navigate Enter open Esc close