Lesson 09 intermediate

Testing in GitLab CI

Run the test pyramid in a pipeline: unit, integration with service containers and end-to-end tests, JUnit reports in merge requests, coverage percentage and line-by-line coverage, parallel and parallel:matrix jobs, test splitting, flaky test handling and keeping feedback under ten minutes.

Tests are why most teams build a pipeline in the first place. But a pipeline that runs npm test and prints 3,000 lines of output is only half the value. GitLab can read your test results, show failures right in the merge request, track coverage line by line and spread a slow suite across several machines. This lesson wires all of that into Orbit’s pipeline.

The test pyramid in a pipeline

LayerWhat it checksSpeedRuns
Static checksFormatting, lint, typesSecondsEvery push, first
Unit testsFunctions and modules in isolationSeconds to a minuteEvery push
Integration testsThe API with a real database and cacheA few minutesEvery merge request
End-to-end testsThe deployed app through a browser or HTTPMinutesAfter deploying to a review app or staging

Order matters: cheap checks run first and fail fast, expensive ones run only after cheap ones pass. With needs, the static checks do not even wait for the build.

Show test results in the merge request

Most test runners can write JUnit XML. Upload it as a report and GitLab shows a Test summary on the merge request: how many tests ran, which failed, the failure message, and which tests are newly failing compared with the target branch.

unit-tests:
  stage: test
  image: node:24-alpine
  needs: [build-api]
  script:
    - npm ci
    - node --test --test-reporter=junit --test-reporter-destination=junit.xml --test-reporter=spec --test-reporter-destination=stdout
  artifacts:
    when: always
    reports:
      junit: junit.xml

Two details matter:

  • when: always uploads the report even when tests fail, which is exactly when you need it.
  • Using two reporters keeps the readable output in the job log and writes XML for GitLab.

Every major test framework has a JUnit option: pytest --junitxml, Jest with jest-junit, Go with gotestsum --junitfile, Maven Surefire by default.

Coverage

GitLab tracks coverage in two ways.

The percentage

The coverage keyword takes a regular expression; GitLab reads the job log, extracts the number and shows it on the job, the merge request and the pipeline graphs:

unit-tests:
  script:
    - npx c8 --reporter=text-summary --reporter=cobertura npm test
  coverage: '/Lines\s*:\s*(\d+\.?\d*)%/'

Line-by-line coverage in the diff

Upload a Cobertura XML report and GitLab paints the merge request diff: a green bar next to lines covered by tests, orange next to lines that are not. Reviewers see at a glance whether the new code is tested.

  artifacts:
    when: always
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

Coverage is a signal, not a goal. 90% coverage of trivial getters is worth less than 60% that covers every error path. Watch the trend and the diff, not a hard number that people learn to game.

Integration tests with services

Lesson 8 introduced services. Here is the full integration job, with a wait for the database before tests start:

integration-tests:
  stage: test
  image: node:24-alpine
  needs: [build-api]
  services:
    - name: postgres:17-alpine
      alias: db
  variables:
    POSTGRES_DB: orbit_test
    POSTGRES_USER: orbit
    POSTGRES_PASSWORD: orbit-test-only
    DATABASE_URL: postgres://orbit:orbit-test-only@db:5432/orbit_test
  before_script:
    - apk add --no-cache postgresql17-client
    - until pg_isready -h db -U orbit; do sleep 1; done
    - npm ci
  script:
    - npm run db:migrate
    - npm run test:integration -- --test-reporter=junit --test-reporter-destination=integration.xml
  artifacts:
    when: always
    reports:
      junit: integration.xml
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

The password is visible in the YAML on purpose: the database lives only inside this job and is destroyed with it. Never reuse real credentials for test services.

End-to-end tests against a review app

End-to-end tests exercise the whole deployed system. Run them after the review app from lesson 7 is up, against its URL:

e2e-tests:
  stage: verify
  image: mcr.microsoft.com/playwright:v1.56.0-noble
  needs: [deploy-review]
  variables:
    BASE_URL: https://$CI_COMMIT_REF_SLUG.review.orbit.example.com
  script:
    - npm ci
    - npx playwright test --reporter=junit,line
  artifacts:
    when: always
    paths: [playwright-report/, test-results/]
    reports:
      junit: results.xml
    expire_in: 14 days
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Keep the Playwright image version equal to the @playwright/test version in package-lock.json, otherwise the browsers will not match. Saving traces and screenshots as artifacts turns “it failed in CI” into “here is exactly what the browser saw”.

Run jobs in parallel

parallel: N — split one suite

When the unit tests take eight minutes, run four copies of the job and give each a quarter of the files. GitLab sets CI_NODE_INDEX (1 to N) and CI_NODE_TOTAL (N) in each copy:

unit-tests:
  parallel: 4
  script:
    - npm ci
    - FILES=$(find test -name '*.test.js' | sort | awk "NR % $CI_NODE_TOTAL == $CI_NODE_INDEX - 1")
    - node --test $FILES

Many runners split by timing instead of file count for better balance: Jest has --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL, Playwright has --shard, and pytest has plugins such as pytest-split.

parallel:matrix — the same job, different settings

Test Orbit against the two Node.js versions it supports and two databases, in one definition:

compat-tests:
  stage: test
  image: node:$NODE_VERSION-alpine
  services:
    - name: $DB_IMAGE
      alias: db
  parallel:
    matrix:
      - NODE_VERSION: ["22", "24"]
        DB_IMAGE: ["postgres:16-alpine", "postgres:17-alpine"]
  script:
    - npm ci
    - npm run test:integration

That creates four jobs, named after their variables, all running at the same time.

Flaky tests

A flaky test passes and fails on the same code. It is worse than no test, because people learn to click retry and stop trusting red pipelines. A pragmatic policy:

  1. Measure. The merge request test summary shows how many times a failing test also failed recently on other pipelines, which is how flaky tests give themselves away.
  2. Quarantine. Move a known flaky test to a separate job with allow_failure: true so it stops blocking merges while still being visible.
  3. Fix or delete within a deadline, usually one sprint. A quarantined test nobody fixes is a deleted test with extra steps.
  4. Use retry only for infrastructure failures (runner_system_failure, stuck_or_timeout_failure), not for script_failure.

Keep feedback fast

TechniqueLesson
Cache dependency downloads3
needs so jobs start as soon as their inputs are ready3
Run lint with needs: [] so it starts instantly3
rules:changes to skip unaffected parts on merge requests6
Cancel outdated pipelines with interruptible6
Split slow suites with parallel9
Push slow end-to-end tests to after the merge or to a nightly schedule9

Aim for a merge request pipeline under ten minutes. Measure it under Analyze → CI/CD analytics, which shows pipeline duration and success rate over time.

Orbit’s test stage

stages: [build, test, deploy, verify]

lint:
  extends: .node-job
  stage: test
  needs: []
  script: npm run lint

unit-tests:
  extends: .node-job
  stage: test
  needs: [build-api]
  parallel: 2
  script:
    - npx c8 --reporter=text-summary --reporter=cobertura node --test --test-shard=$CI_NODE_INDEX/$CI_NODE_TOTAL --test-reporter=junit --test-reporter-destination=junit.xml --test-reporter=spec --test-reporter-destination=stdout
  coverage: '/Lines\s*:\s*(\d+\.?\d*)%/'
  artifacts:
    when: always
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

Node’s built-in runner supports --test-shard, so the split needs no extra tools.

Exercise

Your unit test job fails, but the merge request shows no test summary. The job has artifacts:reports:junit: junit.xml. What is the most likely cause?

  1. JUnit reports only work on GitLab Ultimate
  2. The artifacts are uploaded only on success by default; add when: always
  3. The file must be called report.xml
  4. Test summaries only appear on the main branch
Show answer

Answer: B. Artifacts default to when: on_success, so a failing job never uploads its report. Setting when: always makes the report available exactly when you need to see which test failed.

Exercise

A job defines parallel:matrix with NODE_VERSION [“22”, “24”] and DB_IMAGE with two values. How many jobs does GitLab create?

  1. 2
  2. 4
  3. 1, running each combination in sequence
  4. 8
Show answer

Answer: B. A matrix creates one job for every combination of the listed values: 2 Node.js versions times 2 database images is 4 jobs, all able to run in parallel.

Key takeaways

  • Order tests from cheapest to most expensive and fail fast.
  • Upload JUnit reports with when: always to see failures in the merge request.
  • coverage shows a percentage; a Cobertura report shows covered lines in the diff.
  • Services give integration tests a fresh real database per job.
  • parallel splits a slow suite; parallel:matrix tests several configurations at once.
  • Quarantine and fix flaky tests; never hide them with retries.

FAQ

Do test reports need a paid GitLab tier?

No. JUnit test summaries, coverage percentage and Cobertura coverage visualisation in merge requests work on the Free tier.

How do I see which test is slow?

The test report on the pipeline page lists every test with its duration. Sort by time and start with the slowest file.

Should end-to-end tests block a merge?

A small, reliable smoke suite should. A large, slower suite usually runs after merge on staging or on a nightly schedule, with failures alerting the team.

Can I run only the tests affected by a change?

Some tools support it (Jest’s --changedSince, Nx and Turborepo affected graphs, Bazel). Combine them with a full run on main so nothing slips through.

How do I test the Dockerfile itself?

Build the image in the merge request pipeline without pushing a release tag, then run a container-structure test or a quick docker run smoke test that hits the health endpoint.

↑ ↓ navigate Enter open Esc close