Lesson 11 advanced

Deploying to AWS from GitLab CI

Deploy Orbit to AWS with no stored access keys: GitLab OIDC ID tokens, an IAM identity provider and a role whose trust policy checks project, branch and environment, then push the image to ECR, roll out on ECS, publish the web app to S3 and CloudFront, and wait for a healthy deployment.

This is where the pipeline earns its keep: a merge to main ends with Orbit running on AWS. The API image goes to Amazon ECR and runs on Amazon ECS with Fargate; the web front end goes to an S3 bucket behind CloudFront. The infrastructure itself is created with Terraform, as in lesson 12 of the Terraform Tutorial; this lesson focuses on the pipeline that ships code onto it.

The most important part comes first: how the pipeline gets AWS permissions without any AWS access key stored in GitLab.

Why not store access keys?

The old way: create an IAM user, generate an access key, paste it into a masked CI/CD variable. It works, and it is the most common way CI pipelines leak cloud access:

  • The key never expires unless someone rotates it, and nobody does.
  • Anyone who can run a job that sees the variable can copy it out.
  • It grants the same permissions to every branch, every job and every pipeline.

OIDC: short-lived credentials on demand

GitLab can sign a JSON Web Token for each job, describing exactly who is asking: which project, branch, environment and pipeline. AWS can trust GitLab as an identity provider, check those claims and hand back temporary credentials that expire within an hour.

Job ──(1) ID token signed by GitLab──▶ AWS STS
                                        (2) checks signature and claims:
                                            project = orbit-team/orbit
                                            branch  = main
                                            environment = production
    ◀─(3) temporary credentials, 1 hour──
Job uses them, then they expire. Nothing is stored anywhere.

Claims in a GitLab ID token include project_path, ref, ref_type, ref_protected, environment, environment_protected, pipeline_source and a sub claim that combines several of them, such as project_path:orbit-team/orbit:ref_type:branch:ref:main.

Set up the trust on AWS

1. Add GitLab as an identity provider

In IAM, add an OpenID Connect provider with URL https://gitlab.com and audience https://gitlab.com (use your own GitLab URL on self-managed). With Terraform:

resource "aws_iam_openid_connect_provider" "gitlab" {
  url            = "https://gitlab.com"
  client_id_list = ["https://gitlab.com"]
}

2. Create a role the pipeline can assume

The trust policy is the security boundary. It must check which project and which branch may use the role, not just that the token comes from GitLab. Without the sub condition, any project on GitLab.com could assume your role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/gitlab.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "gitlab.com:aud": "https://gitlab.com",
          "gitlab.com:sub": "project_path:orbit-team/orbit:ref_type:branch:ref:main"
        }
      }
    }
  ]
}

Attach a permissions policy with only what the deploy needs: push to the orbit-api ECR repository, update the orbit ECS service, write to the web bucket and create CloudFront invalidations. Create a second role with the same trust but a different sub if review apps need AWS too, with fewer permissions.

Tip: use one role per environment. orbit-deploy-staging trusts main; orbit-deploy-production can additionally require the environment claim to be production and environment_protected to be true, so only a job deploying to the protected production environment can assume it.

Get credentials in the job

Ask GitLab for an ID token with id_tokens, write it to a file and point the AWS CLI at it. The AWS SDKs and CLI understand AWS_ROLE_ARN plus AWS_WEB_IDENTITY_TOKEN_FILE natively and call STS for you:

.aws-oidc:
  image:
    name: amazon/aws-cli:2.31.0
    entrypoint: [""]
  id_tokens:
    AWS_ID_TOKEN:
      aud: https://gitlab.com
  variables:
    AWS_REGION: eu-west-1
    AWS_ROLE_SESSION_NAME: gitlab-$CI_PROJECT_ID-$CI_PIPELINE_ID
  before_script:
    - echo "$AWS_ID_TOKEN" > /tmp/web-identity-token
    - export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/web-identity-token
    - aws sts get-caller-identity

AWS_ROLE_ARN comes from an environment-scoped CI/CD variable, so staging and production jobs get different roles without any if. aws sts get-caller-identity prints which role you are, which makes trust-policy mistakes obvious in the log. The entrypoint: [""] override is needed because the image’s entrypoint is aws itself.

Push the image to ECR

Lesson 8 pushed to GitLab’s registry. ECS pulls faster and without extra credentials from ECR, so the pipeline copies the image there:

push-ecr:
  extends: .aws-oidc
  stage: package
  image: docker:29
  services: [docker:29-dind]
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  needs: [build-image]
  before_script:
    - apk add --no-cache aws-cli
    - echo "$AWS_ID_TOKEN" > /tmp/web-identity-token
    - export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/web-identity-token
  script:
    - ECR="$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com"
    - aws ecr get-login-password | docker login -u AWS --password-stdin "$ECR"
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - docker pull "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
    - docker tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" "$ECR/orbit-api:$CI_COMMIT_SHORT_SHA"
    - docker push "$ECR/orbit-api:$CI_COMMIT_SHORT_SHA"
  environment:
    name: staging
    action: prepare
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

environment: action: prepare gives the job the staging role and variables without recording a deployment. Turn on tag immutability on the ECR repository so a tag can never be overwritten.

Roll out on ECS

An ECS service runs a task definition, which names an image. To deploy, register a new revision of the task definition with the new image, then point the service at it and wait until the new tasks are healthy:

deploy-api-staging:
  extends: .aws-oidc
  stage: deploy
  needs: [push-ecr, unit-tests]
  resource_group: staging
  script:
    - yum install -y -q jq
    - IMAGE="$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/orbit-api:$CI_COMMIT_SHORT_SHA"
    - aws ecs describe-task-definition --task-definition orbit-api-staging --query taskDefinition > taskdef.json
    - >
      jq --arg IMAGE "$IMAGE"
      '.containerDefinitions[0].image = $IMAGE
      | del(.taskDefinitionArn, .revision, .status, .requiresAttributes, .compatibilities, .registeredAt, .registeredBy)'
      taskdef.json > new-taskdef.json
    - NEW_ARN=$(aws ecs register-task-definition --cli-input-json file://new-taskdef.json --query taskDefinition.taskDefinitionArn --output text)
    - aws ecs update-service --cluster orbit-staging --service orbit-api --task-definition "$NEW_ARN"
    - aws ecs wait services-stable --cluster orbit-staging --services orbit-api
  environment:
    name: staging
    url: https://staging.orbit.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

aws ecs wait services-stable keeps the job running until the new tasks pass their health checks, so a broken release fails the pipeline instead of looking green. Enable the ECS deployment circuit breaker with automatic rollback on the service, and ECS reverts to the previous task definition by itself when new tasks keep failing.

The amazon/aws-cli image does not include jq, so the job installs it first. For faster jobs, build a small deploy image with both tools and push it to your registry.

Publish the web front end

The front end is static files. Sync them to S3 and invalidate the CloudFront cache so users get the new version:

deploy-web-staging:
  extends: .aws-oidc
  stage: deploy
  needs: [build-web]
  script:
    - aws s3 sync web/dist/ "s3://$WEB_BUCKET/" --delete --cache-control "public,max-age=31536000,immutable" --exclude index.html
    - aws s3 cp web/dist/index.html "s3://$WEB_BUCKET/index.html" --cache-control "no-cache"
    - aws cloudfront create-invalidation --distribution-id "$CF_DISTRIBUTION_ID" --paths "/index.html"
  environment:
    name: staging
    url: https://staging.orbit.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Hashed asset files (app.3f9c2a.js) can be cached forever because their names change with their content. Only index.html, which points at them, must never be cached. That is why only /index.html needs invalidating.

Production

Production repeats the staging jobs with environment: production, when: manual, and production values for AWS_ROLE_ARN, WEB_BUCKET and CF_DISTRIBUTION_ID scoped to the production environment. Use extends so the script is written once:

.deploy-api:
  extends: .aws-oidc
  stage: deploy
  script:
    - ./ci/ecs-deploy.sh "$ECS_CLUSTER" orbit-api "$CI_COMMIT_SHORT_SHA"

deploy-api-staging:
  extends: .deploy-api
  needs: [push-ecr, unit-tests]
  resource_group: staging
  environment: { name: staging, url: "https://staging.orbit.example.com" }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-api-production:
  extends: .deploy-api
  needs: [deploy-api-staging]
  resource_group: production
  interruptible: false
  environment: { name: production, url: "https://orbit.example.com" }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Moving the ECS commands into ci/ecs-deploy.sh keeps the YAML readable and lets you run the same script locally when debugging.

Troubleshooting

ErrorUsual cause
Not authorized to perform sts:AssumeRoleWithWebIdentityThe sub or aud in the trust policy does not match the token: wrong project path, branch, or audience
InvalidIdentityToken: No OpenIDConnect provider foundThe IAM OIDC provider is missing or its URL differs from the token issuer
Incorrect token audienceThe aud in id_tokens differs from the provider’s audience list
ECR no basic auth credentialsdocker login to ECR ran in a different job or the region is wrong
wait services-stable times outNew tasks fail health checks; read the ECS service events and the container logs

To see what GitLab actually sends, decode the token payload in a test job (never in a job that has other secrets): echo "$AWS_ID_TOKEN" | cut -d. -f2 | base64 -d. Compare its sub with your trust policy character by character.

Exercise

A trust policy only checks gitlab.com:aud = https://gitlab.com. What is the risk?

  1. None, the audience is enough to identify your project
  2. Jobs on feature branches can no longer deploy
  3. Any project on GitLab.com could request a token with that audience and assume your role
  4. AWS rejects the policy because it is invalid
Show answer

Answer: C. The audience is something any GitLab job can ask for. The sub claim (or project_path and ref conditions) is what restricts the role to your project and branch. Always check it.

Exercise

Why does the deploy job end with aws ecs wait services-stable?

  1. To give CloudFront time to invalidate its cache
  2. So the job fails if the new tasks never become healthy, instead of passing as soon as the update is requested
  3. Because update-service does nothing until someone calls wait
  4. To keep the temporary credentials alive
Show answer

Answer: B. update-service returns immediately after accepting the request. Waiting until the service is stable ties the job’s result to whether the release actually works.

Key takeaways

  • Never store long-lived AWS keys in GitLab; use id_tokens and an IAM OIDC provider.
  • The trust policy’s sub condition is the real security boundary: project, ref and environment.
  • One role per environment, scoped by environment-scoped variables, with least privilege.
  • Deploy immutable image tags to ECS and wait for the service to become stable.
  • Cache hashed assets forever and invalidate only index.html on CloudFront.

FAQ

Does this work with Google Cloud and Azure too?

Yes. Google Cloud uses Workload Identity Federation and Azure uses federated credentials on an app registration. Both accept GitLab’s ID token with the same id_tokens keyword; only the cloud-side setup differs.

How long do the temporary credentials last?

By default one hour, configurable on the role up to its maximum session duration. Long deploys can request a longer session, but most jobs finish well within an hour.

Can review apps deploy to AWS too?

Yes, with a separate role that trusts merge request pipelines and only has permissions on review resources. Never let feature branches assume the production role.

Should the pipeline also run Terraform?

It can. Many teams keep infrastructure in its own project with its own pipeline (plan on merge requests, apply on main) and the application pipeline only ships code. Lesson 12 shows how the two pipelines can trigger each other.

What about Kubernetes on EKS?

Replace the ECS steps with aws eks update-kubeconfig and kubectl or Helm, or let a GitOps tool such as Argo CD pull the new image tag. The OIDC part stays exactly the same.

↑ ↓ navigate Enter open Esc close