Lesson 07 intermediate

Terraform Data Sources

Read information about infrastructure Terraform does not manage, pin images by digest, look up cloud IDs instead of hard-coding them, fetch secrets with ephemeral resources, and adopt existing objects with import blocks and terraform query.

Not everything your configuration needs is created by that configuration. The shared reverse proxy network was created by another team. The latest approved machine image is published by a build pipeline. Your AWS account ID depends on which credentials run the plan. Data sources let Terraform look these things up at plan time instead of you pasting IDs into code.

Resource versus data source

resourcedata
What Terraform doesCreates, updates, destroysOnly reads
Appears in plan as+, ~, -<= (read)
Destroyed by terraform destroyYesNever: nothing to destroy
Referenced asdocker_network.orbit.iddata.docker_network.proxy.id

The syntax is the same block shape with data instead of resource. The arguments are filters (what to look for) and the attributes are results.

Reading something another team owns

On the shared staging host, a reverse proxy runs on a network called edge, created by the platform team’s own configuration. Orbit’s web container must join it, but Orbit must never modify or delete it:

data "docker_network" "edge" {
  name = "edge"
}

resource "docker_container" "web" {
  name  = "orbit-${var.environment}-web"
  image = docker_image.web.image_id

  networks_advanced {
    name = data.docker_network.edge.name
  }
}

output "edge_subnet" {
  value = data.docker_network.edge.ipam_config
}

If the edge network does not exist, plan fails with a clear “not found” error, which is what you want: a missing dependency should stop the deploy, not create a surprise copy.

Pinning an image by digest

Tags like nginx:1.27-alpine are moved by their publisher when a patch is released. The registry data source turns a tag into the immutable digest it points to right now:

data "docker_registry_image" "web" {
  name = "nginx:1.27-alpine"
}

resource "docker_image" "web" {
  name          = data.docker_registry_image.web.name
  pull_triggers = [data.docker_registry_image.web.sha256_digest]
}

When the publisher moves the tag, the digest changes, pull_triggers changes, and the next plan shows the image (and the container using it) being replaced. You see the update in a plan instead of discovering it after a restart.

Lookups you will use in the cloud

Lesson 12 uses AWS. These four data sources appear in almost every AWS configuration:

data "aws_caller_identity" "current" {}

data "aws_availability_zones" "available" {
  state = "available"
}

data "aws_ssm_parameter" "al2023_ami" {
  name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64"
}

data "aws_iam_policy_document" "read_uploads" {
  statement {
    actions   = ["s3:GetObject"]
    resources = ["arn:aws:s3:::orbit-uploads-${data.aws_caller_identity.current.account_id}/*"]
  }
}
  • aws_caller_identity gives the account ID of whoever runs the plan, so bucket names and ARNs are never hard-coded per account.
  • aws_availability_zones lists the zones of the region instead of assuming a, b and c exist.
  • The SSM parameter is the official, always-current Amazon Linux image ID. It is simpler and less error-prone than searching AMIs by name with aws_ami.
  • aws_iam_policy_document builds IAM JSON from HCL: no hand-written JSON, no quoting mistakes.

Reading another configuration’s outputs

Orbit’s network lives in its own configuration and state, and the application configuration needs the subnet IDs. The terraform_remote_state data source reads another state’s outputs:

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "orbit-tfstate-123456789012"
    key    = "orbit/network/terraform.tfstate"
    region = "eu-west-1"
  }
}

locals {
  private_subnet_ids = data.terraform_remote_state.network.outputs.private_subnet_ids
}

It works, but it needs read access to the whole network state, secrets included. Two alternatives are often better: publish the values to a parameter store and read them with a normal data source, or look the subnets up directly with aws_subnets and a tag filter.

Secrets: use ephemeral resources, not data sources

A data source’s results are saved in state. If you read a database password with data "aws_secretsmanager_secret_version", the password ends up in the state file. Terraform 1.10 added ephemeral resources, which fetch a value for the duration of a run and never write it to state or plan:

ephemeral "aws_secretsmanager_secret_version" "db" {
  secret_id = "orbit/staging/db"
}

provider "postgresql" {
  host     = aws_db_instance.orbit.address
  username = "orbit_admin"
  password = jsondecode(ephemeral.aws_secretsmanager_secret_version.db.secret_string)["password"]
}

Ephemeral values can flow into provider blocks, other ephemeral values and write-only arguments. Lesson 15 puts the full secret story together.

Bringing existing objects under Terraform: import blocks

Data sources read things you do not want to manage. When you do want Terraform to take over something created by hand, import it. Since Terraform 1.5 that is a block in code, reviewed like any other change:

import {
  to = docker_volume.uploads
  id = "orbit-staging-uploads"
}

You can even let Terraform write the resource block for you:

$ terraform plan -generate-config-out=generated.tf
  # docker_volume.uploads will be imported
Plan: 1 to import, 0 to add, 0 to change, 0 to destroy.
$ terraform apply

Review generated.tf, tidy it up (remove computed attributes, replace hard-coded values with references) and move it into your normal files. After the apply, the import block can be deleted. Import blocks also accept for_each (1.7+) for importing many objects at once, and since 1.16 they work inside modules.

Finding what to import: terraform query

Importing needs an ID for every object, which is painful when you are adopting a whole account built by hand. Terraform 1.14 added terraform query, which lists existing infrastructure through providers that support it. You write list blocks in files ending in .tfquery.hcl:

# discover.tfquery.hcl
list "aws_instance" "all" {
  provider = aws
}
$ terraform query
$ terraform query -generate-config-out=to-import/instances.tf

The first command prints what was found. The second writes import and resource blocks for every result into a new file, ready to review. Provider-specific filters (by tag, by name) go in a config { } block inside list; check the provider documentation for which resource types support listing.

When data is read

Terraform reads data sources during plan whenever their arguments are known. If an argument depends on a resource that does not exist yet, the read is deferred to apply, and every attribute that depends on it shows (known after apply). That can make a plan much less informative, so avoid feeding data sources with values from resources created in the same run when a plain reference would do.

Do not add depends_on to data sources “just in case”: it has the same effect and pushes the read to apply time on every run.

Quick reference

TaskSyntax or command
Read an existing objectdata "docker_network" "edge" { name = "edge" }
Reference itdata.docker_network.edge.id
Current AWS account IDdata.aws_caller_identity.current.account_id
Another state’s outputsdata.terraform_remote_state.network.outputs.<name>
Fetch a secret without storing it (1.10)ephemeral "<type>" "<name>" { ... }
Adopt an existing object (1.5)import { to = ..., id = "..." }
Generate config for importsterraform plan -generate-config-out=generated.tf
List existing infrastructure (1.14)terraform query
Generate imports from a queryterraform query -generate-config-out=to-import/file.tf

Exercise

Orbit’s containers must join the platform team’s existing edge network, and Orbit must never delete it. How should Orbit’s code refer to it?

  1. With a data “docker_network” block that looks it up by name
  2. With a resource “docker_network” block with the same name
  3. With an import block, so Orbit’s state tracks it
  4. By hard-coding the network ID
Show answer

Answer: A. A data source only reads, so Orbit can use the network without owning it. A resource or import would make Orbit’s terraform destroy delete the shared network.

Exercise

You read a database password with a data source and it works. Why is an ephemeral resource the better choice?

  1. Data sources cannot read Secrets Manager
  2. Ephemeral resources are faster
  3. Data source results are stored in state, so the password would sit in the state file; ephemeral values are never stored
  4. Data sources are deprecated since Terraform 1.10
Show answer

Answer: C. Everything a data source returns is saved in state. Ephemeral resources (1.10+) keep the secret in memory only for the run.

Key takeaways

  • Data sources read; resources manage. A data source is never destroyed.
  • Look up IDs, accounts, zones and images instead of hard-coding them.
  • Pin container images by digest with docker_registry_image and pull_triggers.
  • terraform_remote_state works but exposes the whole state; prefer a parameter store or direct lookups.
  • Fetch secrets with ephemeral resources so they never reach state.
  • Adopt existing objects with import blocks and generated config; find them with terraform query (1.14).

FAQ

Is terraform import (the command) still used?

It still works, but the import block is better: it shows up in plan, can be reviewed in a pull request, can generate configuration and can import many objects with for_each. Use the command only for quick one-offs.

Why does my plan say a data source will be read during apply?

One of its arguments depends on something not known until apply, often a resource created in the same run or a depends_on. Pass known values where you can so the read happens at plan time.

Can a data source fail the plan if nothing matches?

Most do, with a “not found” or “no matching” error. A few list-style data sources (such as aws_subnets) return an empty list instead; add a postcondition if an empty result should be an error.

Do data sources cost money?

They only make read API calls. Very large configurations with hundreds of lookups can hit API rate limits, which is another reason to keep configurations reasonably small.

↑ ↓ navigate Enter open Esc close