Lesson 08 intermediate

Terraform count and for_each

Create many similar resources from one block, understand why count can recreate the wrong things when a list changes, use for_each with maps and sets, move between the two safely and fix the Invalid for_each argument error.

Orbit is growing: the API now runs three worker containers, and each environment needs its own set of volumes. Copying the same resource block five times works until you need to change all of them. Terraform has two meta-arguments that turn one block into many instances: count and for_each. Choosing the right one matters more than it looks.

count: N identical copies

count takes a number. Terraform creates that many instances, addressed by position, and count.index tells each one which it is:

variable "worker_count" {
  type    = number
  default = 3
}

resource "docker_container" "worker" {
  count = var.worker_count

  name    = "orbit-${var.environment}-worker-${count.index}"
  image   = docker_image.api.image_id
  command = ["orbit", "worker", "--queue", "default"]
}
$ terraform state list
docker_container.worker[0]
docker_container.worker[1]
docker_container.worker[2]

Reference one instance with docker_container.worker[0].id, or all of them with a splat: docker_container.worker[*].name.

count as an on/off switch

The most useful everyday trick with count: create something only when a flag is on.

variable "enable_adminer" {
  type    = bool
  default = false
}

resource "docker_container" "adminer" {
  count = var.enable_adminer ? 1 : 0

  name  = "orbit-${var.environment}-adminer"
  image = docker_image.adminer.image_id
}

output "adminer_url" {
  value = one(docker_container.adminer[*].name)
}

one() returns the single element, or null when the list is empty, so the output works in both cases.

The trouble with count and lists

Now the workers become different queues, and someone drives count from a list:

variable "queues" {
  type    = list(string)
  default = ["emails", "thumbnails", "exports"]
}

resource "docker_container" "worker" {
  count   = length(var.queues)
  name    = "orbit-worker-${var.queues[count.index]}"
  command = ["orbit", "worker", "--queue", var.queues[count.index]]
  image   = docker_image.api.image_id
}

The emails queue is retired, so the list becomes ["thumbnails", "exports"]. The plan:

$ terraform plan
  # docker_container.worker[0] must be replaced   (emails -> thumbnails)
  # docker_container.worker[1] must be replaced   (thumbnails -> exports)
  # docker_container.worker[2] will be destroyed  (exports)
Plan: 2 to add, 0 to change, 3 to destroy.

You removed one worker, and Terraform wants to rebuild all three. Instances are identified by position, and every position after the removed item shifted. With containers that is a short blip; with databases, disks or DNS records it is an outage.

for_each: instances with names

for_each takes a set of strings or a map, and identifies each instance by its key instead of its position:

resource "docker_container" "worker" {
  for_each = toset(var.queues)

  name    = "orbit-${var.environment}-worker-${each.key}"
  command = ["orbit", "worker", "--queue", each.value]
  image   = docker_image.api.image_id
}
$ terraform state list
docker_container.worker["emails"]
docker_container.worker["exports"]
docker_container.worker["thumbnails"]

Remove emails from the list now and the plan destroys exactly worker["emails"] and nothing else.

With a set, each.key and each.value are the same string. With a map, you can carry per-instance settings:

variable "workers" {
  type = map(object({
    concurrency = number
    memory_mb   = optional(number, 256)
  }))
  default = {
    emails     = { concurrency = 4 }
    thumbnails = { concurrency = 2, memory_mb = 1024 }
    exports    = { concurrency = 1 }
  }
}

resource "docker_container" "worker" {
  for_each = var.workers

  name    = "orbit-${var.environment}-worker-${each.key}"
  image   = docker_image.api.image_id
  memory  = each.value.memory_mb
  command = ["orbit", "worker", "--queue", each.key, "--concurrency", tostring(each.value.concurrency)]
}

output "worker_names" {
  value = { for queue, c in docker_container.worker : queue => c.name }
}

count or for_each?

SituationUse
Create something or not, based on a flagcount = var.enabled ? 1 : 0
N truly identical copies, where any one can gocount
Instances that differ, or that you may remove from the middlefor_each
Anything holding data (volumes, databases, buckets)for_each
A list from a variablefor_each = toset(var.list)

When in doubt, use for_each. Its only cost is that keys must be strings and known at plan time.

Switching from count to for_each without rebuilding

You inherit the count version and want to switch. Changing the code alone would destroy every [0], [1], [2] and create new keyed instances. Add moved blocks to map old addresses to new ones:

moved {
  from = docker_container.worker[0]
  to   = docker_container.worker["emails"]
}

moved {
  from = docker_container.worker[1]
  to   = docker_container.worker["thumbnails"]
}

moved {
  from = docker_container.worker[2]
  to   = docker_container.worker["exports"]
}

The plan shows three moves and zero replacements. Also note: adding count to a resource that had none moves the single instance to [0] automatically, so a plain on/off switch never needs a moved block.

The “Invalid for_each argument” error

Error: Invalid for_each argument
  The "for_each" set includes values derived from resource attributes that
  cannot be determined until apply...

Terraform must know every key at plan time, because the keys are the instance addresses. This fails:

resource "docker_volume" "cache" {
  for_each = toset([docker_container.api.id])   # ID is unknown until apply
  name     = "cache-${each.key}"
}

Fixes, from best to worst:

  • Use keys you choose, such as names from a variable, and put unknown values in each.value only. Values may be unknown; keys may not.
  • Split the configuration so the value already exists when the second part is planned.
  • terraform apply -target=... to create the dependency first. It works once but leaves a trap for the next person.

for_each on modules and imports

count and for_each also work on module blocks, which is how you create one cache per service:

module "cache" {
  source   = "./modules/cache"
  for_each = toset(["api", "sessions"])

  name    = "orbit-${var.environment}-${each.key}-cache"
  network = docker_network.orbit.name
}

Read outputs with module.cache["api"].address. And since Terraform 1.7, import blocks accept for_each too, so adopting ten existing volumes is one block:

import {
  for_each = toset(["emails", "thumbnails", "exports"])
  to       = docker_volume.spool[each.key]
  id       = "orbit-staging-spool-${each.key}"
}

Quick reference

TaskSyntax
N copiescount = 3, then count.index
Optional resourcecount = var.enabled ? 1 : 0
Single optional valueone(resource.x[*].attr)
Instances from a listfor_each = toset(var.names)
Instances from a mapfor_each = var.workers, then each.key, each.value
One instancedocker_container.worker["emails"]
All names as a list (count)docker_container.worker[*].name
All names as a map (for_each){ for k, c in docker_container.worker : k => c.name }
Rename an instancemoved { from = x[0] to = x["emails"] }

Exercise

Volumes are created with count from the list [“a”, “b”, “c”]. You remove “a”. What does the plan do?

  1. Destroys only the volume for “a”
  2. Replaces the volumes at positions 0 and 1 and destroys position 2, because every later item shifted
  3. Nothing, because count ignores list contents
  4. Fails with Invalid count argument
Show answer

Answer: B. count addresses instances by index. Removing an item shifts all later ones, so their configuration changes. With for_each = toset(…) only the removed key would be destroyed.

Exercise

Which for_each value causes “Invalid for_each argument” during plan?

  1. toset(var.queue_names)
  2. A map variable whose values include IDs from other resources
  3. toset([docker_container.api.id]), where the container is created in the same run
  4. An empty set
Show answer

Answer: C. Keys must be known at plan time because they become instance addresses. Unknown values are fine inside each.value, but not as keys.

Key takeaways

  • count addresses instances by position; for_each by key.
  • Removing an item from the middle of a count list shifts and rebuilds everything after it.
  • Use count for on/off switches and truly identical copies; for_each for everything else.
  • Switch from count to for_each with moved blocks so nothing is recreated.
  • for_each keys must be known at plan time; unknown values belong in each.value.
  • Modules and import blocks (1.7+) support for_each too.

FAQ

Can I use count and for_each on the same resource?

No, a block can have one or the other. If you need a flag and a map, filter the map: for_each = var.enabled ? var.workers : {}.

Why does for_each not accept a list?

Lists have order and can contain duplicates, which is exactly what makes count fragile. Convert with toset(), or build a map with a for expression when you need richer keys.

Can I use for_each over a list of objects?

Turn it into a map first, keyed by a unique attribute: for_each = { for w in var.workers : w.name => w }. Lesson 9 covers for expressions.

How do I reference the previous instance of a resource in the same for_each?

You cannot: instances of one block are created in parallel with no order between them. If one must come after another, split them into separate blocks.

↑ ↓ navigate Enter open Esc close