Lesson 13 intermediate

Terraform Provisioners

How local-exec, remote-exec and file provisioners work, why HashiCorp calls them a last resort, how to run scripts cleanly with terraform_data, and how the newer actions feature (1.14 to 1.16) replaces many provisioner use cases.

Terraform is declarative: it compares desired state with real state and makes API calls. Sometimes you need a plain imperative step: run a database migration after the API is deployed, warm a cache, call a webhook, clear a CDN. Provisioners were Terraform’s original answer. They still work, but they are the part of Terraform most likely to cause trouble, and since Terraform 1.14 there is often a better tool: actions.

The three provisioners

A provisioner is a block inside a resource that runs once, after the resource is created:

  • local-exec runs a command on the machine running Terraform.
  • remote-exec connects to the new machine over SSH or WinRM and runs commands there.
  • file copies files or folders to the new machine over the same connection.
resource "docker_container" "api" {
  name  = "orbit-${var.environment}-api"
  image = docker_image.api.image_id

  provisioner "local-exec" {
    command = "./scripts/wait-for-http.sh http://localhost:${var.api_port}/healthz 60"
  }
}

remote-exec and file need a connection block describing how to reach the machine:

resource "aws_instance" "legacy" {
  # ...
  connection {
    type        = "ssh"
    host        = self.public_ip
    user        = "ec2-user"
    private_key = file("~/.ssh/orbit-legacy.pem")
  }

  provisioner "file" {
    source      = "files/orbit.conf"
    destination = "/tmp/orbit.conf"
  }

  provisioner "remote-exec" {
    inline = [
      "sudo mv /tmp/orbit.conf /etc/orbit.conf",
      "sudo systemctl restart orbit",
    ]
  }
}

Inside provisioners and connections, use self to refer to the resource being created.

How provisioners behave

  • Creation-time only by default. A provisioner runs once, when the resource is created. Changing its command later does nothing to existing resources.
  • Failure taints the resource. If a creation-time provisioner fails, the resource is marked tainted and the next apply destroys and recreates it. Set on_failure = continue to ignore failures instead.
  • Destroy-time provisioners run before the resource is destroyed, with when = destroy. They can only reference self, not other resources or variables, and they do not run if you delete the resource block from your code.
  • Invisible in plans. The plan shows that a resource will be created, not what the script will do. Reviewers approve blind.

Why they are a last resort

HashiCorp’s own documentation calls provisioners a last resort. The reasons are practical:

ProblemConsequence
Terraform cannot model what the script didNo drift detection, no plan, no undo
Run only at creationConfiguration changes need a full replacement to take effect
remote-exec needs network access and credentials from the runnerOpen SSH ports, private keys in CI, firewalls in the way
Failures taint resourcesA flaky script can recreate a production server
Depend on tools on the runnerWorks on your laptop, fails in CI

Better alternatives, by use case

You want to…Use instead
Install software on a new VMuser_data / cloud-init (lesson 12)
Ship a fully configured serverA prebuilt image (Packer, EC2 Image Builder)
Keep servers configured over timeAnsible, or a container platform
Create something the provider supportsThe provider’s resource (users, databases, DNS records…)
Run an operation with no state (invalidate, invoke, notify)An action (1.14+)
Run a script when something changesterraform_data with triggers_replace

Running a script when something changes: terraform_data

When you truly need a script, attach it to a terraform_data resource rather than to the real resource. terraform_data is built into Terraform (no provider needed) and is replaced whenever triggers_replace changes, which re-runs its provisioner:

resource "terraform_data" "db_migrate" {
  triggers_replace = [docker_container.api.image]

  provisioner "local-exec" {
    command = "docker exec ${docker_container.api.name} orbit migrate --yes"
  }
}

Now the migration runs once after the first deploy and again every time the API image changes, and a failing migration taints the small terraform_data resource, not the API container. Since Terraform 1.16, terraform_data also has a store block for holding ephemeral and sensitive values between plan and apply.

Actions: operations Terraform can plan

Many provisioner scripts are really “call one API after something changes”: invalidate a CDN, invoke a function, start a job. Terraform 1.14 added actions for exactly that. An action is defined by the provider, appears in the plan, and runs either when a resource changes or on demand.

Orbit’s static assets live in S3 behind CloudFront. After index.html changes, the CDN cache must be cleared:

action "aws_cloudfront_create_invalidation" "web" {
  config {
    distribution_id = aws_cloudfront_distribution.web.id
    paths           = ["/*"]
  }
}

resource "aws_s3_object" "index" {
  bucket       = aws_s3_bucket.web.id
  key          = "index.html"
  source       = "dist/index.html"
  etag         = filemd5("dist/index.html")
  content_type = "text/html"

  lifecycle {
    action_trigger {
      events     = [after_create, after_update]
      actions    = [action.aws_cloudfront_create_invalidation.web]
      on_failure = continue
    }
  }
}

The plan lists the action next to the change that triggers it, so reviewers see it. The available events are before_create, after_create, before_update and after_update, plus before_destroy and after_destroy since 1.16. Also new in 1.16, on_failure decides what happens when an action fails: halt stops the apply, taint marks the triggering resource for replacement and continue carries on.

You can also run an action by hand, without changing any resource. Useful when someone asks “can you just clear the cache?”:

$ terraform plan -invoke=action.aws_cloudfront_create_invalidation.web
$ terraform apply -invoke=action.aws_cloudfront_create_invalidation.web

Which actions exist depends on the provider version; the AWS provider ships actions such as Lambda invocation and CloudFront invalidation, and the list grows with each release.

Quick reference

TaskSyntax
Run a command locally after createprovisioner "local-exec" { command = "..." }
Run commands on the new machineprovisioner "remote-exec" { inline = [...] }
Copy a file to the new machineprovisioner "file" { source = ..., destination = ... }
Run at destroywhen = destroy inside the provisioner
Ignore a failing provisioneron_failure = continue
Re-run a script when something changesterraform_data with triggers_replace
Define an action (1.14)action "<type>" "<name>" { config { ... } }
Trigger it on changelifecycle { action_trigger { events = [...], actions = [...] } }
Run an action on demandterraform apply -invoke=action.<type>.<name>

Exercise

A local-exec provisioner that registers a new server with monitoring fails during apply. What happens to the server resource?

  1. Nothing, provisioner failures are only warnings
  2. Terraform deletes the server immediately
  3. It is marked tainted, so the next apply destroys and recreates it
  4. The provisioner is retried automatically until it succeeds
Show answer

Answer: C. A failing creation-time provisioner taints its resource by default. Use on_failure = continue, or better, move the step to terraform_data or an action so a flaky script cannot force a server rebuild.

Exercise

After every change to index.html in S3, the CloudFront cache must be invalidated, and reviewers should see that in the plan. What fits best in Terraform 1.17?

  1. A local-exec provisioner on the S3 object that calls the AWS CLI
  2. An aws_cloudfront_create_invalidation action triggered by an action_trigger on the S3 object
  3. A cron job outside Terraform
  4. remote-exec on the CloudFront distribution
Show answer

Answer: B. Actions are planned, visible to reviewers, run by the provider without extra tools on the runner, and can be triggered by resource changes or invoked by hand.

Key takeaways

  • Provisioners run imperative steps that Terraform cannot plan, track or undo.
  • They run only at creation (or destroy), and failures taint the resource.
  • Prefer cloud-init, images, configuration tools or provider resources.
  • For scripts that must re-run on change, use terraform_data with triggers_replace.
  • Actions (1.14+) model operations like invalidations and invocations, show in plans and can be run with -invoke.
  • 1.16 adds destroy events and on_failure modes to actions.

FAQ

Are provisioners deprecated?

No. They are supported and still useful in edge cases, such as bootstrapping a system with no API. They are just the option to reach for last.

Why didn’t my changed provisioner command run?

Provisioners run when their resource is created. Changing the command does not recreate the resource. Put the script in terraform_data with a trigger, or use terraform apply -replace to force one run.

What happened to null_resource?

terraform_data (built in since 1.4) replaces null_resource from the null provider. Use triggers_replace where you used triggers. Existing null_resource blocks can be migrated with a moved block.

Can actions replace Ansible?

No. Actions call single provider operations; they do not configure servers. They replace the “call one API after a change” kind of provisioner, which was the most common kind.

↑ ↓ navigate Enter open Esc close