Stage 4 · Provision

Terraform State Operations

State Recovery

State backups, force-unlock, state rm, state mv, and restoring from backend versioning — recovering from state corruption.

8 min readInfrastructure as CodeProvision

State Corruption Scenarios

Stale lock — a crashed CI job left a lock in DynamoDB.
Corrupted state — manual edits or failed writes corrupted the state file.
Lost state — the state file was accidentally deleted.
Version mismatch — the state was written by a different Terraform version.
Concurrent writes — two processes wrote to the same state file without locking.

State Backups

Creating and restoring backups


$ terraform state pull > backup-$(date +%Y%m%d).json

# Restore from backup
$ terraform state push backup-20240115.json

# Backup before risky operations
$ terraform state pull > before-migration.json
$ terraform state mv ...  # risky operation
$ terraform state pull > after-migration.json

terraform state pull downloads the state file. Always backup before state operations. terraform state push uploads a state file to the backend. Verify the backup before restoring.

Force Unlock

Force unlock workflow


$ terraform apply
Error: Error acquiring the state lock

  Error message: ConditionalCheckFailedException: The conditional request failed
  Lock Info:
    ID:        abc-123-def
    Path:      s3://my-terraform-state/prod/vpc/terraform.tfstate
    Operation: OperationTypeApply
    Who:       user@machine
    Version:   1.5.0
    Created:   2024-01-15 10:00:00 +0000 UTC

# Verify the lock is stale
# Check if the process is still running

# Force unlock
$ terraform force-unlock abc-123-def

# Or manually delete the DynamoDB item
$ aws dynamodb delete-item \
    --table-name terraform-locks \
    --key '{"LockID": {"S": "s3://my-terraform-state/prod/vpc/terraform.tfstate"}}'

Force unlock releases the lock by ID. Only use it when you are certain no other process is running. The lock info shows who acquired it and when. Verify the process is dead before unlocking.

State Remove

Removing resources from state


$ terraform state rm aws_instance.legacy
$ terraform state rm 'aws_instance.web["deprecated"]'
$ terraform state rm module.old_name.aws_instance.web

# Remove all resources in a module
$ terraform state list | grep module.old | xargs -I {} terraform state rm {}

terraform state rm removes resources from state without destroying them. The resource continues to exist but Terraform no longer manages it. Use this for resources that are no longer managed by Terraform.

State Move

Moving resources in state


$ terraform state mv aws_instance.web aws_instance.web_server
$ terraform state mv 'aws_instance.web["a"]' 'aws_instance.web["primary"]'
$ terraform state mv module.old module.new
$ terraform state mv aws_instance.web module.compute.aws_instance.web

terraform state mv renames or moves resources in state. The resource is not recreated. Always backup before running state mv commands.

Restoring from Versioning

S3 versioning restore


$ aws s3api list-object-versions \
    --bucket my-terraform-state \
    --prefix prod/vpc/terraform.tfstate

# Restore a specific version
$ aws s3api get-object \
    --bucket my-terraform-state \
    --key prod/vpc/terraform.tfstate \
    --version-id "abc123" \
    restored-state.json

# Push the restored state
$ terraform state push restored-state.json

S3 versioning stores every version of the state file. Use list-object-versions to find the version ID. get-object downloads a specific version. Push the restored state to recover from corruption.

Azure blob versioning restore


$ az storage blob list \
    --container-name tfstate \
    --account-name tfstate \
    --prefix prod/vpc/terraform.tfstate \
    --output table

$ az storage blob download \
    --container-name tfstate \
    --account-name tfstate \
    --name prod/vpc/terraform.tfstate \
    --file restored-state.json \
    --snapshot "2024-01-15T10:00:00.0000000Z"

Azure Blob Storage supports blob versioning and snapshots. Use the snapshot timestamp to download a specific version. Push the restored state to recover.

Test state recovery regularly

Practice state recovery in a test environment. Verify you can restore from backups, versioning, and force-unlock. Document the process and keep it accessible to the team.

State push can corrupt state

terraform state push overwrites the current state. If you push the wrong file, you corrupt the state. Always verify the backup content before restoring. Use terraform state pull to verify the current state first.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.