Secure Terraform in AWS: Remote State, Checkov & OpenTofu

The Transition from "ClickOps" to Infrastructure as Code (IaC)

Provisioning cloud resources on public cloud providers like Amazon Web Services (AWS) using web management consoles (ClickOps) introduces systemic vulnerabilities in enterprise environments: lack of historical audit trails, absence of peer review gates, unmonitored configuration drift, and frequent human errors that inadvertently expose databases or open management ports to the Internet.

Infrastructure as Code (IaC) resolves these operational liabilities by defining cloud architecture declaratively and reproducibly in version-controlled Git repositories. Within this domain, Terraform and its Linux Foundation open-source fork, OpenTofu, represent the dominant industry standards.

However, adopting IaC without rigorous security controls introduces critical attack vectors of its own:

• State File Leakage (terraform.tfstate): The Terraform state file contains a full inventory of deployed resources and persistently stores processed secrets in plaintext (RDS master passwords, TLS private certificates, API tokens, and access credentials).
• Concurrency Race Conditions: Multiple engineers or parallel CI/CD jobs executing apply without state locking can irrevocably corrupt production state.
• Uninspected Misconfigurations: Accidental deployment of overly permissive Security Groups (0.0.0.0/0), unencrypted S3 buckets, or missing KMS keys due to the lack of static security analysis in deployment pipelines.
• Configuration Drift: Out-of-band manual changes made directly in the cloud console that break alignment between running infrastructure and declarative source code.

Enterprise-grade IaC requires safeguarding the entire delivery lifecycle: from encrypted remote state storage to automated static analysis using tools like Checkov.

---

DevSecOps IaC Pipeline: From Pull Request to Automated Deployment

IaC security must be embedded directly into automated CI/CD pipelines as progressive quality gates, following the core principles of what is DevSecOps:

graph TD
    subgraph Dev_Environment ["👨‍💻 Developer Environment"]
        Dev["Engineer / SRE"]
        Git["Git Commit & Push"]
    end

    subgraph CI_Pipeline ["⚙️ CI/CD Pipeline (GitHub Actions / GitLab CI)"]
        Lint["terraform fmt & validate"]
        SecretScan["Secret Detection: Gitleaks"]
        IaCScan["Static Security Scan: Checkov / Trivy"]
        Plan["terraform plan -out=tfplan"]
        Gate{"Security Quality Gate Passed?"}
    end

    subgraph Approval_Gate ["🛡️ Governance & Approval"]
        Review["Peer Plan Review (PR Approval)"]
    end

    subgraph CD_Deployment ["🚀 Deployment in AWS Cloud"]
        LockDB["DynamoDB: Acquire State Lock"]
        Apply["terraform apply tfplan"]
        StateS3["S3 Bucket: Save tfstate (SSE-KMS)"]
        UnlockDB["DynamoDB: Release State Lock"]
    end

    Dev --> Git
    Git --> Lint
    Lint --> SecretScan
    SecretScan --> IaCScan
    IaCScan --> Plan
    Plan --> Gate
    Gate -->|Failure: Hard Block| Dev
    Gate -->|Passed| Review
    Review --> LockDB
    LockDB --> Apply
    Apply --> StateS3
    StateS3 --> UnlockDB

    classDef dev fill:#e1f5fe,stroke:#0288d1,stroke-width:2px;
    classDef ci fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px;
    classDef gate fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
    classDef deploy fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;

    class Dev,Git dev;
    class Lint,SecretScan,IaCScan,Plan,Gate ci;
    class Review gate;
    class LockDB,Apply,StateS3,UnlockDB deploy;

---

1. Hardening the Remote Backend: S3 and DynamoDB

By default, Terraform records state locally in a terraform.tfstate file. In team and production environments, state must be stored in a secured AWS remote backend combining Amazon S3 for durable encrypted persistence and Amazon DynamoDB for concurrency state locking.

S3 Bucket Hardening Baseline:

1. Block All Public Access: Set BlockPublicAccess = true across all bucket ACLs and policies.
2. Object Versioning: Enabled to permit recovery against state corruption or unintended deletions.
3. Mandatory Encryption at Rest (SSE-KMS): Encrypt with a Customer Managed Key (CMK), avoiding shared default AWS keys.
4. Secure Transport Policy: Enforce aws:SecureTransport = true to mandate TLS 1.3/1.2 for all API operations.

The backend configuration in Terraform/OpenTofu is declared as follows:

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.40"
    }
  }

  backend "s3" {
    bucket         = "corp-production-terraform-state-eu-west-1"
    key            = "core-infra/vpc-rds.tfstate"
    region         = "eu-west-1"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:eu-west-1:123456789012:key/b2c3d4e5-6789-01ab-cdef-222222222222"
    dynamodb_table = "terraform-state-lock-production"
  }
}

The DynamoDB locking table requires a single String primary key named LockID. Whenever terraform apply or plan runs, Terraform acquires a lease lock in DynamoDB; any concurrent execution is halted immediately with Error: Error acquiring the state lock, preventing race conditions.

---

2. Secret Management in Terraform: Overcoming the State Dilemma

A common misconception is assuming that setting sensitive = true on a Terraform variable encrypts its value.

The sensitive = true attribute merely suppresses output in console logs (stdout) during plan and apply; the plaintext value is still written directly into the JSON state file.

Antipatterns to Avoid:

• Storing database passwords in local .tfvars files committed to Git.
• Generating passwords using the randompassword provider and passing them to awsdb_instance: the password will be serialized unencrypted into the state.

Recommended Architectural Solution: Managed Secrets Rotation

Delegate credential generation to specialized services like AWS Secrets Manager with native automated rotation:

# Create AWS Secrets Manager container governed by KMS
resource "aws_secretsmanager_secret" "db_credentials" {
  name                    = "production/rds/mysql-app"
  kms_key_id             = aws_kms_key.infrastructure_key.arn
  recovery_window_in_days = 0
}

# Master password is NOT declared in Terraform; RDS auto-generates and rotates it
resource "aws_db_instance" "production_database" {
  identifier                  = "corp-production-db"
  allocated_storage           = 100
  engine                      = "mysql"
  engine_version              = "8.0"
  instance_class              = "db.r6g.large"
  manage_master_user_password = true # RDS manages and rotates password in Secrets Manager
  master_user_secret_kms_key_id = aws_kms_key.infrastructure_key.arn
  username                    = "dbadmin"

  vpc_security_group_ids = [aws_security_group.db_sg.id]
  db_subnet_group_name   = aws_db_subnet_group.private.name
  publicly_accessible    = false
  storage_encrypted      = true
  kms_key_id             = aws_kms_key.infrastructure_key.arn

  skip_final_snapshot = false
  final_snapshot_identifier = "corp-production-db-final-snapshot"
}

With managemasteruser_password = true, AWS RDS provisions the secret in Secrets Manager and handles continuous rotation. The secret is never handled by Terraform nor stored in the state file.

---

3. Static IaC Security Analysis with Checkov in CI/CD

Static Application Security Testing (SAST) for infrastructure allows engineering teams to identify security misconfigurations before resources are created in the cloud. Checkov scans HCL code against over 1,000 security policies derived from CIS Benchmarks.

GitHub Actions Workflow Integration:

The following workflow enforces automated security scanning on Pull Requests modifying Terraform files:

name: "Terraform DevSecOps Quality Gate"

on:
  pull_request:
    paths:
      - 'terraform/**'

permissions:
  contents: read
  pull-requests: write

jobs:
  security-scan:
    name: "Checkov Security & Compliance Scan"
    runs-on: ubuntu-latest
    steps:
      - name: "Checkout Code"
        uses: actions/checkout@v4

      - name: "Run Gitleaks Secret Detector"
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: "Set up Python & Install Checkov"
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: "Run Checkov Static Analysis"
        run: |
          pip install checkov
          checkov --directory terraform/ \
                  --framework terraform \
                  --output cli \
                  --output github_failed_only \
                  --soft-fail false \
                  --skip-check CKV_AWS_144 # Documented exception if justified

If code introduces a Security Group rule with 0.0.0.0/0 on port 3306 or provisions an unencrypted S3 bucket, Checkov exits with status code 1, halting the PR build and preventing merge.

---

4. OpenTofu: Client-Side Native State Encryption

Following HashiCorp's 2023 license transition to the Business Source License (BSL), the open-source community under the Linux Foundation introduced OpenTofu.

OpenTofu provides seamless drop-in compatibility with HCL syntax and existing providers, while delivering an essential security capability: Native Client-Side State Encryption.

Configuring Native State Encryption in OpenTofu:

Rather than relying solely on server-side S3 encryption, OpenTofu encrypts state payloads on the client before network transmission:

# OpenTofu native configuration (v1.7+)
terraform {
  encryption {
    key_provider "aws_kms" "production_kms" {
      kms_key_id = "arn:aws:kms:eu-west-1:123456789012:key/b2c3d4e5-6789-01ab-cdef-222222222222"
      region     = "eu-west-1"
    }

    method "aes_gcm" "state_encryption" {
      keys = key_provider.aws_kms.production_kms
    }

    state {
      method   = method.aes_gcm.state_encryption
      enforced = true
    }

    plan {
      method   = method.aes_gcm.state_encryption
      enforced = true
    }
  }
}

Even if an attacker gains unauthorized read access to the S3 bucket or captures a tfplan file, the payload cannot be decrypted without explicit kms:Decrypt permissions on the specified KMS key.

---

5. Production IaC Governance Recommendations

To achieve high operational maturity in corporate IaC deployments:

1. Deployment Exclusively Through CI/CD: Revoke manual apply permissions from individual developer workstations. Only CI/CD runners federated via OpenID Connect (OIDC) should possess privileges to alter cloud infrastructure.
2. Deterministic Plan Enforcement: Always execute apply using pre-generated and approved plan files (terraform apply tfplan), guaranteeing that deployed changes match the reviewed Pull Request.
3. Automated Drift Detection: Schedule daily background cron jobs running terraform plan -detailed-exitcode. Any out-of-band manual modification triggers immediate alerting via Slack or PagerDuty.
4. Architectural Privilege Separation: Separate infrastructure pipelines into discrete tiers (networking, databases, compute) so that database runners cannot manipulate IAM policies or Kubernetes clusters.

---

Enterprise Security Checklist for Terraform & OpenTofu

• [x] Remote S3 state secured with public access block, object versioning, and KMS CMK encryption.
• [x] Deployment concurrency race conditions mitigated via DynamoDB state locking.
• [x] Sensitive database credentials managed through AWS Secrets Manager, keeping secrets out of tfstate.
• [x] Mandatory SAST quality gates implemented in CI/CD using Checkov and Gitleaks.
• [x] Provisioning pipelines authenticated via cloud OIDC federation, eliminating static AWS access keys.
• [x] Automated drift detection executed on schedule to identify unmanaged manual infrastructure modifications.

If you need to audit your organization's infrastructure as code, establish secure AWS DevSecOps pipelines, or migrate to OpenTofu with client-side state encryption, explore my Cybersecurity Consulting Services or reach out directly via About Me.