What a DevOps Audit Actually Covers (and What It Finds)

What a DevOps Audit Actually Covers (and What It Finds)

An infrastructure audit is an awkward purchase. The deliverable is a document, the value is judgement rather than hours, and you cannot tell a good one from a bad one until it lands on your desk. Meanwhile your team already has a list of things they know are wrong, so the obvious question is whether you are paying someone to write that list down again.

Here is what a serious audit inspects, how it is run, what it costs you in your own team's attention, and when to skip it.

What "audit" should mean here

An audit is a bounded, read-only assessment of your infrastructure, delivery process, and operational practice, ending in a prioritised set of findings with severity, effort, and business consequence attached to each one.

Three words there carry weight. Bounded: a defined scope and end date, usually one to three weeks, not an open engagement. Read-only: the auditor changes nothing, so the assessment cannot itself cause an incident. Prioritised: the output ranks findings, which is what separates an audit from a scanner report. If a proposal lacks all three, you are buying something else.

The domains a real audit covers

Domain	Evidence examined
Cost	Spend by service and tag, Savings Plan and reserved instance coverage, unattached volumes, idle load balancers, cross-AZ and egress transfer, non-production running outside working hours
Security posture	Ingress rules open to the internet, encryption at rest and in transit, where secrets live and who can read them, image and dependency scan results, patch levels, findings sitting unread in the provider's own security services
Reliability and single points of failure	Availability zone spread, replica counts, database failover config, health check and timeout settings, shared dependencies every service touches, six months of incident tickets
CI/CD maturity	Pipeline definitions, time from merge to production, pipeline failure and rerun rate, approval gates, whether a rollback path exists and has been used, build reproducibility
IaC coverage and drift	Share of live resources actually managed by Terraform or CloudFormation, refresh-only plan results, console changes visible in audit logs, module and provider version pinning
Observability	Metric, log, and trace coverage per service, retention settings and their cost, the alert inventory, ratio of alerts fired to incidents declared, which dashboards were opened during the last outage
Access control	IAM policies with wildcard actions or resources, long-lived static keys, separation of human and machine identities, break-glass procedure, evidence that leavers were actually removed
Backup and DR	Schedules and retention, date of the last successful restore test, stated RTO and RPO versus what the setup can achieve, cross-region and cross-account copies
Documentation and bus factor	Runbook coverage for the top ten alerts, realistic onboarding time for a new engineer, count of systems only one person can deploy or debug

Cost and security are the domains buyers ask for. They are also the ones scanners handle best, which makes them the least valuable part of a human audit. An experienced reviewer earns the fee on reliability, CI/CD, drift, and bus factor, because all four require reading your actual code and talking to your actual engineers.

Reliability work is a hunt for single points of failure nobody has written down: the one NAT gateway, the single-writer database with a replica nobody has ever promoted, the shared Redis that four services treat as optional and one treats as mandatory, the certificate renewed by hand each year by someone who left in March.

CI/CD maturity is measured, not described. How long from merge to production, honestly, including waiting for a human to click approve? What fraction of pipeline runs fail for reasons unrelated to the change? Can you roll back, and when did you last do it? A team with a fast, boring pipeline can fix almost anything else. A team without one keeps regenerating the same problems.

Drift is the gap between your Terraform and your reality. Most teams over-report their infrastructure-as-code coverage, because they count the resources in state and not the resources in the account.

How a good audit is actually run

Four inputs, in roughly this order.

Read-only access. A dedicated IAM role with a managed read-only policy, ideally with an external ID and an expiry date, plus read access to repositories and observability tooling. If a provider asks for admin, ask why. The usual reason is laziness on their side.

Interviews. Forty-five minutes each with the people who carry the pager, plus the engineering lead and, if cost is in scope, whoever owns the bill. Engineers tell you in ten minutes what a scanner cannot find in a week: which deploy everyone dreads, which alert gets muted, which service nobody wants to touch.

Tooling scans. Provider-native tools first, since you already pay for them, then whatever the auditor brings. Fast and largely automated, which is why it should not be the bulk of the fee.

Manual review. Reading Terraform modules, pipeline definitions, Kubernetes manifests, and incident write-ups. This is where the fee goes, and it cannot be shortened without hollowing out the result.

A clean sequence: access and scans in the first two or three days, interviews in the first week, manual review through the second, a draft walkthrough, then the written deliverable. The walkthrough matters. Findings written without a chance to say "we know, that is deliberate, here is why" produce documents that get ignored.

What it costs you in your team's time

The invoice is not the whole price. Budget six to ten hours of engineering time: an hour to provision access, three to five hours of interviews across two or three people, an hour or two of follow-up questions, and an hour for the walkthrough. A provider who needs zero time from your team is producing a scanner dump.

On the invoice side, the drivers are scope (how many domains), environment and account count, whether Kubernetes is involved, how much of the infrastructure is code you can read versus clicked-together resources, and depth (a survey versus a review that reads every module). Review work scales with surface area rather than headcount, so a multi-account setup with several clusters costs considerably more than one account with a handful of services.

Checks you can run before you pay anyone

Run these first. If they come back clean, an audit will find less than you hope. If they come back ugly, you have a scope.

# --- AWS: cost and encryption basics (all read-only) ---

# Unencrypted EBS volumes
aws ec2 describe-volumes --filters Name=encrypted,Values=false \
  --query 'Volumes[].{ID:VolumeId,Size:Size,AZ:AvailabilityZone,State:State}' \
  --output table

# Volumes detached from everything and still billed every month
aws ec2 describe-volumes --filters Name=status,Values=available \
  --query 'Volumes[].{ID:VolumeId,Size:Size,Created:CreateTime}' --output table

# Elastic IPs not associated with anything (charged hourly while idle)
aws ec2 describe-addresses \
  --query 'Addresses[?AssociationId==`null`].{IP:PublicIp,Alloc:AllocationId}' \
  --output table

# Security group rules open to the whole internet
aws ec2 describe-security-groups --query \
  'SecurityGroups[?IpPermissions[?IpRanges[?CidrIp==`0.0.0.0/0`]]].{ID:GroupId,Name:GroupName,VPC:VpcId}' \
  --output table

# Buckets with no public access block configured at all
for b in $(aws s3api list-buckets --query 'Buckets[].Name' --output text); do
  aws s3api get-public-access-block --bucket "$b" >/dev/null 2>&1 \
    || echo "NO PUBLIC ACCESS BLOCK: $b"
done

# Credential hygiene: key age, MFA, unused passwords, in one report
aws iam generate-credential-report >/dev/null
aws iam get-credential-report --query Content --output text \
  | base64 --decode | column -s, -t     # macOS: use `base64 -D`

The Kubernetes checks cover the reliability side. Missing resource limits cause noisy-neighbour incidents, and single-replica deployments survive precisely until the next node rotation.

# --- Kubernetes: reliability and hygiene (read-only) ---

# Containers with no memory limit set
kubectl get pods --all-namespaces -o json | jq -r '
  .items[] as $p | $p.spec.containers[]
  | select(.resources.limits.memory == null)
  | "\($p.metadata.namespace)/\($p.metadata.name)  container=\(.name)"'

# Deployments running exactly one replica
kubectl get deploy --all-namespaces --no-headers \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,REPLICAS:.spec.replicas' \
  | awk '$3 == 1 {print $1"/"$2}'

# Namespaces with workloads but no PodDisruptionBudget
comm -23 \
  <(kubectl get deploy -A --no-headers | awk '{print $1}' | sort -u) \
  <(kubectl get pdb -A --no-headers 2>/dev/null | awk '{print $1}' | sort -u)

# Images pinned to a moving tag
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{range .spec.containers[*]}{.image}{" "}{end}{"\n"}{end}' \
  | grep ':latest'

# Pods not required to run as a non-root user
kubectl get pods -A -o json | jq -r '
  .items[] | select((.spec.securityContext.runAsNonRoot // false) != true)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Then the drift check, which is usually the most uncomfortable of the three. A refresh-only plan compares state to reality without proposing or applying any change.

# Detect drift without changing anything.
# Exit code 0 = state matches reality, 2 = drift found, 1 = error.
terraform plan -refresh-only -detailed-exitcode

# How much of the account is actually managed here?
terraform state list | grep -c '^aws_instance\.'
aws ec2 describe-instances \
  --filters Name=instance-state-name,Values=running \
  --query 'length(Reservations[].Instances[])'

If the second number is much larger than the first, your infrastructure-as-code coverage is a story rather than a fact. Repeat for security groups, RDS instances, and IAM roles.

What the deliverable should look like

A finding is useful only when it carries four things: what is wrong, what happens if it stays wrong, how much work the fix is, and what to do first. A usable format is one page per finding, with a severity, an effort estimate in engineer-days, the business consequence in plain language, and the specific resource or file it applies to.

Ahead of the findings, expect a short executive summary an engineering leader can forward without editing, and a sequenced remediation plan: this week, this quarter, this year, with dependencies noted (you cannot enforce pipeline policy before the pipeline exists).

Do not accept a PDF export of a scanner, five hundred unranked medium-severity rows, or recommendations that all point toward the auditor's own retainer.

What audits commonly find

These are patterns, not measurements, and every environment differs. Commonly:

• Infrastructure-as-code coverage below what the team believed, with the gap concentrated in resources created during incidents.
• Non-production environments sized like production and running around the clock.
• Backups configured and never restored, so the recovery time objective is a guess.
• One person who is the only one who can deploy a particular service.
• An alert set nobody trusts, the noisiest ones routed to a muted channel.
• Long-lived access keys belonging to former employees or to a service replaced two years ago.
• A pipeline that deploys but cannot roll back, so every bad release becomes a forward-fix under pressure.
• Log and metric retention left at the default and never revisited, quietly becoming a top line on the bill.

None of these are exotic. They accumulate because they are individually small and collectively nobody's job.

Telling a genuine audit from a sales exercise

Signals it is real: read-only access with a defined expiry, a written scope naming the domains covered, interviews with your engineers, a draft walkthrough before the final document, findings that include things to stop doing as well as start, and several recommendations you can implement yourself with no outside help.

Signals it is a sales exercise: a fixed-length report promised before anyone has looked at your environment, no engineer interviews, generic best-practice findings that never name your own resources, severity ratings that are all high, and a remediation plan whose every step requires the auditor.

The strongest test: ask what the audit would say if your environment turned out to be in good shape. A provider who cannot describe that outcome is not selling an assessment.

When you do not need one

If your team can already name the top five problems, agrees on their order, and knows how to fix them, an audit will mostly confirm the list. That is a capacity problem, and the money is better spent on the fixing than on the finding. Buying an audit here is often really about getting an outside voice to repeat what your engineers have been saying. That is sometimes a legitimate need, but call it what it is and buy less of it.

Skip it too if you are pre-launch with a couple of services and one environment, if you are mid-migration and the target architecture is already decided, or if you audited within the last year and have shipped nothing structural since.

The case is strongest when you inherited the environment, when the people who built it have gone, when the spend moved and nobody can explain why, when a compliance review or due diligence is coming, or when incidents are rising and no one can say which risk to spend the next quarter on.

If you want an outside read on any of that, we run scoped, read-only infrastructure reviews and hand back prioritised findings you can act on with or without us. A short call is enough to work out whether an audit is the right purchase for where you are, or whether you should skip straight to the fixing.