Signs Your Engineering Team Needs DevOps Help

Signs Your Engineering Team Needs DevOps Help

Most teams do not decide one morning that they have a DevOps problem. It is quieter. Releases slip by a day, then a week. One person gets pulled into every infrastructure question. An incident gets fixed, and three months later something suspiciously similar takes the site down again.

Each of those looks like a local problem with a local cause. Together they are usually one thing: nobody owns the path from a laptop to production, so that path decays a little every sprint.

Below are seven signs, each with a diagnostic to confirm it, why it worsens if ignored, and a first fix you can do this week without buying anything. Where useful I refer to DORA's four key metrics (deployment frequency, lead time for changes, change failure rate, time to restore service), which give shared language for problems otherwise argued about on vibes.

Sign 1: Deploys are batched, scheduled, and slightly scary

You recognise this when releases happen on a fixed day, someone is nominated release captain, and nobody ships on Friday afternoon.

Measure it. If you tag releases, count them per month and count the commits riding along in each:

# Releases per month over the last year
git log --tags --simplify-by-decoration --date=format:'%Y-%m' \
  --pretty='%ad' --since='1 year ago' | sort | uniq -c

# How big was the last batch? (commits between the previous tag and HEAD)
git rev-list --count "$(git describe --tags --abbrev=0 HEAD^)"..HEAD

# No tags? Merges to main are a decent proxy for deploys
git log --merges --first-parent main --since='6 months ago' \
  --date=format:'%Y-%m' --pretty='%ad' | sort | uniq -c

This compounds because batch size and fear feed each other. Big releases fail more often, failures make people release less often, and the next batch is bigger still. You also lose bisection: with forty changes in one release, finding the guilty one takes an afternoon.

First fix: do not rebuild the pipeline, shrink one batch. Take the lowest risk service you own, deploy it on every merge to main, and put anything half-finished behind a flag. Then have someone who did not write the rollback procedure run it against staging while you time them. If it takes over five minutes or needs one specific person, fix that first. Cheap rollback is what makes frequent deploys safe.

Sign 2: One person is the only route to production

The tell is social, not technical. Infra questions in Slack end up with the same name, releases get planned around one person's holidays, and one laptop has the only working Terraform credentials.

Confirm it with history:

# Who has actually touched infrastructure code in the last year?
git log --since='1 year ago' --pretty='%an' -- \
  infra/ terraform/ helm/ .github/workflows/ | sort | uniq -c | sort -rn

If one name has ten times the commits of the next, that is your bus factor. Then ask in a retro: if this person vanished for two weeks, what could we not do? If the answers include deploying, rotating a secret, or restoring a backup, that is the finding.

This compounds unfairly. The person who knows everything is interrupted constantly, so they never get quiet time to write it down or automate it. Then they burn out or take a better offer, and it leaves with them.

First fix: take the three riskiest operations from that list. For each, someone other than the expert writes the runbook, and someone else again runs it against a non-production environment while the expert watches without touching the keyboard. Every place the runbook is wrong is knowledge that was about to walk out the door.

Sign 3: The same incident keeps coming back with a new name

You have a postmortem that references an earlier postmortem. Disk fills up again, the queue backs up again, and on-call has a folder of alerts they know how to silence but not how to fix.

Self-check: put six months of incidents in a table with four columns: trigger, contributing cause, agreed action items, and whether those shipped. The last column is the one that matters. Track change failure rate next to it, meaning the share of deploys needing a hotfix or rollback soon after.

This compounds because unfinished remediation is a promise the team stops believing. Postmortems become ritual, and the alert that has fired five times gets ignored a little faster each time until the real one is missed.

First fix: freeze new action items until the open ones are closed or explicitly dropped, and dropping is a legitimate recorded choice. Give remediation the same tracking, owner, and deadline as feature work rather than a wiki page. Then convert one recurring alert into a check that runs before deploy, so the failure becomes a build error instead of a page at 2am.

Sign 4: "Works on staging" stopped meaning anything

Staging passes and production breaks. Instance sizes differ, someone fixed something in the cloud console months ago and never wrote it into code, and production has environment variables that exist nowhere in the repo.

Drift has a direct read. Run a refresh-only plan against production with nothing pending, then compare workloads across environments:

# Any diff here is a change someone made outside of code
terraform plan -refresh-only -no-color | tee drift.txt

# Compare running images and replica counts between environments
for ns in staging production; do
  kubectl -n "$ns" get deploy -o \
    custom-columns='NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image,REPLICAS:.spec.replicas'
done

# Pods with at least one container running with no resource limits
kubectl get pods -A -o json | jq -r '
  .items[]
  | select(any(.spec.containers[]; .resources.limits == null))
  | "\(.metadata.namespace)/\(.metadata.name)"'

This compounds because trust is the product staging sells. Once it has lied a few times, people stop treating a green run as evidence and real testing quietly migrates to production, while you keep paying for the environment.

First fix: eliminate one difference completely. Usually that is the artifact: build the image once, promote that exact image through environments, and inject everything environment specific as configuration. Then run the refresh-only plan nightly and post the output to a channel, so drift surfaces in a day rather than during an incident.

Sign 5: The cloud bill grows faster than traffic

The bill rises every month and the explanation is always "we're growing". Finance asks which team spent what, and engineering estimates.

Self-check: divide monthly spend by a business number you track, such as requests served or active users, and plot the ratio over twelve months. Growth raises the total and keeps the ratio flat; waste makes it climb. Then look at ownership, because unowned resources are where waste hides:

# Resources with no tags at all are usually resources nobody owns
aws resourcegroupstaggingapi get-resources --region eu-west-1 \
  --query 'ResourceTagMappingList[?length(Tags)==`0`].ResourceARN' --output text

# Unattached EBS volumes still bill every hour
aws ec2 describe-volumes --filters Name=status,Values=available \
  --query 'Volumes[].{ID:VolumeId,GiB:Size,Created:CreateTime}' --output table

This compounds because unowned infrastructure never gets deleted. Nobody removes a resource they cannot attribute, in case something depends on it, so waste becomes a permanent floor under your costs until someone mandates a percentage cut under time pressure.

First fix: require an owner tag on everything created from now on and enforce it in CI for your infrastructure code, so the rule cannot rot. Clean up what those commands found, after checking with whoever created them. Set a budget alert at a threshold that would genuinely surprise you.

Sign 6: Onboarding an engineer to infrastructure takes weeks

New hires ship application code in their first few days, then wait a month before anyone lets them near deployment. The setup docs are out of date and the real instructions live in a Slack thread from last year.

Self-check: give your next new engineer a scripted exercise. From a fresh machine and written docs only, get the app running locally and deploy a trivial change to staging. Nobody helps unless they are fully stuck, and every blocker gets recorded. Do not judge the person: their list is a ranked backlog of everything broken in your setup path.

This compounds because it caps what hiring buys you: if each new engineer costs weeks of senior time, growing the team makes your seniors less productive before it makes them more productive.

First fix: take the top three blockers and collapse them into one command, whether that is a make target, a script, or a devcontainer. The goal is one documented command from clean checkout to running environment. Re-run the exercise with the next hire and see if the time drops.

Sign 7: Nobody can answer "what changed?" during an incident

The first twenty minutes of every incident go on asking who deployed what, whether the migration ran, and if anyone touched the load balancer. Nothing shows deploys, config changes, feature flag flips, infrastructure applies, and vendor status on one timeline.

Self-check: take your most recent incident and, using data alone with no memories and no asking colleagues, reconstruct every change in the twenty-four hours before it started. Time yourself. More than fifteen minutes, or an incomplete answer, means you have no change visibility, and that delay is added to time to restore service on every incident.

This compounds because it slows recovery exactly when speed matters. It also pushes teams toward change freezes, which feel safe and are not: the eventual unfreeze is a large batch, which takes you back to sign 1.

First fix: build one change feed. Every deploy emits an event with service, version, commit SHA, actor, and timestamp into one place everyone can see, such as a channel, dashboard annotations, or a small table. Add infrastructure applies and flag changes to the same feed.

The seven signs at a glance

Sign	What it actually costs	Severity	First fix
Batched, scary deploys	Slow lead time, high change failure rate	High	Shrink one batch, rehearse rollback
Single point of knowledge	Total exposure to one person leaving	Critical	Runbooks written and executed by other people
Repeating incidents	Senior time burnt twice, alert fatigue	High	Close open remediation before opening new items
Environment drift	Testing that proves nothing	Medium	Promote one immutable artifact, check drift nightly
Bill outgrowing traffic	Permanent cost floor, panic cuts later	Medium	Owner tags, delete idle resources, budget alerts
Slow infra onboarding	Hiring stops adding throughput	Medium	One command from checkout to running environment
No answer to "what changed?"	Longer time to restore, freeze reflex	High	One change feed for deploys, applies, and flags

Severity means how fast the problem stops being recoverable on your own.

Hire, partner, or fix the process?

If several of these are familiar, the question is what kind of help you need.

Fix the process yourself. Right call when the signs you recognised are about discipline rather than missing skills: unclosed remediation, no change feed, no rehearsed rollback. Those are days of work, not months. The tradeoff is opportunity cost, and the risk is a migration half done then abandoned when a deadline appears. Only do this if someone gets protected time.

Hire a platform or DevOps engineer. This fits when infrastructure work is continuous, already takes more than half of someone's week, and you can describe the role concretely enough to interview for it. Hiring takes months, seniors are expensive, and you need someone who can technically assess candidates. Watch for the trap in sign 2: one hire into a team with no other infrastructure knowledge recreates the single point of failure under a new name, and lone platform engineers with nobody to review their work tend not to stay long.

Bring in a fractional or outsourced partner. This fits when you need senior judgement across several areas at once (pipeline, Kubernetes, cost, observability) but not forty hours a week of it, or when the scope is bounded and you need it sooner than hiring could deliver. The tradeoffs are context and dependency: any outsider spends time learning your system, and if nothing is written down while they work, you rented a solution rather than acquired one. Guard against that in the contract. Infrastructure code lives in your repositories, runbooks are written for your engineers, and there is a named internal owner from day one.

There is a fourth answer nobody sells you: do nothing on purpose. If you are pre-product-market-fit and the system is two services and a database, slow manual deploys are a defensible tradeoff. Just make it a decision you revisit on a date rather than a thing that happened to you.

If you worked through the self-checks and the answer is a partner, that is what we do at InstaDevOps: senior DevOps on a monthly retainer, with everything we build living in your repositories. Either way, run the diagnostics first. They cost an afternoon and tell you more than any sales conversation.