The energy cost of flaky CI
- 20 Jun 2025 |
- 04 Mins read
The useful version of “The energy cost of flaky CI” is narrower than the slogan: Sustainable pace for people and sustainable resource use for systems are the same habit: refuse unnecessary churn.
What follows stays on that claim: mechanisms, concrete scenes, and the failure modes that appear when the claim is ignored.
On The energy cost of flaky CI, refuse waste that buys no safety — retry storms, flaky CI, always-on idle fleets — and assign an owner who can change sizing or schedules. Unowned green metrics are decoration.
Flaky CI burns minutes and attention. Quarantine with deadlines; infinite noise trains people to ignore real failures.
What fails when you skip it
Teams already have scars. Naming the failure modes precisely is faster than inventing a framework brand.
Retry storms celebrated as resilience while they burn budget and hide bugs.
Dashboards of carbon without owners who can change schedules or sizing.
Always-on environments for demos that could be ephemeral.
Greenwashing metrics that never change an architecture decision.
Controls that answer those failures
Architecture forums that never change a sizing or schedule decision are performing green theater. Require a reversible experiment with an owner.
Retry amplification should have a budget and an alert. Unbounded retries are a distributed denial of your own reliability.
Right-sizing after a traffic shape change is a leadership conversation because finance, platform, and product all feel the “what if” differently. Bring numbers.
Measure compute waste in product teams: idle envs, redundant builds, chatty meshes with no owner. Numbers create permission to delete.
Measuring compute waste without an owner who can delete or resize is dashboards as decoration. Pair every metric with a decision rights note.
Efficiency as reliability practice shows up when you delete a redundant pipeline stage and incident escape rate does not rise. That is the experiment worth running.
Write one page while the decision is still warm — context, options, choice, owner, revisit date. Verbal alignment on The energy cost of flaky CI evaporates under ordinary calendar pressure, and Slack archaeology is a poor substitute for a decision record.
A concrete scene
A team celebrates resilience because the client retries aggressively. Production is stable-ish and the bill is not. They built a retry storm and called it fortitude. Cap retries and fix the root timeout.
Flaky tests burn CI minutes nightly. Engineers ignore failures. Carbon and attention both leak. Quarantine with a deadline beats living with noise.
Architecture review asks about carbon once, records a slogan, changes no sizing decision. Next quarter the same waste remains. Sustainability reviews need owners who can change schedules or footprint.
Tradeoffs
Right-sizing can feel like risk. Mitigate with scale-up paths and load tests, not eternal overprovision.
Carbon-aware batch shifts can delay results. Make the delay explicit and reversible.
Deleting pipeline stages requires courage and good eval of what still protects users.
Efficiency is reliability practice. If a carbon or cost signal never changes an architecture decision, the forum is performing virtue.
Measure what you claim to care about. If The energy cost of flaky CI only appears in kickoff slides and never in review, incident, or planning artifacts, it is branding. Put a verification signal where people already look.
What to do this month
- Track flaky test quarantine time as a reliability SLO.
- Add one sustainability question to every architecture review with a named follow-up owner.
- Ephemeralize non-prod by default; justify always-on in writing.
- Budget retries: cap and alert when amplification exceeds a threshold.
Carbon-aware scheduling without slogans means shifting batch work when it is safe and measuring whether load actually moved. Unmeasured virtue is marketing.
Design for fewer retries, not prettier dashboards. Retries multiply compute and hide root causes. Cap amplification and alert when it exceeds a threshold.
Sustainable pace and sustainable systems share a refusal: do not buy speed with hidden churn — flaky tests, retry storms, or always-on idle fleets.
CI minutes are both money and attention. Quarantine flakes with deadlines; infinite quarantine is how noise becomes culture.
Put sustainability questions in architecture forums: what did we choose that forces waste forever? If the forum cannot change a decision, it is theater.
Ephemeralize non-prod by default. Always-on demo and staging fleets are often habit, not requirement.
The energy cost of flaky CI includes engineer attention. Fix flakes as reliability work with an SLO on quarantine time.
Right-size infrastructure as leadership work: challenge defaults that force always-on waste for rarely-used paths. “Just in case” environments need a written justification.
Carbon-aware batch jobs shift to night without checking downstream SLA commitments. Green intent creates customer pain. Constraints must be negotiated, not silently applied.
A team discovers staging mirrors production capacity while serving a handful of QA users. Right-sizing frees budget and reduces the blast radius of bad deploys.
Staging runs at production scale “for realism” while load profiles show it is idle 20 hours a day. Right-sizing is leadership because nobody else can say no cleanly.
One more practice worth installing: Report idle environment hours next to cloud spend in engineering reviews.
Optimism without an owner is how demos become liabilities. Name the human before you name the tool.
Close
Refuse waste that buys no safety. That is both green and kind to the on-call.
Carry the claim as a habit, not a brand: The energy cost of flaky CI either compounds ownership or it was applause.