The deploy worked. Going back is the part nobody has tried.
Nothing here is on fire, and this page is not going to pretend otherwise. Your releases go out and your stack stays up. But the rollback has never been run against a system that had real data in it, and that is the one version of it that counts. So I run one, inside a window you agree, with the clock going.
Almost everything on this subject is already free, including mine.
Two kinds of reader arrive here and both are owed the same honesty, so this section is longer than it is on any other page in this section.
If you came from my own checker, it was right and it was complete. The Docker Compose production checker finds all eleven of the file-based findings this page is about, names the line, gives you the line number, and lists what passed as well as what failed. It needs no account and no email address. None of it is discovered here and none of it is held back.
And if you got here from an article and have run nothing, the free options are still the right first move. They are named below rather than skated past, because a page that hid them would be hoping you did not look.
- My own engagement builder prints this page’s scope, free, with hours on it
- The engagement builder lists the deployment pipeline work — build and deploy automated from the same commit, a rollback that is one command and has been used once before you need it, a runbook, a second person walked through a real deploy — with an hour band attached and an ungated PDF. That is this page’s subject matter, on this site, for nothing. It does the same for the staging environment.
- A language model writes the pipeline and the rollback script in seconds
- A correct workflow file, a compose file with healthchecks and limits, a rollback wrapper, and a clear explanation of all eleven findings. Most of it will be right. I am not going to sell you a document you can generate while reading this sentence.
- DORA already named the number that matters, and the tooling is open source
- DORA defines Failed Deployment Recovery Time as “the time it takes to recover from a deployment that fails and requires immediate intervention” — one of the five metrics it now publishes. Google’s Four Keys implementation is Apache-2.0 and free, though it has been archived and read-only since January 2024. It also computes that number from your deployment and incident history, which is the one thing a team with manual deploys and no rollback does not have.
- And the linters read the same files I would
-
docker compose config, hadolint, Trivy and Docker Scout all read your files for nothing and some of them do it better than I would by hand. Run them.
So the honest part, before anything else. If the checker came back clean, or you have rolled a real release back this year and know how long it took, close this tab. Fix the lines the checker named and get on with your week.
The question none of them answers is the same one: when you decide to go back, how long is it until the old version is serving correctly — and what does not go back with it?
Everything above either reads a file or counts deploys that already happened.
The fault that costs you the morning is in neither. It is in the going back, and three things put it out of reach of anything that describes rather than does.
Three reasons the number does not exist yet
- It has to be caused. The state a rollback must undo does not exist until a release has been running. Nothing can read it off a file, because it is not in the file.
- The forward deploy is what creates the state the rollback has to survive. This is the hard one. A fresh staging copy has no rows written by the new schema, no cache populated in the new format, and no queue holding messages the old version cannot parse. Roll back there and it works every time — which is exactly why it works every time until it matters. The only rehearsal that proves anything starts from a system the new version has already been running on.
- The number is not the one your pipeline reports. A pipeline reports how long the redeploy took. The number you actually experience runs from deciding to go back to the old version serving correctly, and it contains the migration that does not reverse, the asset a CDN is still handing out, and the twenty minutes working out who has the password.
That second point is a property of how deployments work rather than a complaint about anyone's tooling. No vendor can ship a feature that makes it untrue, which is why this page rests on it.
The vendors say it about their own products, so I do not have to.
The button says roll back. Read what the people who built it wrote underneath.
-
AWS — CodeDeploy user guide
CodeDeploy rolls back deployments by redeploying a previously deployed revision of an application as a new deployment. These rolled-back deployments are technically new deployments, with new deployment IDs, rather than restored versions of a previous deployment.
A rollback ships the old code forward over whatever the failed release left behind. It does not restore a previous state. Source -
AWS — CodeDeploy user guide
As part of this process, CodeDeploy will not try to revert or otherwise reconcile any actions taken by any scripts in previous deployments during subsequent redeployments, whether manual or automatic rollbacks.
Anything your release scripts did stays done. If a script wrote something, going back does not unwrite it. -
AWS — CodeDeploy user guide
the files you expected to be retained might be removed by the failed deployment. The files are not on the instance when the rollback deployment runs.
Quoted with its condition attached, because it matters: AWS is describing what happens if the deployment that failed was configured to overwrite rather than retain files. Narrower than it first reads, and still the case a lot of people are in. -
Redgate — Flyway undo migrations
Undo migrations assume the whole migration succeeded and should now be undone. This does not help with failed versioned migrations on databases without DDL transactions.
The sharpest sentence any vendor has written on this: the undo feature does not help with the case you actually needed it for. Redgate also puts undo behind a paid edition, which is a statement about how they tier it rather than what it costs. Source -
Redgate — Rollbacks, Undos and Undon’ts
If a database that is being 'undone' contains data that you need to keep, there is a danger that the previous version can't accommodate the data.
Which is the whole problem in one line, from the company that sells the tool for it. Source
These are quoted narrowly and deliberately. None of them says rollbacks do not work, and this page does not say it either. They say a rollback moves code and leaves state where it is — which is precisely why the interesting number is measured rather than assumed.
One record of something that happened, and one column no file can fill in.
This is what the review produces. The figures below are an example of the shape rather than anybody’s real system — but the columns are the real columns, and none of them can be filled in by reading anything.
The rollback record
The drill deploys the release immediately before the one you are running — a version already proven in production — onto a system that has days of the current release’s data in it. That is mechanically a rollback, and it is the only kind that proves anything. Afterwards the system returns to the release it started on.
| Segment | Clock | What happened |
|---|---|---|
| Decision | 00:00 | The named contact on the client side gave the go. The clock starts here, not at the deploy — the minutes spent deciding are minutes the old version was not serving. |
| Redeploy | 00:00 → 00:04 | The predecessor image pulled, started, containers reporting healthy. This is the only number a pipeline reports. |
| Database | 00:04 → 00:31 | The newer release had added a NOT NULL column with a default. The older one does not write it, so every insert failed. Twenty-seven minutes to find that out and write the backfill. |
| Cache | 00:31 → 00:38 | The session payload format had changed between the two releases. Flushed. Every signed-in user was signed out. |
| CDN | 00:38 → 00:52 | The asset manifest was still handing out the newer build’s hashed filenames, which the old code does not produce. Purged, then waited for the edge to catch up. |
| People | within the above | Twenty of those minutes went on finding who held the database password. It is in the record because it was in the window, and a record that quietly dropped it would be selling a tidier morning than anyone gets. |
| Old version serving correctly | 00:52 | Checkout confirmed working against the older release, by the client, before the clock was stopped. |
Four minutes of that was the deploy. The other forty-eight were state. Every segment below the redeploy is something your pipeline has no opinion about, and it is why a green rollback in a dashboard and a recovered system are two different claims.
And what did not go back
| What stayed | Why | What it took |
|---|---|---|
| 1,840 rows written by the newer release | A rollback ships the old code forward. It does not un-write data. | Backfilled by hand, one column, after the window closed |
| A config file the release had overwritten | It was not on the instance when the rollback ran. | Restored from the copy taken before the window opened |
| 312 queue messages consumed in the new format | Already acknowledged. The older consumer cannot parse them. | Replayed from the dead-letter queue |
| Every signed-in session | The cache flush was the fix, and the fix had this cost. | Nothing. People signed in again |
The abort criterion, written down before the window opened: stop and return to the current release immediately if any real customer’s checkout fails, or if the clock passes sixty minutes. It was not hit — the window closed at fifty-two, with eight minutes of margin. The margin is printed rather than rounded away, because a record that only ever shows comfortable numbers is not a record.
The eleven findings, with the column the checker cannot produce
The free checker already gives you the finding, the line and the fix, and this review does not resell them. What it cannot know is what changing each one costs on your system on the day — which of them is a one-line diff and which is a window with the database down. Three rows of eleven, to show the shape.
| Finding | Severity | The fix — already free | What changing it costs on the day |
|---|---|---|---|
| compose.db_no_volume | High | Add a named volume mapped onto the data directory | The recreate that attaches the volume is the same recreate that deletes the data. The data comes out first and the database is down while it does. The free checker says so itself — applying the fix “is itself a recreate”. This is a window, not a diff. |
| compose.image_latest | Warn | Pin the tag you are actually running | One line in the file — once you know what is running, which latest does not say. Read the digest off the running container with docker inspect first. If the tag has moved since the last deploy, pinning today’s latest pins a version nobody here has ever run. |
| compose.no_mem_limit | Warn | Set a memory limit on the service | Pick the number wrong and the container is killed under exactly the load it was fine at yesterday. It wants a fortnight of observed peak usage, or a generous limit set now and tightened later. Two visits, not one. |
Note that the three costs have nothing in common. One is a risk in the act of fixing it, one is a piece of finding-out before you can fix it at all, and one needs a fortnight of watching before the number can be chosen honestly. That is the difference between a list of fixes and a plan.
What is free here, and what is not
Once, in one place, rather than a locked row next to every finding.
Free, always, with no account and no email address
- All eleven findings from the compose checker — by id, by name, with the line, the line number and the evidence. Uncapped.
- The checks that passed, so the report reads as a report rather than a sales pitch.
- The fix for every one of them, and the explanation of why it matters.
- The whole deployment plan from the engagement builder, its deliverable lists, its hour bands and its PDF.
- Everything on this page, including the record above.
What you would be paying for
- The drill itself — a rollback performed against your system, in a window, with the clock running.
- The record it produces: your segments, your numbers, your list of what did not go back.
- The fourth column written against your actual configuration rather than the general case.
- Re-runs, so a rollback proven in August is still proven in November.
That list is the same whether your file comes back clean or covered in findings. A compose file with nothing wrong in it still has a rollback nobody has timed, and a page that only offered you something when it found a problem would be selling alarm rather than work.
This interrupts a running application, and I want the limits in writing.
A rollback drill cannot use a canary. The whole reason it is worth doing is that the state which breaks it is written by real use, and something carrying no traffic writes none. So the boundary is in the engagement rather than in my good intentions.
The drill deploys your own immediate predecessor release. Nothing bad is deliberately shipped. That version already ran in production, so the mechanical path is identical to a real rollback and the risk is not.
- An agreed window, in writing, with a named person on your side reachable throughout it. No drill starts without one.
- The abort criterion is written before the window opens, not judged during it, and the drill stops on it without a discussion.
- No schema change is introduced by the drill. If the release between your two versions contains one, that is something we find before the window rather than during it, and repairing it is tier 3 work.
- Nothing is deleted, and the previous state is recorded verbatim before anything moves.
- A backup taken by you, verified by you, before the window. This page does not sell the backup and does not vouch for it — that is the resilience review.
You can also just drive it yourself. You run the deploy and the rollback, I sit on the call with the clock and the record. Same deliverable, slightly longer, and no credential ever moves. If you would rather not hand anyone a deploy button, that is a reasonable position and it does not put this out of reach.
Deliberately deploying a release known to be broken is something a client can ask for. It is never the default, and never on a first engagement.
Seven things go wrong, and only one of them loses data.
Four are found by reading your compose file, free, in about ten seconds. Three only exist as your own answer to a question, because nothing external can observe them. The finding ids are printed so a report and this page can be matched up.
- compose.db_no_volume, compose.bind_relative, compose.bind_over_image_path State is written where a container recreate destroys it
- The only High in this list, and the only one that loses something you cannot get back. With no named volume a database writes into the container’s writable layer, which is deleted by
docker compose down, by anything that recreates the container to change one environment variable, and by a routine image update. The free checker puts it better than I will: the data is there right up until the first time somebody does something completely normal. Relative bind mounts belong with it, because they resolve against whatever directory the command was run from. - compose.no_restart, compose.no_healthcheck, compose.depends_no_condition Nothing brings the stack back, and nothing knows whether it is working
- No restart policy means a container that exits stays exited, including after the host reboots at four in the morning for a kernel update. No healthcheck means nothing can tell the difference between a process that is running and one that is serving. And
depends_onwithout a condition waits for a container to exist, not to be ready — which is why the application comes up before the database and works fine every time you test it by hand. - compose.no_mem_limit, compose.log_unbounded, compose.no_cpu_limit One container can take the whole host with it
- An unbounded container competing for memory gets something killed, and the kernel picks. An uncapped log driver fills the disk, at which point everything on the box stops at once and the thing you need to read to find out why is the thing that filled it. The CPU limit sits a tier below the other two on purpose and the checker says why: the kernel keeps scheduling everything, so nothing gets killed and nothing stops. Slower is a genuinely different outcome from dead.
- compose.image_latest, compose.image_untagged No version is named anywhere, so there is nothing to go back to
latestis a moving tag and an image with no tag at all silently means the same thing with less visibility. Two machines running the same file can be running different software, and a container recreated for an unrelated reason can come back on a new major version of the database. It also removes the ability to roll back, because there is no previous version named anywhere — which is where this fault touches the next one from the opposite direction.- Self-reported — the engagement builder Deploys are manual, there is no rollback, and one person owns the process
- These three travel together and share a consequence: releases get rarer, which makes each one bigger, which makes each one riskier. It arrives here as your own answer rather than as something measured, and it is the fault that makes every other one on this list slower to fix. It is not merged with the one above it, because pinning a tag is a line in a file and building a deploy path is a piece of work, and one repair is what makes one fault.
- Self-reported — the engagement builder There is nowhere to rehearse a change, so the deploy is never rehearsed
- Staging is worth less than people claim and considerably more than nothing. Its real value is not that the code gets tested — it is that the deploy itself gets rehearsed, which is the part that usually breaks. That only holds if the path to staging is the same path as the path to production. A staging environment deployed by hand tests nothing about the deploy.
- Self-reported — the downtime cost calculator Cleanup costs more than the outage did
- If you spend longer putting things right than you spent down, the missing piece is almost always a rollback nobody has tested or a restore path nobody has rehearsed. This is the same repair as the fifth fault, seen from the other end — that one records the absence, this one puts a number on it. Both are here because one is the reason to care and the other is the thing to buy.
There is no critical finding in that list and I am not going to invent one. One High, nine Warnings and one note, which is an accurate description of a stack that works until something ordinary happens to it. That is the whole character of this page: not a fire, just a set of assumptions nobody has checked.
Four ways in
Everyone starts with the drill, because until a rollback has been timed nobody knows which of the eleven findings actually matters — including me.
Deployment Review
Everyone starts here. It also scopes everything else.
- One rollback drill in an agreed window, with the clock running, on the boundary above
- The rollback record — what went back, how long each segment took, and what did not go back at all
- Your compose file read against all eleven findings, with the fourth column: what changing each one costs on your system
- What is already correct, not just what is not
- Nothing is changed. The system ends the window on the release it started on
- Or you drive every step and I measure — same record, no access needed
Review + Repair
Not a starting point. Follows the review.
The tier where the rollback becomes one command that has been used.
- Images pinned to the versions actually running, so a rollback target exists at all
- Named volumes for state, with the data moved out and back rather than trusted to survive a recreate
- Restart policies, healthchecks and
condition: service_healthyso the stack comes back on its own and waits for what it needs - Memory limits and a capped log driver
- A second drill after the repair, so the fix is proved rather than asserted. Included, not quoted
Review fee credited — you pay the difference, not both.
Deployment Baseline
Not a starting point. Follows the review.
The tier where the rollback survives a release that changed the database.
- Everything in the repair
- The migration handled so the rollback survives it — the expand-and-contract work, which is the answer to a previous version that cannot accommodate the data
- Staging that shares the deploy path, so testing the deploy tests the deploy rather than testing the code
- A second person walked through a real deploy, so it is not one person’s knowledge
- Handover, so your team can run the drill without me
Review fee credited — you pay the difference, not both.
Release Drills
Because a rollback proven in August proves less in November.
- A rollback drill each month against the release currently live, timed the same way
- The result diffed against the last run, so a number getting worse is visible before it matters
- New stateful components checked into the drill as they appear
- A one-page record each month you can hand to anyone who asks
- Monthly in advance, cancel anytime, never auto-renewing
Five things about the packages, before you ask
- The cap is one application, one deploy path and one agreed window. The drill is the thing being bought and that is the unit it runs on. An application here means the set of things that have to go back together — if your web tier and your worker fleet are deployed separately and could be rolled back independently, that is two deploy paths and I will say so before quoting rather than after.
- Tier 3 is the one marked, and it is not the one most people buy. That is what the label says: most complete, not most popular. Tier 2 makes your rollback work for a release that did not change the database. Tier 3 is the tier where it survives one that did — which every vendor quoted above agrees is the hard case. Marking tier 2 would be marking the easy half.
- The second drill in the repair tier is included, not quoted. Charging extra to prove my own fix worked would be the exact failure this page is written against.
- The review fee is credited for 30 days. Come back inside a month and you pay the difference. After that it lapses, and not as a sales tactic: releases happen, and a drill from last quarter would have to be run again anyway.
- What moves the size of the job is state, not service count. In order: an irreversible database migration in the release; a stateful component with data written by the new version; a cache or CDN in front that keeps serving the new version’s assets after the code has gone back; no previous version named anywhere, so there is no rollback target until one is created; a queue holding messages in the new format; and how many people have to be awake for the window.
Release Drills is not monitoring and it is not alerting. Monitoring watches that something you want up is up, and if that is what you need it is hosted monitoring and a different service. This rehearses the going back — there is nothing to watch, because the thing being tested only exists while somebody is doing it.
There are no figures on this page, and that is deliberate rather than coy. A fixed number would be too high for one stateless application with a pinned tag and too low for a release carrying a schema change with a CDN in front of it. Tell me the shape of it in the form — what is live, what came before it, and whether there is a migration between them — and you get a figure back before anything is committed to.
What this cannot promise
Stated here rather than discovered later.
Five things I will not claim
- A drill measures the path that existed on the day. A rollback proven in August proves less in November, because the release process moved underneath it. That is what the monthly tier is for, and it is why tier 1 is not sold as permanent.
- A rollback does not undo. AWS documents that about its own product and it is true of every deployment system: rows written by the new version stay written unless somebody wrote the code to remove them.
- A rehearsal on a clean environment proves the mechanism, not the outcome. If you already rehearse on fresh staging, that is genuinely worth having and it is not the same thing. This page exists because the difference between those two is where the morning goes.
- The drill can abort, and an aborted drill is a finding. It has a criterion, the criterion is honoured, and the result is reported as what it is. It is not re-run free by default, because the window was used and the answer was produced. A drill that could not run at all is a different thing and is not billed as a result.
- Nothing here is a security service. The eleven findings behind this page are precisely the ones that are not security findings — the checker routes those elsewhere. If you are worried about a published database port, that is the container security review and I would rather send you there than stretch this.
What you have to supply
Short list, and the last two are the ones people forget.
Before the window opens
- The compose file, or the deploy scripts, for the read.
- A window, agreed in writing, with a named person reachable throughout it.
- The two release identifiers: what is live now, and the one immediately before it.
- Whether the release between them contains a schema change. Asked before the window rather than discovered in it, because that answer changes what the drill is.
- Access to whatever performs the deploy — or a person driving it while I measure.
- The abort criterion, agreed and written down before the window opens.
- A backup taken before the window, by you, verified by you.
Six things this is not
Each one is real work and some of it is work I do. None of it is this page.
Quoted separately, or somebody else’s
- Building a CI/CD pipeline from nothing. The free engagement builder prints that scope with hours on it. This page reviews and proves; building is quoted separately.
- Moving you to Kubernetes, or off compose. Different job, different page, and not one to bundle into a drill.
- Container security — the container security review, and the split between the two is drawn in the checker’s own code rather than by me.
- Backups and restores — the resilience review times a restore, which is a different failure and a different clock.
- Monitoring — hosted monitoring if you want somebody watching the stack.
- Hardening the server itself — VPS hardening. Writing your application’s database migrations is also not this; reviewing whether they reverse, and designing the path so they do, is tier 3 and is in scope.
How payment works
Plainly, so nothing about it is a surprise later. There is no account to create, no portal to log into and no card stored anywhere.
-
You apply through the form
No payment at this point, and no commitment. Tell me what is live, what came before it, and whether there is a migration in between.
-
I review it and confirm the scope
If it is not worth doing, I say so here and it costs you nothing. If it is, you get the scope, the window and the abort criterion in writing before any invoice exists.
-
An invoice arrives, payable within 48 hours
It carries an invoice number and a due date. Unpaid past 48 hours, the application lapses and the slot is released to somebody else.
-
Work begins once payment is received, and once the window is agreed in writing
Not before, and the second half of that sentence is not a formality — the written window and the named contact are what make it safe to touch a running application. For one-off work there is nothing to suspend after delivery, so payment comes first. That is the standard arrangement and it runs both ways.
-
Release Drills, if you take it, is monthly in advance
Non-payment simply stops the work. Nothing is billed silently in the background and you can stop whenever you like.
If you are still building
Everything above assumes a system with data in it and customers on it. Before launch the same work is a fraction of the cost, and the drill is genuinely free of risk.
What that involves
- Named volumes and pinned tags decided once, at the point where changing them costs nothing. Retrofitting a volume under a database with real rows in it is a window with the database down. Choosing it now is a line in a file.
- The deploy path built so staging and production share it from the first day, which is the only way testing the deploy ever tests the deploy.
- A rollback run before there is anybody to notice. You find out what your recovery number actually is on a day when it does not matter, which is the cheapest that answer will ever be.
- The migration pattern chosen before the first one ships. Expand and contract costs almost nothing as a habit and a great deal as a retrofit.
Mention that you are pre-launch in the form and I will quote it as its own piece of work. It is cheaper, and it is the version of this engagement I would rather sell.
Run the free checker first. Then tell me what a rollback would have to survive.
It costs nothing, needs no signup and no email address, and it prints the finding id next to the line it found. If it comes back clean, or you have rolled a real release back this year and know how long it took from the decision, you have saved a fee and I would rather tell you that than take it. If nobody has ever timed it, that is exactly what this is for.
Prefer to talk? Book a free call ↗ · Or hire me on Upwork ↗ · Typical reply within one business day.
Questions
Your free checker already found all eleven of these. Why pay you?
Can a language model not just write me the pipeline and the rollback script?
We have staging. Have we not already tested this?
Are you going to deliberately break our production system?
Can we do it without giving you access to anything?
Our release has a database migration in it. Does that change things?
Is this a security review?
How is this different from restoring a backup?
What if the drill goes wrong?
Do you do Kubernetes?
revisionHistoryLimit “specifies the number of old ReplicaSets to retain to allow rollback” and that “by default, it is 10”, so even the target has a shelf life. Say so in the form and I will tell you honestly whether it is worth quoting.