I told a lie to a room full of people in 2017. The people were engineers. The people were managers. I sat in a chair. I looked at a table. I told the people that the data was safe. I told the people that the backups worked.

I had a document in my hand. The document was thick. The document had a blue cover. I believed the document. I had not tested the document. I had not run the script. I had only read the log. The log said the script finished. The log did not say the data was saved.

I felt a sense of pride. I felt like a man who had parallel parked perfectly on his first try. This feeling was a mistake.

The Forty-Three Hour Reality

The data was not safe. The data was gone. I found this out three days later. A server failed. The server was in Virginia. I opened the document. I followed Step 1. I followed Step 2. Step 3 asked for a key. I did not have the key. Nobody had the key.

The man did not answer his phone. The document was a list of wishes. The document was not a plan. I spent forty-three hours in a cold room. I tried to fix the mistake. I could not fix the mistake. We lost the data.

Institutions Love a Document

Institutions love a document. A document is cheap. A document is quiet. A document sits in a folder. The folder is on a drive. The drive is shared. People look at the document during an audit. The auditor sees a version number. The auditor sees a review date.

The auditor sees a name. The auditor checks a box. The auditor is happy. The manager is happy. A document does not change when the system changes. The system is a living thing. Engineers add code to the system. Engineers change the database. Engineers move files to a new bucket.

The document stays the same. The ink does not move. The pixels do not shift. A stale plan is more dangerous than no plan. If you have no plan, you are nervous. You look at the system. You check the cables. You verify the backups. If you have a stale plan, you go to sleep. You trust the paper. You should not trust the paper.

The Disconnect of Step 5

The audit happens next week. A worker opens the continuity document. The worker reads Step 3. Step 3 mentions a storage bucket. The storage bucket was migrated in March. The document does not know about March.

The worker reads Step 5. Step 5 mentions a person named Janice. Janice left the company in January. Janice took her passwords with her. Janice is at a beach now. Janice does not care about the document.

The worker reads Step 7. Step 7 says to restore the database from a snapshot. The worker looks at the dashboard. The dashboard is green. The worker thinks the snapshots are good. The snapshots are not good. They have been failing since the retention policy changed in June.

The Verification Loop

The alert does not trigger because the bucket is empty. The dashboard remains green because the alert is silent. This is a chain of failures. Each link in the chain is a human choice. Each choice was made to save time. Each choice was made to reduce anxiety.

A recovery plan is a script for a play. A play needs a rehearsal. You must stand on the stage. You must say the lines. You must move the props. If you do not rehearse, the play will fail. The audience will leave. In software, the audience is the customer. The customer does not like a dark screen.

If the numbers match, the data is real. If the numbers do not match, the data is a ghost. You run this script every hour. You do not wait for the audit. You do not trust the document. You trust the checksum.

Moving Beyond Deliverables

Many companies hire a team to build a product. The team builds the product. The team writes a document. The team gives the document to the company. The team leaves. This is a deliverable. A deliverable is a dead thing. It is a snapshot of a moment.

The better way is sustained capacity. You need a team that owns the roadmap. You need a team that stays with the code. You need a team that runs the recovery plan once a month. This is what

Digital Heroes

provides for its clients.

The tech lead is a real person. The tech lead attends the calls. The tech lead writes the status report. There is no manager between the engineer and the problem. When the system changes, the tech lead changes the plan. The plan stays alive. The plan is a tool, not a totem.

The Interest on Hidden Debt

I see this mistake in many places. I see it in startups. I see it in large banks. A founder has a deadline. The founder has investors. The founder needs a feature. The founder does not want to pay for a rehearsal.

The founder wants to pay for the feature. This is a debt. The debt has interest. The interest is paid in a crisis. The crisis happens on a Saturday. It happens at 3:14 AM.

The founder wakes up. The founder opens the document. The document is useless. The founder realizes that the document was a form of insurance that does not pay out.

Killing the Document

A document is a substitute for action. It is the cheapest substitute. Institutions like cheap things. They like things that survive an inspection. An inspector looks for a file. An inspector does not look for a restored database. This creates a gap.

The gap is where the disaster lives. To close the gap, you must kill the document. You must replace the document with a process. A process is a habit. A habit is something you do because you are afraid of the alternative.

“I stopped lying in 2018. I started testing. I deleted a production database on purpose. I did this on a Tuesday. I did this at 10:00 AM.”

– A Post-Recovery Narrative

My team was watching. They were nervous. I was nervous. We followed the plan. Step 3 failed. We fixed Step 3. Step 5 was slow. We made Step 5 fast. Step 7 worked perfectly. We wrote down what we learned. We updated the plan.

The plan was now a true story. It was not a lie. We felt a different kind of calm. This calm was earned. It was the calm of a person who knows where the fire exit is because they have walked through it.

State Management

We must look at the tools we use. We use cloud providers. We use automated deployments. We use monitoring tools. These tools are complex. Complexity hides errors. A green light on a screen can mean many things. It can mean the system is working. It can mean the monitor is broken.

It can mean the system is so broken that it cannot even report an error. You only know the truth when you pull the plug. Pulling the plug is scary. It is also the only way to be sure.

Software engineering is not about writing code. It is about managing state. The most important state is the state of the system when everything is on fire. If you do not know that state, you do not know your system. You only know a dream. Dreams do not serve traffic. Dreams do not process payments.

A Clean Handover

The engagement must end with a handover. The handover must be clean. You need the code. You need the infrastructure. You need the documentation. But more than that, you need to know that these things work together.

You need a team that embeds security and quality into every sprint. You do not add them at the end. If you add them at the end, they are just more pages in a document. They are more boxes for the auditor to check. They are more lies to tell in a room full of people.

I look back at 2017 often. I remember the blue cover of the document. I remember the font. I remember the signature on the last page. None of those things helped me. The document was a ghost. It haunted the server room.

It told me everything was fine while the data leaked out of the hole in the floor. I carry the memory of the forty-three hours I spent in that cold room. That memory is the best recovery plan I have ever owned.