Most of them do not break. They quietly stop being good, which is a much harder failure to notice, because the pipeline still runs on schedule and throws no errors and produces output on time. The output is just fine. Fine is the failure state nobody watches for, and a system that fails this way can run for a month while you tell people it is working.
The second reason is that it was automated before it was understood, and those two failures are related.
Fine is a failure state
A system that breaks loudly gets fixed the same day, because something visible stopped happening and someone complained.
The quiet version is different. Everything appears healthy. The dashboard is green because the dashboard was built to check that output existed, and output does exist. What nobody built was a check on whether the output was any good, and so nothing anywhere in the system has an opinion about that.
I found this in my own content pipeline by accident, opening one output while looking for something else and reading it properly for the first time in weeks. It was correct, on brand, and completely inert. Then I looked at the previous eleven. Same. Eleven pieces produced reliably, on time, by a system I had been describing to people as working.
The engagement data had been saying this the whole time. I had not been reading it, because the thing was automated, and automated had come to mean handled.
Automating before understanding
The other failure is upstream and it is the cause of the first one.
Five successful manual runs teach you the steps. They do not teach you the exceptions, and the exceptions are the entire content of the job. The week the input arrived in a different format. The time the source piece was late and everything downstream ran on something half-finished. The day a link was wrong and nothing anywhere noticed.
Four weeks of running it by hand surfaces those, and the log of exceptions is the real product of that month, not the work you produced during it. A system built without that log has no handling for the things that actually happen.
The distinction underneath all of this, between a tool that does a task and a system that runs a process, is worth reading separately in the difference between AI tools and AI systems and the difference between using AI and working with AI.
The Publishing Playbook
Sixth and last of the six, and it is last deliberately. Automating distribution before you have a context brain and a content system means automating the production of noise at speed.
The build.
- Write down the actual steps you take by hand, including the annoying ones and the judgment calls you make without noticing. This list is the specification and it is always longer than you expect.
- Run it by hand for four weeks, logging every exception.
- Automate only the mechanical steps. Anything that requires deciding whether something is good enough stays with you.
- Attach a quality check rather than an existence check. Something a person reads on a schedule. Weekly is usually enough.
- Write the failure question underneath it before it goes live.
The failure question
This is the line I now put under every system before it runs:
If this ran for a month and I never looked at it, what would be true, and how would I find out?
It has killed several automations before I built them, which is the point rather than a side effect. It forces you to name the failure that does not announce itself, and then to build the specific thing that would tell you.
The trap
Building this playbook first, because it feels like progress.
Distribution is visible in a way the other five are not. Things start moving, output appears on a schedule, and from the outside it looks like a system is working. What is actually happening is that a pipeline with nothing to say has been handed a megaphone.
The second trap is treating four weeks as excessive caution. It felt slow to me too, and it is the fastest thing I do now, because a system built on a month of real exceptions does not need rebuilding in the spring.
What this is really about
I went into my own version of this thinking I had made an engineering mistake, and came out understanding it was a management one.
I had delegated to a system the way you would delegate to someone whose work you had never reviewed, and then not reviewed it, and then been surprised nobody caught it. If a person on my team had produced eleven inert pieces in a row unnoticed, I would not conclude that the person was the problem. I would conclude that I never built a review step.
That is what delegating to AI is. Not coding. Management. And every management failure mode you already know applies, including the specific one where something quietly stops being good and everyone keeps saying it is fine.
Frequently asked
Why do AI automations stop working after a few weeks? Usually they do not break, they quietly stop producing good output. Most are built to verify that output exists rather than that it is good, so a degraded pipeline looks identical to a healthy one.
How long should I run a process manually before automating it? About four weeks of real use, which is how long the exceptions take to appear. A handful of runs teaches the steps but not the edge cases.
How do I monitor an AI workflow for quality? Attach a question before it goes live: if this ran unwatched for a month, what would be true and how would you find out. Then build the check that answers it and put a human read on a schedule.
What should I never automate? Any step that requires judging whether output is good enough. That judgment is the value, not the bottleneck.
Why did my automation produce bad work without failing? Because nothing in it was measuring quality. It optimised for the thing it was measured on, which was output existing.