How do you know an AI automation was built to last, not just to demo?

Why doesn’t “it works” mean “it’s maintainable”?

A demo proves an automation can run once, in front of the person who built it, under conditions they controlled. It proves nothing about what happens when an API changes shape, a field gets renamed upstream, or the person who wrote it leaves the project.

Those are two different questions, and a lot of AI automation work only answers the first one. The build looks finished because it does the thing. Whether anyone besides the builder can keep it doing the thing is a separate property, and it doesn’t show up in a demo.

What are buyers actually saying went wrong?

A thread on r/Entrepreneur titled “Vibe-coded automations are becoming a real problem” collected 82 points and 96 comments describing exactly this gap. Buyers described automations built by hired “experts” that had no error handling, credentials hardcoded directly into workflow configs, and logic nobody could explain, code that worked, but nobody knew why.

One comment from that thread is worth quoting in full, because it names the mechanism precisely: “ai didn’t lower the bar for quality, it lowered the bar for faking competence long enough to cash the invoice.” AI tooling makes it faster to produce something that runs. It does not make that thing maintainable, and a builder who stops at “runs” can now get there faster than ever.

The four signals that actually separate the two

These aren’t abstract best practices. They’re the specific things buyers in that thread pointed to, and they’re checkable without reading the underlying code line by line.

1. Does it handle errors, or does it just stop?

Open the workflow’s failure path, or ask the builder to show you one. When a step fails, an API times out, a field comes back empty, does anything log it, retry it, or alert a human? Or does the workflow just stop, with no signal to anyone that it stopped? Silent failure is the single most common way a “working” automation quietly becomes a non-functioning one, weeks before anybody notices.

2. Where do the credentials live?

Hardcoded into the workflow config is a specific, checkable failure mode, not a vague security concern. If an API key or password is pasted directly into the automation platform’s settings rather than pulled from a secrets manager or the platform’s own credential store, everyone with edit access to that workflow has that credential, and revoking access for one person means rebuilding the automation, not clicking a button.

3. Is the logic documented, or does it just happen to work?

“Logic that works by accident” is the phrase from the buyer thread, and it describes a workflow whose author can’t explain why a particular step does what it does, only that changing it breaks something. That’s not a hypothetical. It’s the state a workflow ends up in when it was iterated by trial and error against one specific test case and never written down. Ask for a one-page explanation of what the automation assumes about its inputs. If nobody can produce one, nobody currently understands the system well enough to safely change it.

4. Is it modular, or is it one block that does everything?

A workflow built as five identifiable steps, lead comes in, gets enriched, gets scored, gets routed, gets logged, can have one step fixed or swapped without touching the rest. A workflow built as one long chain with no clear boundaries can’t. Ask what happens if you need to add one more condition to the routing step. If the honest answer involves touching logic three steps away from routing, the system has zero modularity, whatever it looks like on the surface.

How do you check this without an engineer on staff?

You don’t need to read the underlying automation platform’s node graph yourself. You need the builder to walk you through it and watch how they answer four questions:

1. “Show me what happens when this step fails.” If they have to build the failure case live because it’s never been tested, that’s the answer.
2. “Where are the credentials stored?” A confident answer names a secrets store or the platform’s built-in credential vault. A vague answer, or one that involves opening the workflow settings to check, is the answer.
3. “Walk me through why this step does what it does.” Confidence and fluency here are a real signal. Hesitation, or “it just needs to be like this or it breaks,” is the other kind of signal.
4. “If I asked you to change one condition here, what else would you have to touch?” The honest scope of the answer tells you how modular the build actually is.

None of these require you to evaluate code. They require you to notice whether the person answering sounds like they built something they understand, or something that happened to work the last time they ran it.

What if you already have one of these problems?

Fixing it doesn’t usually mean a full rebuild. In rough order of urgency:

• Hardcoded credentials get rotated first, independent of everything else. That’s an exposure, not a maintainability nice-to-have, and it doesn’t wait for a broader fix.
• Error handling can often be added to an existing workflow without touching its core logic: a failure branch that logs and alerts is usually additive, not a rewrite.
• Documentation can be written after the fact, by someone willing to trace through the existing logic and write down what it assumes. Slower than documenting during the build, but far cheaper than reverse-engineering it blind during an outage.
• Modularity is the one that sometimes does need real rework, because it’s a structural property, not an add-on. If a single workflow has become unmanageable, that’s usually the point where a second opinion is worth more than another patch.

What does this mean for how we build?

This is our own standard, not a survey of the category, since nobody audits every vendor’s delivery practice, us included. Every Sprint hands over full documentation and Loom walkthroughs. The running system and credentials are yours, not locked inside our account. Scoped access, audit logging, and human-in-the-loop controls ship on every agent by default, not as something you have to ask for.

If you already have an automation you’re not sure about, whether you built it, hired someone else, or inherited it, the $1,500 Audit reviews it against these same signals as part of mapping your workflows, and hands you a build-ready blueprint either way, whether you fix it with us or with whoever built it originally.