Why Most AI Pilots Stall Before Production, and How to Get Yours Live

More than half of AI pilots do not make it into everyday work. Some do: our own CRM and client portal are in everyday use. In our practice the model is rarely the reason. Pilots stall on the brief, on knowledge nobody wrote down, on requirements that move, and on a bill nobody expected. Here is what we have seen on our own products and on client projects, and what gets a pilot over the line.

IN SHORT

Why AI pilots stall

An AI pilot usually fails in the gap between what can be drawn and how the business actually works. Tools like Claude Code, Cursor and Codex have made the first version almost free. A working demo that took a team months three years ago can now appear in a day. Trying used to be expensive, so few ideas got tried. Now trying is cheap, so far more ideas reach the pilot stage, and the filter has moved. The question is no longer whether you can build it, but whether you can push it through and whether it is still fresh for the market. What did not get faster is understanding the business: why a filter behaves the way it does, why an old system has that odd rule, who pays for the tokens when people actually start using the thing.

That is why so many pilots look great on the demo and then stay there. It was built to show, not to carry real work on an ordinary Monday.

AI shortened the road to a pilot. Not the road to production.

We see this from both sides. We build AI-assisted products for ourselves, including our own CRM and Disorder, a concept fashion store, and we build them for clients: an ERP system, AI assistants, platforms that came to us as raw MVPs. Some of these went live, one is still at pilot level, and one is being rewritten while the requirements move under it. Each one shows a different reason pilots stall.

THE NUMBERS

What the research actually says

You will find numbers anywhere from half to 95 percent. They measure different things, so it is worth knowing which is which before quoting one.

Read together, the picture is consistent: more than half of AI pilots do not become part of everyday work. The 95 percent from MIT is the harshest figure, and it measures something stricter than "reached production": whether the pilot changed the numbers. Be careful with round figures that float around without a link to the publisher. We could not trace several of them to any original report.

The surveys list the familiar reasons: data that is not ready, unclear value, rising costs, weak governance. All true. What they do not show is what this looks like inside one project, so the rest of this article is about that.

THREE PILOTS

Three days, three stores, one winner

Disorder is our own concept project: an online store for an invented brand of unconventional punk and rave clothing, built far enough that it could launch. Out of curiosity we ran an experiment on it. The same task went to Claude Code through different people, each writing the prompt the way they understood the job: a senior product manager, a mid-level project manager, and a version based on the full research and brief.

We gave ourselves a strict time limit. One working day for research, one for the brief, one to put everything in order. At the end of the third day we had three working store pilots. Three years ago that would have sounded unreal.

The three products were completely different. Each could pass as a pilot: pages render, products show, the cart works. But only one of them worked like a store, and it was the one built on the full brief. That brief was written by our founder, who has acted as both project manager and product owner for 15 to 20 years, and who knows why certain things in stores are done the way they are and where the standard logic breaks.

Every prompt produced a pilot. Only one produced a store.

The research behind that brief included customer journey maps of competitors, how their filters and product pages work, and a study of clothing stores in the same niche that are successful there: high traffic and trust by Ahrefs and Semrush, fast pages, few rage clicks. The choice of technology for the final product came out of the same research, and the store itself is built on our own core.

THE BRIEF

Why the model still needs a real spec

The most telling difference was the filters. Claude Code knows a great deal about how online stores work. Yet with a simple brief that did not spell out the filter logic, it did what a model does with freedom: every product parameter became its own entity, and every combination of parameters became its own URL. The result was an enormous number of near-identical pages, which is unacceptable for search.

The store with the detailed brief did not have that problem, because the brief said which filter combinations deserve their own page, which ones stay on one URL, and why. That decision belongs to the business and to SEO, and it has to be made before the code. We wrote about the same order of work in our article on e-commerce SEO built into the store.

A filter is a business decision before it is code.

The same goes for design. A model can draw almost anything from a prompt and a couple of reference links, and the result looks like a template. The interface gets less generic and starts to work when the model receives the reasons: why we want this journey, why this product page is built this way, what the shopper is trying to do at each step. A beautiful picture of a filter is easy. The question is how it behaves with 40 sizes, three colors out of stock and a sale that ends tonight.

You can ask Claude Code to write the brief for you. It will write a good one, and it will still miss things, because a lot of what matters is not on the internet. It lives in the experience of the people who have built and run such products.

UNWRITTEN RULES

What the old system knew

This matters most when the new system replaces an existing one. A true pilot, a new product in a new market, is allowed to be naive: you test, you do not know a lot yet, and you lean on general experience.

A replacement is different. The old system, however ugly, contains years of decisions. Some odd rule in invoicing, a status that seems unnecessary, a field everyone fills in a particular way. Often those decisions are not in any comment, ticket or document. The developers who made them have left, and the reason left with them.

A model sees the code, not the history. If nobody on the project knows why the previous product was built the way it was, the new one will lose those rules without anyone noticing, and the business will discover it in production, usually with a client on the other end.

This is the part of AI adoption that cannot be delegated to a prompt. Someone has to sit with the people who use the old system and write down what it actually does, including the things it does by accident.

NOT A PILOT

Why our CRM skipped the pilot stage

Our own sales system is the opposite case, and it shows what changes the odds. We built a custom CRM covering leads, projects and invoices in about a month, with one senior developer who also tested when needed, an architect reviewing the work, and Claude.

Nothing stopped it, and the reason is simple: we never treated it as a pilot. Before the first line of code we had already decided to say goodbye to the previous CRM. We knew what Claude Code, Cursor and Codex can and cannot do at their current level. And we had five years of using and customizing different CRMs behind us, so the brief came from real use, not from a feature list.

There is one more thing that made the scope harder than it sounds. Nobody thinks of a CRM as just a CRM anymore. Owners want purchasing, some planning, pieces of ERP inside it, the same way SEO today is a mix of search, AI answers, trust signals and social channels. Products answer more questions than they used to, so the brief has to answer more questions too.

MOVING TARGET

When requirements change mid-build

Not every project has that clarity. We are building an ERP system for an architecture and construction business: purchasing materials from different countries and brands, logistics, currency rates and how they move, prepayments, stock, which warehouse everything goes to, and onboarding new contractors.

It has not failed, but it is not in production either. The requirements changed substantially along the way, and now the specification is being rewritten while the tasks themselves keep changing in real time.

This is one of the most common reasons pilots stall, and it is rarely about AI. Fast generation makes it tempting to start building before the business has settled what it wants. The code can be rewritten in days. Rewriting it every week is still a cost, and it wears out the people who have to test each new version.

What helps is unglamorous. Name one person on the client side who makes decisions. Freeze a part of the system that is already clear and launch it, instead of waiting for the whole thing. And write down why each requirement changed, so the next change is a decision, not a surprise.

THE BILL

When the pilot succeeds and costs more

Our AI website assistants are ready as a framework. We have tested them on our internal products and on client projects, including a high-traffic construction portal, and they work well. They are waiting for the client sites themselves to launch.

The test exposed a problem that rarely shows up in a pilot: success costs money. On the construction portal people talk to the assistant a lot, and the owner had counted on a smaller monthly spend. The assistant did nothing wrong. It was simply used more than the budget assumed.

A pilot runs on a handful of test conversations, so the usage bill looks small. In production the price of a language model is paid per conversation, and the number of conversations grows with traffic and with how useful the assistant is. A pilot that is judged only on whether it works will pass, and then fail on the invoice.

So the budget question belongs in the pilot, not after it. How many conversations do you expect at real traffic? Which questions can be answered from a fixed knowledge base instead of a full model call? Where does a cheaper model do the job? What is the monthly limit, and what happens when it is reached? We wrote more about how such assistants work in AI website assistants are not chatbots.

LIGHT LOAD

A pilot that works because nobody pushes it

Honesty requires one example from our own house. Our client portal is live and clients use it, but by our own standard it is still at pilot level: we know the list of things that need finishing. We are simply overloaded with client work right now, and the portal keeps getting moved down the queue.

It works fine, and part of the reason is that the load is light. A pilot with a few users forgives a lot: slow queries, rough edges, cases nobody thought of. The same system under ten times the traffic would show them.

This is how pilots get stuck without anyone deciding anything. Nothing breaks, so nobody decides to finish it, and the pilot slowly turns into permanent infrastructure without ever being built as such. The fix is to give it a date: either it gets the remaining work and becomes a product, or it is replaced.

THE GAP

From demo code to a system you can run

Even the winning Disorder pilot could not be launched in the form Claude Code produced. Pilots like that work well as a demonstration and save an enormous amount of time, but a real launch still needs a proper backend, the right stack, security and hosting. In our case the pilots were built on Next.js, and the version we would actually launch is built on our own core.

The other part of the gap is checking. Generating is fast. Verifying that the output is right, that the filter does not create thousands of pages, that the assistant does not invent prices, takes people and time, and that cost grows with every change. We covered it in The Cost of AI Isn't Generation. It's Verification.

And the code has to live somewhere a team can work with it. Many client projects we see are not in any repository at all, which makes AI-assisted work on them risky. A repository, a staging copy and a short manual for the AI tools are the minimum, which is what Your Codebase Needs a Manual for AI is about.

BEYOND THE CODE

When the idea has no road to market

Not every stalled pilot is a complex system. Sometimes the task is trivial: a simple website that the owner plans to run ads to. Nothing has to grow across markets or publish every week. The pilot still never goes live, and the reason is money for everything that comes after the build. People often have no hosting of their own and no idea what it costs to keep a site running, maintained and advertised. A website that simply exists under your name and a system that brings in complex inquiries, gets read and earns trust are two very different jobs with very different budgets.

AI adds its own twist here. Language models are inclined to agree. Ask one whether your idea is good and you will usually hear that it is great. The tendency is well documented: in April 2025 OpenAI rolled back a ChatGPT update because it had become too flattering. So people spend weeks building something that already exists, made by companies that hold the market and have years of head start. A model will rarely tell you that unless you ask it to look for competitors and argue against you.

We now see this in our own numbers. We recently launched an MVP Review service, and in the first month and a half it received four requests. All four turned out to be the same story. The founders believed they had a fresh know-how worth patenting, and in every case very similar products had already launched and were doing reasonably well, some six months earlier, some a year and a half. Several of those products were more developed than the MVP in front of us. With no real difference to point to, raising money for them was not realistic, and the founders had already spent a lot of time.

Several of them sent us screenshots of their AI conversations: the model had "done market research", looked around, found nothing similar and praised the idea. The quality of such research varies a lot. A quick answer from a lighter, faster model and a careful run in a deeper reasoning or research mode can reach opposite conclusions on the same question. Every chat interface carries a line under the reply saying the model can make mistakes and the answer should be checked. In the excitement of a new idea, and with vibe coding making the first version feel almost free, that line gets skipped. Four out of four is a small sample collected over six weeks, so treat it as a signal rather than a statistic. It is a loud signal.

Then there are good ideas in real niches that stall for reasons that have nothing to do with the product. The founder has no way to reach investors or people who could back it. Or the technology needs a patent, and patenting in the United States is neither quick nor cheap: practitioner estimates put a typical utility patent at roughly $15,000 to $25,000 through grant once attorney work is included, and software claims face extra scrutiny.

A pilot proves the idea can be built. Not that it can live.

The numbers behind this are sobering, even if no study measures "ideas that won a niche" directly. Of all US business establishments opened in 2013, about half were still operating after five years and about a third after ten, according to the Bureau of Labor Statistics. Among startups that had already raised a seed round in recent years, Carta's data shows that fewer than a third moved on to a Series A within two years. Those are companies that already convinced someone to invest. The share of raw ideas that become a lasting business is smaller still, and that is the honest context for any statistic about pilots.

CHECKLIST

Getting a pilot into production

What we would agree before starting any AI pilot, ours or a client's:

• A decision about the old way. If the old system or process is not going to be switched off when the pilot succeeds, the pilot has nowhere to go.
• A success measure and a decision date. Not "let's see how it goes", but what has to be true by which date for it to go live, or be closed.
• A brief written by someone who has run this kind of product, with the reasons behind the logic, not just the screens. For a store this means filters, product pages and URL rules first.
• The unwritten rules of the current system, collected from the people who use it every day.
• One decision-maker on the business side, and a log of why requirements change.
• A usage budget at real traffic, with limits and a plan for what is answered without a full model call.
• Real data early, not a cleaned-up sample that hides the hard cases.
• A path from demo code to a supported system: repository, staging, backend, security, monitoring, and someone responsible for each.
• A load test or a soft launch before the pilot turns into infrastructure by default.
• A budget for life after launch: hosting, support, advertising and content, counted before the build, not after.
• An honest check of the idea: who already does this, how they got there, and who will fund the next stage. Ask the model to argue against you, not to agree.

If several of these are missing, a pilot may still be the right move, because it is now the cheapest way to see an idea. Just call it what it is, a demonstration, and budget the road to production separately.

FAQ

Sources

• iTnews: Half of AI pilot projects make it to production, Gartner
• CIO Dive: AI project failure rates are on the rise, S&P Global
• CIO.com: 88% of AI pilots fail to reach production, but that's not all on IT
• Gartner: Over 40% of agentic AI projects will be canceled by end of 2027
• Gartner: Lack of AI-ready data puts AI projects at risk
• U.S. Bureau of Labor Statistics: 34.7 percent of business establishments born in 2013 were still operating in 2023
• Carta: Most seed startups never reach Series A
• BitLaw: What does a patent application cost?
• TechCrunch: OpenAI explains why ChatGPT became too sycophantic