The Missing Net Effect in Product Development

Since May 2025, I have been blogging primarily about AI topics, focusing solely on technical aspects and experiments I’ve conducted, rather than socio-political dimensions. This will likely mark the end of this series for now, as I plan to return my focus to other technical areas.

Massively Accelerated Software Development

One of the most frequently repeated claims is that AI massively accelerates software development (and other similar fields) and enables enormous productivity gains. There are countless such claims. Figures on productivity gains are thrown around indiscriminately: sometimes twice as fast, sometimes ten or a hundred times faster; lately, even within a single sentence, as in the statement from Cloudflare’s CEO: “team members that were two, 10, even 100 times more productive than they had been before.”¹

A “perceived” increase in productivity is certainly widespread. But what are the final results? Twice as many products? 100 times more products? 100 times fewer bugs?

And certain (individual) tasks in software development are indeed significantly faster. But the list of such tasks has hardly expanded in recent years:

• Prototypes (the boundary with simple apps is fluid)
• Creating tools
• Help in research, summaries, translations, sentiment analysis
• Migrations (can run with tests in a feedback loop)
• Unit test generation
• Natural language interfaces
• Reviews

Tasks that are often accomplished with AI but yield only mediocre results:

• Evaluations, Assessments
• Concepts
• Complex applications
• Using AI for tasks for which specialized, deterministic tools already exist

However, there are relatively few quantitative assessments of the actual net effect in software development, by which I mean reputable studies.

Most importantly, AI does not work for outsourcing thinking or decision-making, even if you have relatively simple base rules.² Letting it make assessments is even more dangerously, as it’s bascially only a echo chamber.³

For example: If I receive a convincing but incorrect answer regarding the iOS Geofencing API and have AI generate a prototype that appears to confirm there is no solution within the set constraints, I may still need to read Apple’s documentation and conduct experiments to find a valid solution. In such cases, the process is no faster than simply reading the documentation from the start. On the contrary, it slows you down.⁴

Product development is inherently an iterative process. The shortcut of defining everything upfront and then letting someone generate the entire product in one go has never truly worked,⁵ except when you’re trying to copy something that already went through this process. In such cases, AI can act as a booster. There are good reasons why we build an MVP first and then move forward incrementally.

And in this cycle, actual programming is only a small part. You must get feedback from customers, weigh new features against existing ones, and operate within complex economic and legal constraints.

Even if you lack deep knowledge of the specific software architecture or domain, you can now rapidly generate code. Developers can be assigned to any project and any tech stack on demand. You can even automate code reviews. So why wouldn’t this lead to a substantial net productivity gain?

The problem seems to be complexity, and the fact that you don’t know what you don’t know.⁴ When AI makes tasks suddenly appear easy and quick to perform, it triggers human overconfidence bias.

To illustrate this, a quote from Wikipedia:⁶

Svenson (1981) surveyed 161 students in Sweden and the United States, asking them to compare their driving skills and safety to others’. For driving skill, 93% of the U.S. sample and 69% of the Swedish sample rated themselves in the top 50%; for safety, 88% of U.S. respondents and 77% of Swedish respondents placed themselves in the top 50%.

Mathematically speaking, this is a miracle.

Another issue is the sheer volume of easy, rapidly producible output. For instance, a single person today can generate hundreds of “technical books” in a few months,⁷ books that look legitimate and are hard to assess at first glance; I nearly fell for it myself.⁸ The fact that these appear on reputable retailers doesn’t help. Fast and abundant does not mean good or helpful, in fact, it’s often the opposite.

This situation reminds me of what I read about the 1982 video game crash, triggered when the market became flooded with cheap, low-quality content until consumers lost confidence and the entire ecosystem collapsed.

Things that turn out to be useless after only a short time, slop such as worthless books, articles, and the like, are much harder to identify when it comes to software.

You quickly generate code; due to its sheer volume, it’s no longer reasonably reviewable and gets approved with a “LGTM.” You submit contributions for problem areas or projects you only barely understand.

I can certainly understand why this creates a subjective feeling of high productivity.

However, the problems emerge downstream. While it is obvious that turning these hundreds of “technical books” into useful knowledge resources would require a disproportionate, massive effort, it is much more subtle when it comes to software development.

• More outages and failures
• Root cause analysis becomes difficult
• Fewer thorough reviews due to volume
• Less testing, excessive trust in AI
• More bugs, more edge-cases, customers perceive less quality
• Bug fixes under time pressure costs money and more resources
• Redundancies mean a single bug fix must be repeated and adapted across multiple contexts (a nightmare in complex workflows, many scenarios need testing, or tests are dropped)

From Subjective Impressions to Quantitative Metrics

Companies that offer developer analytics are of interest because they have insights into their customers’ workflows. One such company is Faros, which was founded in 2019 by former Salesforce employees. Their data is based on a two-year dataset of 22,000 developers and 4,000 teams on the Faros platform.⁹

They report higher productivity (task completion has increased by 34% and the number of completed epics by 66%), which is somewhat to be expected given the speed of generation.

Reviews

The report mentions that 31% more PRs are merged without any review, and the median review time has increased by 500%.

Churn Rate

The churn rate measures how often code segments need to be modified within a specific time period after they were originally created. The higher the number of changes, the higher the churn rate.

While a certain degree of change is natural, as requirements evolve over time, good software architecture aims to establish design principles that allow features to be added incrementally through extensions rather than rewriting large portions of the existing code.

High churn rates indicate design instability, which is often caused by tight coupling. Code with high churn is statistically more prone to errors. It is a clear indicator of technical debt and rushed development, impairs maintainability and speed, and forces teams to spend more time fixing regressions or re-learning recently changed code. Or they deliver subpar quality.

Faros also notes a different possible explanation:

Legacy systems that had accumulated for years are finally being replaced, generating high deletion volumes that reflect productive architectural work rather than failure.

Anyhow, according to Faros, the increase in the churn rate is 861%.

Quick Start - slow progress after a while

After initial rapid progress, things quickly stall, you’re stuck and make only slow progress. You get regressions. Sometimes you also get weak results that require massive AI effort to improve. The winner here are the AI providers.

Additional AI Usage to Cope with Slop

Then you also have to deal with even more slop: half-baked, incomplete artifacts, and mountains of text that you still have to grapple with somehow. Who reads all this? This is for sure counter-productive. One way to somehow manage the additional workload, of course, is to use even more AI.

Stalled Personal Skill Development

And then there’s the problem of stalled personal skill development, loss of overview, competence, and skills, which diminishes your own abilities.¹⁰

Core Competencies

But when it comes to core competencies, there are no shortcuts: I have to learn, experiment, and fail until I get it. Of course, this process is slower at first, which is why it’s unpopular, but it’s the only way to go if you want to deal with complex issues in the medium or long term.

A heavy reliance on AI can ultimately cause progress to stall within a relatively short period of time; in contrast, a step-by-step approach, in which each individual step is fully understood, leads to faster and better results in the medium to long term, even if the initial pace is slower.

Pragmatic AI Usage

Does that mean I don’t use AI anymore? Of course I do. I’m pragmatic. Anything that’s only marginally relevant to me but needs to be done and saves me time is a good candidate. For example, I needed some XPath expressions for web scraping. I sent the request along with an HTML example to a local LLM (I currently prefer GLM-4.7 Flash for such things), and a few minutes later, the expressions were returned to me. That saved me hours of work. I only need XPath every few years; it’s not usually important for my work.

Another example is sentiment analysis of reviews in different languages, where I found that a locally running instance of Qwen3-Next-80B-A3B delivers good results: A small Go program sends the data to the local LLM and retrieves the categorization and translations.

An interesting side effect of using AI in a granular way is that smaller tasks can now be handled by local LLMs, which are more powerful than many people assume.

For more demanding tasks, I still have the option of using more powerful models. I’ve even started trying out an EU alternative called Cortecs,¹¹ which offers stricter data protection and no-training guarantees. There, I have access to models hosted in the EU, such as GLM 5.1 from Z.ai. Unlike subscriptions with Anthropic or OpenAI, this isn’t subsidized, of course you pay for actual usage.

However, when one focuses solely on the truly useful applications of AI, demand and costs turn out to be surprisingly low.