The Surprise: When AI Reduces Developer Productivity

yuval bloch

The Surprise: When AI Reduces Developer Productivity

Once in a while, I run into a paper that makes me say, “F*ck.” These papers often go under the radar, but their results truly revolutionize how we look at something.

The paper, with its perfectly non-appealing name, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” is one of those. The brilliant interpretation by Cal Newport (the writer of Deep Work) in his podcast Deep Questions adds even more power to this finding.

I came across the paper and the podcast episode as part of my journey of self-experimentation aimed at improving my focus and productivity. I have a habit of reading relevant theory to improve my plans. Since I have recently been paying close attention to what makes me more productive, some of the results—like the fact that AI can reduce productivity in some tasks—were simply reassuring what I already suspected. However, other findings were truly shocking, such as the fact that experienced developers consistently failed to accurately assess the impact of AI on their work or the extent of the reduction.

The Study: 20% Slower, 20% Happier

So, what did the paper—which was released in July 2025 and has already reached 30 citations—find?

The authors actually planned the research to see how much generative AI improved the efficiency of experienced developers in real work tasks. To test this, they recruited experienced developers from major open-source projects and asked them what their main upcoming contributions were. They then randomly assigned these tasks to two groups: one using AI and one not using AI.

They collected data in a few critical ways:

  • Before the task: Developers estimated how much time each task would take with and without AI.

  • After the task: Developers estimated the difference in time taken.

  • Expert estimation: AI experts predicted how they believed AI would impact the task completion time.

  • Actual time: They measured the time from the start of the task to its finish (Pull Request accepted).

  • Observation: They recorded the screens of the developers.

The results were truly astounding:

While AI experts estimate a 50% speedup and the developer themself before the task estimate 24% speed up And after the task, the developer estimated a 20% speed up The true result was a 20% slowdown!

I mostly don’t like exclamation marks, but this deserves it: Developers believed they were working 20% faster when they were truly working 20% slower!

It’s not that they were completely clueless about their working speed; there was actually a good correlation between their time estimations and real work time within the AI and non-AI task groups. They were just specifically clueless about the impact of AI on their overall efficiency.

The authors went to great lengths to find artifacts and limitations in their own work; it looked like they almost felt the need to apologize for their results. But everything looked solid. It really seems like these findings tell us something fundamental about adding AI to our workflow.

Interpretation: The Focus Tax

This result can look enigmatic, but luckily, we have the screen recordings to dive in and ask why this happened. I want to start with a small but intriguing result:

The AI group spent almost twice as much time doing nothing. And I don’t mean waiting for an analysis to run or an AI system to generate something, but literally nothing on their computer—their screen showed no activity.

This might look surprising. The AI group should get stuck less, as they have something to help them get out of problems. The reason might lie in the relationship between work and focus.

Throughout our work, we use different levels of focus and attention:

  • Highest Level: Writing complex code.

  • Lower Level: Prompting the AI.

  • No Focus: Waiting for the AI to generate something.

And now comes the tricky part: Moving up the focus level takes time and effort. The AI group probably got stuck trying to return again and again to their deep focus after the AI broke their concentration.

Cal Newport says the key to any efficient knowledge work is long, uninterrupted focus. Even collaboration should be measured by how it impacts the length of our focus.

But I think there is another part to it. Going up the focus level is not just time-consuming; it is mentally hard.

Let’s say you have a part you believe AI can solve. You prompt it and wait for an answer, dropping your focus level. But then the AI generates the wrong code. Now you need to climb back up, but it’s hard, so you convince yourself that you don’t need to do the hard thinking—that you only need to change the prompt a bit. This is the trap of the comfort zone, and from my experience, it’s easy to get stuck there for a surprisingly long time.

Working Efficiently With AI

This does not, by any means, suggest that generative AI is bad or useless. (Disclosure: I use generative AI to edit this post, as I am not a native English speaker and struggle with grammar.) AI is a tool. Use it well, and it improves your work; use it wrongly, and it will damage it.

The only difference is that it is a new tool, so we still lack a lot of information about how to use it right. I hope this report will inspire more research into these questions, but we don’t have the privilege to wait. We live in a capitalist society where the only god is competition.

So, what can we do for now until science catches up with technology?

  1. Self-Experiment: Try to see how different working styles impact your results. Remember that even experienced developers were initially wrong about the impact of AI on their efficiency, so it’s better to use some sort of quantitative, relevant measure for your efficiency.

  2. Be Honest with Yourself: AI makes your work less mentally demanding, so you might feel a strong desire to prove it also makes you more efficient. You must look at it objectively. Incorporating AI into your workflow is also very trendy, so you must ignore the social pressure too.

  3. One Tool at a Time: Good experiments always separate between factors. It is better to incorporate tools slowly than to do it wrongly.

  4. Be Mindful: Try to notice: Did I really get forward, or am I stuck in a comfort zone?

  5. Keep Long Chunks of Focus Working Without AI: This is the first clear result of my self-experiment.

And remember: New is not always better.

It might feel scary, but I sense a great opportunity. The first ones to work correctly with AI will gain an advantage over the rest of the world, which will simply follow the trend. I think it is a rare opportunity to upgrade your status by adapting better to the new technological landscape—or as we say in biology: The fittest will survive.

Further Resources

  • The paper itself: https://arxiv.org/pdf/2507.09089

  • The podcast episode: https://open.spotify.com/episode/4gUlqIPkt0GDYi9ZYVJMSm?si=c77839506be74bb1