AI Assistance Can Hinder Skill Development

In early February, on a train ride I read a study on how AI affects both productivity and skill formation (conceptual understanding, code reading, and debugging) where participants performed tasks requiring new knowledge.

Methodology:

Two groups performed two coding tasks and were required to answer a quiz afterward.

  • Treatment Group: Could use a chat-based AI coding assistant.
  • Control Group: Had no AI access.
  • Both groups did the same warm-up and then had 35 minutes for the tasks.
  • After the coding stage, everyone took a no-AI post-task quiz measuring conceptual understanding, code reading, and debugging.

Results:

  • Learning/Skill Formation: The AI-assisted group scored significantly worse on the quiz (about 17% lower), indicating reduced conceptual understanding, code reading, and debugging.
  • Productivity (Time): The AI-assisted group was not significantly faster in total task completion time. (I need to reread the study at some point to understand how the 4 out of 26 participants in the control group without AI who did not complete the second task within the time limit were taken into account.)
  • Time Shift: With AI, participants spent less “active coding time” and more time interacting with the AI assistant and understanding its outputs (Section 6.3 “Shifts in Active Coding Time”). Some participants spent up to 11 minutes on AI interaction during the 35-minute task.
  • Bad Patterns and Good Patterns: Heavy reliance patterns (e.g., AI delegation, iterative AI debugging, progressive AI reliance) were associated with low quiz scores, while more cognitively engaged patterns (e.g., conceptual inquiry, code+explanation, generation-then-comprehension) had much higher quiz scores.
  • Fixing Errors Helps Understanding: The control group encountered significantly more errors during the task (median 3 vs 1), and resolving these errors independently appears to be a key mechanism for learning.

I find it interesting that this shift from time spent coding to querying AI and understanding the results could be one reason why we sometimes don’t see big productivity gains.

This study is by Anthropic (maker of Claude Sonnet and Opus models): Worth noting that an AI company published research showing AI assistance can impair skill formation. The acknowledgments also thank several Anthropic colleagues plus external researchers for design feedback.

Therefore, an essential skill is learning to identify which tasks should not be delegated to AI, despite AI’s superior execution speed. Key criteria could include whether the task involves core competencies or conceptual understanding.

The paper: How AI Impacts Skill Formation by Judy Hanwen Shen, Alex Tamkin.