The use of artificial intelligence (AI) in coding is shifting the role of software development, but measuring lines of code is no longer a valid measure of developer productivity, a survey by AI software delivery platform Harness has found.
The poll of 700 software developers and managers across the US, the UK, India, France and Germany found that although 89% believe productivity metrics have improved, 81% said they are now spending more time reviewing AI-generated source code.
The survey found that when AI generates code, metrics that measure software development output improve and software development cycle times shorten. The developers surveyed said they feel more productive because AI means they can write more code, are able to tackle more complex problems and can to move faster through familiar work. However, the survey also revealed that one of the major drawbacks in using AI is that developer time spent on code review has increased dramatically.
The current state of AI-based tools means that people are often kept in the loop to avoid mishaps, and the need for having a human in the loop is happening in software development.
On average, the survey reported that 31% of a developer’s day is now consumed by AI-related invisible work that is not being measured. When we asked them where AI creates the most friction, 53% put reviewing AI-generated code as causing the most friction in their work. Over half (52%) said that the most friction was being caused by having to fix subtle bugs in AI code, while 48% said that having to explain the AI-generated code to teammates was causing them the most friction.
However, organisations tend to measure gross output in terms of the amount of code generated. According to Harness, they are not measuring where the productivity gains are being spent.
When asked whether they are worried about the use of AI tools to measure their performance, 96% of the developers polled said they are worried. Most developers who took part in the survey (94%) said tech debt, validation time and developer burnout are missing from their current metric. Over half (54%) expressed concerns over individual performance evaluations based on AI data.
Harness urges IT managers to track AI review time, debugging overhead and the cost of productivity lost due to developers having to switch between different environments. For instance, Harness recommended investigating a reported 20% gain alongside an unmeasured 31% overhead before planning the next investment cycle.
It also recommended that software development organisations within businesses understand fully the volume of code that is being completed, merged and deployed. While AI increases code volume, as Harness points out, it does not automatically increase code delivery.
“AI coding is the first technology shift in modern software that has changed not just what developers build, but how they spend their hours,” said Trevor Stuart, senior vice-president and general manager at Harness. “AI is reshaping the developer’s job entirely, and the measurement frameworks that the industry has relied on for the past decade weren’t built for this new unit of work.”
