Generative AI and labour productivity: a field experiment on coding

Summary

Focus

Generative artificial intelligence (Gen AI) tools hold significant promise for enhancing worker productivity across various fields. These AI models have demonstrated capabilities comparable to humans in areas like clinical care, education, language modelling, art, music and design. A growing body of literature explores commercial and non-commercial applications, ethical considerations, regulatory frameworks, and implications for security and education. However, empirical research on AI's impact on productivity in tasks requiring cognitive abilities remains scarce.

Contribution

We investigate the impact of Gen AI on labour productivity through a field experiment in the coding industry. In September 2023, Ant Group launched CodeFuse, a large language model (LLM) designed to assist programming teams. In our experiment, one group of programmers had access to CodeFuse (the treatment group), while another group did not (the control group). By comparing similar employees from these two groups, we assessed how AI affected their productivity.

Findings

Our findings indicate that LLMs can significantly boost productivity among programmers. Productivity (measured by the number of lines of code produced) increased by 55% for the group using the LLM. Approximately one third of this increase was directly attributable to code generated by the LLM. The remaining productivity gains were likely due to improved efficiency in other coding tasks, as programmers had more time available. However, the productivity gains were statistically significant primarily among junior staff, with a less pronounced effect on senior employees. This difference appears to stem from lower engagement with the LLM by senior programmers, rather than the tool being less useful to them. The rate at which programmers accepted the LLM's suggestions did not vary with experience level, suggesting that the lower impact on senior programmers' productivity was due to less frequent use of the tool.

Abstract

In this paper we examine the effects of generative artificial intelligence (gen AI) on labour productivity. In September 2023, Ant Group introduced CodeFuse, a large language model (LLM) designed to assist programmer teams with coding. While one group of programmers used it, other programmer teams were not informed about this LLM. Leveraging this event, we conducted a field experiment on these two groups of programmers. We identified employees who used CodeFuse as the treatment group and paired them with comparable employees in the control group, to assess the impact of AI on their productivity. Our findings indicate that the use of gen AI increased code output by more than 50%. However, productivity gains are statistically significant only among entry-level or junior staff, while the impact on more senior employees is less pronounced. 

JEL Classification: D22, G31, R30

Keywords: artificial intelligence, productivity, field experiment, big tech