A Case Study of Developer Bots: Motivations, Perceptions, and Challenges

"Continuous integration and deployment (CI/CD) is now a widely adopted development model in practice as it reduces the time from ideas to customers. This adoption has also revived the idea of “shifting left” during software development – a practice intended to find and prevent defects early in the software delivery process. To assist with that, engineering systems integrate developer bots in the development workflow to improve developer productivity and help developers identify issues early in the software delivery process. In this paper, we present a case study of developer bots in ABC company. We identify and analyze 23 developer bots that are deployed across 13,000 repositories and assist about 6,000 developers daily in their CI/CD software development workflow. We classify these bots across five major categories: ConfigViolation, Security, Compliance, Developer Productivity, and Code Quality. By conducting interviews and surveys with bot developers and bot users and by analyzing more than half a million historical bot actions over a period of one and a half years, we present the motivations and use-cases behind these bots, factors impacting their usefulness as perceived by the bot users, and the challenges associated with their use. We discuss these findings against existing literature on developer bots, and highlight characteristics of efficient developer bots."