Some educational researchers have argued that instant feedback on exercises improves student engagement and learning. In the context of computer programming exercises, instant feedback in many classrooms and online courses takes the form of "correct" or "incorrect" responses. In order to assess the utility of such binary instant feedback, we examined the exercise solutions submitted by students enrolled in an undergraduate computer science course offered over several years. Students completed weekly programming exercise sets. While the exercises were identical across the course offerings, some instantiations of the class provided students with binary instant feedback. An automated assessment system evaluated student solutions and promptly indicated if they were "correct" or "incorrect". We examined the effects that such feedback had on students' propensity to cheat on exercises and on their tendency to attempt or avoid subsequent exercises. We found that the students who received binary instant feedback were twice as likely to cheat on a programming exercise as students who were not receiving such feedback. We also found that students attempted fewer exercises when binary instant feedback was provided. These findings suggest that, with regard to computer programming exercises, automated assessment systems that are limited to providing binary feedback can prompt undesirable behaviors, even when the feedback is delivered instantly.