Reviewing the flipped classroom research

Open Universiteit Computer Science Education Research Conference Reviewing the Flipped Classroom Research: Relections for Computer Science Education Michail Giannakos John Krogstie Nikos Chrisochoides 22 Reviewing the Flipped Classroom Research: Relections for Computer Science Education Reviewing the Flipped Classroom Research: Reflections for Computer Science Education Michail N. Giannakos and John Krogstie Norwegian University of Science and Technology (NTNU), Trondheim, Norway and Nikos Chrisochoides Old Dominion University, VA, USA Recent technical and infrastructural developments posit flipped (or inverted) classroom approaches ripe for exploration. Flipped classroom approaches have students use technology to access the lecture and other instructional resources outside the classroom in order to engage them in active learning during in-class time. Scholars and educators have reported a variety of outcomes of a flipped approach to instruction; however, the lack of a summary from these empirical studies prevents stakeholders from having a clear view of the benefits and challenges of this style of instruction. The purpose of this article is to provide a review of the flipped classroom approach in order to summarize the findings, to guide future studies, and to reflect the major achievements in the area of Computer Science (CS) education. 32 peer-reviewed articles were collected from a systematic literature search and analyzed based on a categorization of their main elements. The results of this survey show the direction of flipped classroom research during recent years and summarize the benefits and challenges of adopting a flipped approach in the classroom. Suggestions for future research include: describing in-detail the flipped approach; performing controlled experiments; and triangulating data from diverse sources. These future research efforts will reveal which aspects of a flipped classroom work better and under which circumstances and student groups. The findings will ultimately allow us to form best practices and a unified framework for guiding/assisting educators who want to adopt this teaching style. Categories and Subject Descriptors: K.3.2 [Computer and Information Science Education]: Computer Science Education. H.5.3 [Group and Organization Interfaces]: Collaborative computing, Computer-supported cooperative work, Evaluation/methodology General Terms: Experimentation, Human Factors, Design, Measurement. Authors addresses: michailg@idi.ntnu.no, krogstie@idi.ntnu.no, nikos@cs.odu.edu Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from: Permissions@acm.org. CSERC '14, CSERC November – 06, 2014, Berlin, Conference ’14,055-6 November 2014,Germany Berlin, Germany ©2014 ACM ISBN 978 1 4503 3347 4 $15.00. Copyright 2014 ACM 978-1-4503-3347-4/14/11…$15.00 http://dx.doi.org/10.1145/2691352.2691354 23 Additional Key Words and Phrases: Inverted Classroom, Classroom Flip, Reversing Lecture, Literature review, Benefits, Challenges, Computer Science Education. ACM Reference Format: Giannakos, M. N., Krogstie, J., and Chrisochoides, N. 2014. Reviewing the Flipped Classroom Research: Reflections for Computer Science Education. In Computer Science Education Research Conference (CSERC '14), (November 2014), 7 pages. 1. INTRODUCTION Traditional lecture style is the most common teaching approach used in higher education classes; however, the traditional lecture style of teaching can often place students in a passive role, which typically involves students retaining isolated facts that can later be forgotten. Over the last few decades, instructors have moved away from the traditional lecture approach by increasing technology use as a way to extend and enhance students’ understanding. One strategy recently adopted by many stakeholders is the flipped (or inverted) classroom approach: students use technology to access the lecture and other instructional resources outside the classroom, leaving the in-class time to engage in active learning through problem-based learning and practice activities [Giannakos and Chrisochoides, 2014; Bishop and Verleger, 2013]. The flipped classroom is a specific type of blended learning design that restructures the traditional lesson [Lage et al. 2000]. Recent developments in open education and video lectures have given rise to flipped classrooms. Since this specific type of blended-learning classroom can utilize technology, such as video and other forms of multimedia, to move lectures outside the classroom, students and teachers have time for active learning in the classroom [Bishop and Verleger 2013]. The flipped classroom has been used in a number of education studies [Bishop and Verleger 2013], particularly in higher education, with very encouraging results [Gannod et al. 2008; Mason et al. 2014]. Our motivation for this work is based on the emerging developments in the area of video and learning technologies, creating momentum for the adoption of the flipped approach in CS education. The purpose of this paper is to provide a review of research on the flipped classroom approach in order to summarize the findings, guide future studies, and reflect the major achievements in the area of CS education. This study can provide a springboard for other scholars and practitioners, Open Universiteit Computer Science Education Research Conference especially in the area of CS education, to examine flipped classroom approaches by taking into consideration the prior and ongoing research efforts. 2. BACKGROUND The flipped (or inverted) classroom approach was coined in 2000 [Lage et al. 2000], though relevant research in the area remained limited until the last couple of years. It can be argued that the key factor that increased stakeholders’ interest and postulated the flipped classroom approach as one of the main buzzwords of academic circles was recent developments in open education and technologies. To date, advanced video repository systems have seen enormous growth (e.g., Khan Academy, ITunes U), with social software tools increasing the possibilities to enhance learning material. Most of the communication software tools, including wikis, weblogs, and e-portfolios, can potentially provide a vehicle to promote the flipped classroom approach. In addition, the widespread adoption of many different video learning platforms from all the prominent universities around the world posit this hybrid teaching method ripe for exploration and increase stakeholders’ interest. In 2000, Lage et al. talked about creating an inverted classroom for accommodating many different student learning styles. Afterwards, many other researchers have used the flipped classroom model using different technologies and pedagogical strategies [Bishop and Verleger 2013]. In the last few years, a substantial body of research has been conducted and published. In some studies, we have seen very positive results where the learning performance increases and students’ attitudes and adoption of this instruction model are high [Davis et al. 2013; Stone 2012]; however, we have also seen that students disliked video lectures and claimed that recorded lectures are not appropriate for more difficult course material [Strayer, 2012; Gannod et al. 2008]. Overall, several studies have been conducted, resulting in many useful findings, primarily in the area of CS education. Therefore, with this paper we attempt to provide a review of the flipped classroom approach in order to summarize the findings, guide future studies, and reflect the major achievements in the area of CS education. The current review is unique since it reviews the flipped classroom style through the lens of CS education, analyzes over 30 peer-reviewed articles, offers systematic analysis of both benefits and drawbacks, and provides opportunities to improve future research. 3. 3.1 articles. This process was conducted independently by two experts, a CS educ. researcher and a research librarian. After this process, the two researchers present their results and discussed and solved any discrepancies. The search process uncovered 32 peer-reviewed articles (Table 1). 3.2 Analysis Each collected study was analyzed based on the following elements: educational level of the sample, subject area, methodology type (qualitative, quantitative, mixed), measure type (e.g., Attitudinal, Learning Performance, Attendance), instruments used (e.g., Surveys, Tests, Interviews), sample size, technology used, experiment design (e.g., between groups, within groups), and whether the students’ worked collaboratively or not. 4. RESULTS OF THE REVIEW One can claim that the research on Flipped-Inverted Classroom Research has been growing in recent years. Based on our analysis, we can verify this assumption, as we can see from Table 1 that the number of papers published during the last three years has increased significantly, especially compared to the few and scattered papers before 2012. In this section, Table 1 presents the detailed results, followed by an analysis and discussion of the categories. In addition, Figures 2 and 3 present respectively the benefits and drawbacks of flipped classroom approaches, as they arose from the reviewed articles. 4.1 Sample The categories related to the sample of the articles, include the number of students participating in each study (size) and their educational level (i.e., undergraduate, graduate). The majority of the studies involve undergraduate students (26), with few studies at the graduate level (2) and at a mixed graduate and undergraduate course (3). Regarding the sample size (figure 1), most of the studies (21) have been conducted with less than 100 students, and few (6) can be considered large scale studies (more than 200 students). 600 Sample Size 500 400 METHODOLOGY 300 Article Collection Several procedures were followed to ensure high quality review of the literature of flipped classroom approaches. A comprehensive search of peer-reviewed articles was conducted through June 2014, (short papers, posters, and reports were excluded), based on a wide range of key terms including: Flip the classroom, flipped class, flipping, inverting, reversing, inverted classroom, and derivatives of these terms. A wide variety of databases and meta-databases were searched, including the ACM Digital Library, IEEE Xplore, AACE Digital Library, Academic Search Premiere, EBSCOhost, ERIC, Google Scholar, and Scholars Portal Journals. Additionally, the reference section of each article found was searched in order to find additional 24 200 100 0 sample size of each paper Fig. 1. Graph visualizing the sample distribution of the 32 research papers (sorted by sample size). Reviewing the Flipped Classroom Research: Relections for Computer Science Education Table I. Published Studies of the Flipped Classroom Study Educ. Level UG Subject Area Bio Method. Type Mixed Gehringer and Peddycord 2013 Sarawagi, 2014 Toto and Nguyen 2009 Pierce and Fox 2012 Mixed Mixed UG UG Comp. Archit. CS0 ENG Measure Type Atte, Atti Per Per, Acti Sample Size 400UG30GR 8UG-44 GR 26 89 Tech. Used VL, AR VL Exp. Design BT Collab. Learning Yes BT Yes Atti Atti Instrum ents SR, OEQ, LPT SR, LPT, OBS SR, OBS SR Mixed Quant VL VL Non Non Yes No Mixed Medicine Quant Per, Atti SR, LPT 71 VL No ICT Quant Per, Atti SR, LPT 301 UG Physics Mixed Mixed 30 Simul WT No UG Digital ENG Statistics Mixed Atti 50 ITS BT No Papadopoulos and Roman 2010 Lockwood and Esselstein 2013 Mason at al. 2013 Herold et al. 2012 UG ENG Quant Atte, Atti SR, LPT, OEQ SR, FG, OBS FN, INT, SR, FG LPT, SR 200 UG Atte, Atti, Per Atti VL, Simul AR BT & WT BT & WT WT Davies et al. 2013 UG Bates and Galloway 2012 Warter and Dong 2012 Strayer 2012 63 AR WT Yes UG Mixed Per, Atti SR, OEQ 35 Non Yes Quant Mixed Per, Atti Per 20 106 BT BT Yes Yes UG Quall Atti SR, LPT SR, INT, LPT SR AR, VL VL VL Thomas 2014 Progr. & PS ENG Soft. ENG Progr. Non No McCray 2000 Gannod et al. 2008 UG UG Quant Quant Atti Atti SR - 50 80 BT WT No Yes Enfield 2013 UG Quant Atti, Per SR, OEQ 50 VL, Non Yes Szafir and Mutlu, 2013 Campbell et al. 2013 GR IS Soft. ENG Web Design - VL, Simul VL VL Quant VL BT No CS1 Quant EEG, SR, LPT SR, LPT 48 UG 351 VL WT Yes Foertsch et al. 2002 UG PS, CS Mixed Per, Atti, BM Atte, Per, Atti Atti, 531 VL BT Yes Largent 2013 Day and Foley 2006 Wilson, 2013 UG UG UG CS0 HCI Statistics Mixed Quant Quant Atti Atti, Per, Atti, Per 49 46 - VL VL - Non BT BT Yes Yes Yes Ferreri and O'Connor 2013 Gaughan 2014 Love et al. 2014 Lage et al. 2000 McGivney and Xue 2013 Forsey et al. 2013 GR Medicine Quant Per, Skills SR, LPT, INT SR, OEQ LPT, LPT, OEQ, SR, LPT, SR 399 - BT Yes UG UG UG UG History Math Ec/mics Math Quall Quant Mixed Mixed Per, Atti Per Per, Atti 36 55 189 60 VL VL VL VL Non BT Non BT Non Yes Yes Yes - Sociology Mixed Atti OEQ LPT, SR SR, OEQ LPT, FG, SR SR, FG, INT LPT, SR LPT, SR 80 VL Non No Stone 2012 UG Mixed - No No Murphree 2014 UG History Quant Per, Atti 213 WT No Larson and UG IT Mixed Per, Atti 125 VL WT No Yamamoto 2013 Talbert 2013 UG Math Atti SR 53 VL WT Yes UG, Undergraduate; GR, Graduate; ENG, Engineering; PS, Problem Solving; Atte, Attendance; Atti, Attitudes; Per, Learning Performance; BM, Biometrics; SR, Surveys; OEQ, Open Ended Questions; LPT, Learning Performance Tests, OBS, Observations, INT, Interviews; FG, Focus Groups; FN, Field Notes; VL, Video Lectures; AR, Animated Readings; BT, Between Group; WT, Within Group; 25 Open Universiteit Computer Science Education Research Conference Table II. Type of the research papers Type of methodology No of Studies Quantitative 15 Qualitative 2 Mixed 14 Type of experimental design No of Studies Between 13 Within 8 Non 9 Between & Within 2 Table III. Measures used and Instruments Measure Type Attitudes Attendance No of Studies 27 Instrument Survey No of Studies 29 4.2 Skills Biometrical (EEG) 4 Learning Performance 19 1 1 Open Ended Questions 8 Learning Performance Tests 18 Interviews Observations 4 3 4.4 Technology Used to Flip and Measures Used Subject area With respect to subject area, it is clear that CS/IT subjects are dominant in flipped-learning research, with nearly half of the collected studies (15) conducted in CS/IT subject area. Research has also been conducted in subjects such as Mathematics (4), history (2), and medicine (2). Specifically in the CS/IT subjects, many studies (7) have been focused on introductory courses like IT, ICT, CS0, CS1, problem solving, and introductory programming. Nevertheless, studies have also been conducted in less generic subjects like HCI, Software Engineering, IS, Web Design, and Digital Engineering. 4.3 Data from Focus Group 4 Type of research One of the most important aspects in educational research is the type of methodology. By “type of methodology” we refer to the distinction among quantitative, qualitative, and mixed research. In addition to the type of methodology, our categorization used the type of experimental design, referring to the distinction among the following experimental designs: within-groups, between-groups, non-design, and a combination of within-between research designs. In a between-groups (or between-subjects) research design, we have more than one group, named control and experimental group(s), and the subject is observed only one time, followed by a comparison between the control and experimental groups. In a within-groups (or subjects) design, a subject is observed at the beginning and end of the treatment; then a pre-post comparison follows. Based on this categorization, we can see from Table 2 that the majority of the papers are quantitative (15) and mixed (14,) with few studies (2) being totally qualitative. Regarding the type of experimental design, most of the studies followed a between-groups design (13) and fewer within-groups design (8). Many of the studies (9) did not follow any of these designs, but simply measured the results after the respective treatment. A very small number of studies (2) combined between and within group designs. 26 Concerning the technology used, most of the studies used video lectures (25); some studies also used animated readings (4) and simulations (3); many times animated readings and simulations were used on the top pf the video lectures. Only one study used more complex Intelligent Tutoring System (ITS) technology. Regarding the measure used on the studies, most of the studies used attitudinal (27) and learning performance (19) measures (table 3). On top of the aforementioned measures, some studies also captured students’ attendance (4), skills (1) and biometrical data using EEG (1). In order to be able to capture this information, researchers used different instruments and many times a combination of them. In particular, 29 studies used surveys, 18 performance tests, 8 open ended questions, 4 interviews, 4 data from focus groups, 3 studies used observations and one study used EEG (table 3). 4.5 Summary of benefits Figure 2 (next page) provides a summary of six key benefits regarding the use of the flipped classroom teaching approach. First, as most of the studies investigating students’ learning performance mentioned, flipping the class is a way to improve learning performance. Students particularly welcomed the fact that they had access to materials like video lectures, and they were able to prepare themselves and even learn when, where, and at the pace they wanted; students welcomed their ability to learn independently. Second, more than half the studies analyzed suggest that students have very positive attitudes toward the flip classroom approach, describing the approach as useful, helpful, and flexible. Third, a number of papers indicated high levels of engagement. Based mostly on qualitative data and instructors’ observations, it is stated that even if the performance is sometimes low, student engagement remains at a high level. Fourth, it is found that there is a measurable increase in the quantity of discussions, although the quality of discussions was not assessed in the collected studies. Fifth, flipped classrooms force students to work collaboratively, and qualitative evidences indicate that students Reviewing the Flipped Classroom Research: Relections for Computer Science Education improved their cooperative skills. Finally, a number of articles indicated that learning habits change as a result of this approach and the availability of the extra learning materials, especially when those materials were revisited prior to a revision or examination.       Benefits Increases Learning Performance Positive Attitudes Increases Engagement More Discussions (Qualitatively Measured) Enforces Cooperative Learning Better learning habits       5. Fig. 2. Benefits identified in the reviewed studies 4.6 Summary of challenges Figure 3 provides a summary of three key challenges regarding the use of flipped classroom teaching approach. First, most of the studies mention the high initial cost in terms of preparation time and for the development of the inverted materials; however, this is reduced after the first year of a flipped class since the instructor is better prepared and some of the materials can be reused. Second, students sometimes struggled with the new format and were occasionally unreceptive to follow the structure; however, most of the time they adapted quickly and ultimately found the inverted classroom format to be satisfactory and effective. Finally, a number of articles indicated that by using the flipped classroom approach you might encounter a decrease in attendance, especially in large courses.    Challenges High initial cost and very time consuming for the instructor Students unreceptive with the Structure Decrease of Attendance Fig. 3. Challenges identified in the reviewed studies 4.7 Summary of flipped classroom research This paper reviews the status of flipped class research of the last years and shows that: The number of papers has significantly increased during the last years; The focus of the recent studies is on empirical quantitative and mixed studies; There is a lack of focus on qualitative oriented studies; There is a high emphasis on CS/IT domains with particular focus on introductory courses like introductory programing and problem solving courses (e.g., CS0, CS1); Researchers focused on students’ attitudes and learning performance; and There are certain benefits and challenges of using a flipped class approach in your course (figures 2 and 3). DISCUSSION AND CONCLUSIONS After reviewing over 30 studies on flipped classroom approach, we can agree on the offered affordances and the positive effects on students' performance, attitudes, and engagement. Flipped learning can provide the students opportunities to learn in a more differentiated manner traditional than linear and passive forms [Willey and Gardner 2013]. Students have mentioned that they appreciate the ability to digest the content in a self-paced manner, so long as it was done before the next class period. Though the majority of students completed the required prerequisite tasks on a fairly regular basis, there was consistently a small portion that did not [Davies et al. 2013; Gaughan 2014; Murphree 2014]. In terms of student engagement, flipped learning received the most positive remarks from students in the qualitative surveys (open ended questions). Students perceived the use of classroom activities that activated higher-order thinking to be more engaging [Davies et al. 2013; Wilson 2013]. Additionally, the environment afforded students to remain at higher levels of Bloom’s Taxonomy for longer periods of time [Enfield 2013]. The longer students remain in the higher levels of thinking and problem solving, the more they feel engaged with authentic learning, and the perceived quality of the learning is greater [Wilson 2013]. Flipped learning empowered students through more active learning [Lage et al., 2000]. Rather than having the instructor’s interpretation of the material delivered explicitly during class time where students passively take notes and possibly ask questions, the students were held more accountable for the front-loading of content. This more active role is difficult for some students to adjust to [Bormann 2014], but it was evident that the flipped method was successful overall, especially looking at the percentage of students who prefer a flipped environment to a traditional one [Enfield 2013]. Furthermore, the studies reviewed provide a wide variety of methods for flipping the class. Some studies used quizzes outside the classroom [Enfield, 2013; Mason et al. 2013; Strayer 2009] while others used quizzes inside the classroom [Papadopoulos, 2010]. Some studies emphasized the importance of these quizzes to students and counted them for low-stakes grading [Bormann 2014], while others simply provided them as a resource to the students with no grading benefit [Gaughan 2014]. The current review suggests that, while there are some challenges for students and instructors, there are also a number of benefits. More importantly, there are also opportunities for improving instruction that might not be feasible for traditional teaching approaches. In particular, 27 Open Universiteit Computer Science Education Research Conference students’ learning performance is often observed to increase and others to stay on the same level (compared to traditional instructing approaches); students’ engagement, group-work, and critical thinking have clearly benefited from flipping the classroom. On the other hand, there are challenges, such as the instructors’ claim that they spend significantly more time preparing to flip the class and that students sometimes seem unreceptive to this teaching style and their attendance decreased. 6. FUTURE DIRECTIONS FOR THE FLIPPED CLASSROOM A number of suggestions for further research have emerged from reviewing prior and ongoing work on flipping the classroom. One recommendation for future researchers is to clearly describe the flipped classroom approach by providing detailed information for the materials used, as well as the pedagogical strategies, especially in subjects like IT/CS where technology sometimes has both the role of the content and the medium. This will allow us to identify which aspects, technologies, and concepts of the flipped classroom work better than others and to form best practices, providing a springboard for other scholars. Another recommendation is to focus more on the in-class part of the flipped classroom approach; limited research has been conducted on how instructors can motivate and engage students in active participation and critical discussions, as well as how technology can assist in that direction. Future work should also focus on collecting and triangulating different types of data from different sources. Although the reviewed studies have been conducted using a wide range of collected data, ranging from students’ attitudes and learning performance to even biometric characteristics, the interpretations and triangulation between the different types of the collected data were limited. For example, issues referring to any potential effect of students’ attitudes on their learning performance or attendance have not yet been explored. In-depth qualitative investigation on low performers and adopters is also yet to be conducted. These future research efforts will allow us to understand which aspects of flipping the classroom work better and under which circumstances and students. ACKNOWLEDGMENTS The authors would like to express their gratitude to the research librarian, Christopher Lawton. REFERENCES Bates, S., & Galloway, R. 2012. The inverted classroom in a large enrolment introductory physics course: a case study.. Proceedings of the HEA STEM Learning and Teaching Conference. DOI: 10.11120/stem.hea.2012.071 Bishop, J.L. and Verleger M.A., 2013. The Flipped Classroom: A Survey of the Research, In proceedings of the ASEE Annual Conference, Paper # 6219, Atlanta, GA. Bormann, J. 2014. Affordances of Flipped Learning and its Effects on Student Engagement and Achievement. doctoral dissertation, University of Northern Iowa. Chicago. Campbell, J., Horton, D., Craig, M., & Gries, P. 2014. Evaluating an inverted CS1, In Proceedings of the 45th ACM technical symposium on Computer science education, ACM Press, 307-312. 28 Davies, R. S., Dean, D. L., & Ball, N. 2013. Flipping the classroom and instructional technology integration in a college-level information systems spreadsheet course. Educational Technology Research and Development, 61(4), 563-580. Day, J. A. & Foley, J. D. 2006. Evaluating a web lecture intervention in a human–computer interaction course. IEEE Transactions on Education, 49(4), 420–431. Enfield, J. 2013. Looking at the Impact of the Flipped Classroom Model of Instruction on Undergraduate Multimedia Students at CSUN. TechTrends, 57(6), 14-27. Ferreri, S. P., & O'Connor, S. K. 2013. Redesign of a Large Lecture Course Into a Small-Group Learning Course. American Journal of Pharmaceutical Education, 77(1), 1-9. Foertsch, J., Moses, G., Strikwerda, J., & Litzkow, M. 2002. Reversing the Lecture/Homework Paradigm Using eTEACH® Web‐based Streaming Video Software. Journal of Engineering Education, 91(3), 267-274. Forsey, M., Low, M., & Glance, D. 2013. Flipping the sociology classroom: Towards a practice of online pedagogy. Journal of Sociology, 49(4), 471-485. Gannod, G. C., Burge, J. E., & Helmick, M. T. 2008. Using the inverted classroom to teach software engineering. In Proceedings of the 30th international conference on Software engineering, ACM Press, 777-786. Gaughan, J. E. 2014. The flipped classroom in world history. History Teacher, 47(2), 221-244. Gehringer, E. F., & Peddycord III, B. W. 2013. The inverted-lecture model: a case study in computer architecture. In Proceeding of the 44th ACM technical symposium on Computer science education, ACM Press, 489-494. Giannakos, M.N,. & Chrisochoides, N. 2014. Challenges and Perspectives in an Undergraduate Flipped Classroom Experience: Looking through the Lens of Analytics, In the 44th IEEE Frontiers in Education Conference (FIE ‘14). Herold, M. J., Lynch, T. D., Ramnath, R., & Ramanathan, J. 2012. Student and instructor experiences in the inverted classroom. In Proceedings of the 2012 IEEE Frontiers in Education Conference (FIE), 1-6. Lage, M. J., Platt, G. J., & Treglia, M. 2000. Inverting the classroom: A gateway to creating an inclusive learning environment. The Journal of Economic Education, 31(1), 30-43. Largent, D. L. 2013. Flipping a large CS0 course: an experience report about exploring the use of video, clickers and active learning. Journal of Computing Sciences in Colleges, 29 (1), ACM Press, 84-91. Larson, S., & Yamamoto, J. 2013. Flipping the College Spreadsheet Skills Classroom: Initial Empirical Results. Journal of Emerging Trends in Computing and Information Sciences, 4(10), 751-758. Lockwood, K., & Esselstein, R. 2013. The inverted classroom and the CS curriculum. In Proceeding of the 44th ACM technical symposium on Computer science education, ACM Press, 113-118. Love, B., Hodge, A., Grandgenett, N., & Swift, A. W. 2013. Student learning and perceptions in a flipped linear algebra course, International Journal of Mathematical Education in Science and Technology, 45 (3), 317-324. Reviewing the Flipped Classroom Research: Relections for Computer Science Education tion Mason, G. S., Shuman, T. R., Cook, K. E., 2014. Comparing the Effectiveness of an Inverted Classroom to a Traditional Classroom in an Upper-Division Engineering Course, IEEE Transactions on Education, 56 (4), 430-435, 2013. McCray, G. E. 2000. The hybrid course: Merging on-line instruction and the traditional classroom. Information Technology and Management, 1(4), 307-327. McGivney-Burelle, J., & Xue, F. 2013. Flipping Calculus. PRIMUS, 23(5), 477-486. Murphree, D. S. 2014. “Writing wasn't really stressed, accurate historical analysis was stressed": Student Perceptions of In-Class Writing in the Inverted, General Education, University History Survey Course. History Teacher, 47(2), 209-219. Papadopoulos, C., & Roman, A. S. 2010. Implementing an inverted classroom model in engineering statics: Initial results. In proceedings of the American Society for Engineering Education (ASEE). Pierce, R., & Fox, J. 2012. Vodcasts and active-learning exercises in a “flipped classroom” model of a renal pharmacotherapy module. American journal of pharmaceutical education, 76(10). Sarawagi. N. 2014. A flipped CS0 classroom: applying Bloom's taxonomy to algorithmic thinking. J. Comput. Sci. Coll. 29(6), ACM Press, 21-28. Stone, B. B. 2012. Flip Your Classroom to Increase Active Learning and Student Engagement. In Proceedings from 28th Annual Conference on Distance Teaching & Learning, Madison, Wisconsin, USA Strayer, J. F. 2012. How learning in an inverted classroom influences cooperation, innovation and task orientation. Learning Environments Research, 15(2), 171-193. Szafir, D., & Mutlu, B. 2013. ARTFul: adaptive review technology for flipped learning. In Proceedings of the 2013 ACM annual conference on Human factors in computing systems, ACM Press, 1001-1010. Talbert, R. 2014. Inverting the linear algebra classroom. PRIMUS, 24(5), 361-374. Thomas, M. 2014. iOS app programming using an inverted classroom in a small department. Journal of Computing Sciences in Colleges, 29 (5), ACM Press, 179-185. Toto, R., & Nguyen, H. 2009. Flipping the work design in an industrial engineering course. In IEEE/ASEE Frontiers in Education Conference (FIE'09), 1-4. Warter-Perez, N., & Dong, J. 2012. Flipping the classroom: How to embed inquiry and design projects into a digital engineering lecture. In Proceedings of the 2012 ASEE PSW Section Conference. Willey, K., & Gardner, A. 2013. Flipping your classroom without flipping out. Paper presented at 41 SEFI Conference, Retrieved: https://www.academia.edu/5153284/Flipping_your_classroom _without_flipping_out Wilson, S. 2013. The Flipped Class: A Method to Address the Challenges of an Undergraduate Statistics Course. Teaching of Psychology, 40 (3), 193-199. 29