Received June 22, 2020, accepted July 3, 2020, date of publication July 9, 2020, date of current version July

21, 2020.
Digital Object Identifier 10.1109/ACCESS.2020.3008321

Curriculum, Teaching and Learning, and

Assessments for Introductory
Programming Course
ERUM MEHMOOD 1 , ADNAN ABID 1 , (Member, IEEE),
MUHAMMAD SHOAIB FAROOQ 1 , (Member, IEEE), AND NAEEM A. NAWAZ 2
1 Department of Computer Science, School of Systems and Technology, University of Management and Technology, Lahore 54770, Pakistan
2 Department of Computer Science (CFY), Umm Al-Qura University, Mecca 24231, Saudi Arabia

Corresponding author: Adnan Abid (adnan.abid@umt.edu.pk)

ABSTRACT Learning to program involves acquisition of various skills including problem solving,
fundamental design techniques as well as critical thinking. Generally, most of the novice programmers
struggle to develop all these important skill. The research community has addressed the problem in many
different ways while involving improvisations in curriculum, pedagogical methods, cognitive aspects,
supporting tools, and in designing assessments. This research aims to analyze and synthesize the existing
literature in the aforementioned areas. Research articles pertaining to the area of Introductory Programming
Courses (IPC) have been found using appropriate search queries, while nearly 60 research articles, published
in last ten years, have been carefully selected by employing a systematic filtering process. The scope of this
work only covers the research conducted for IPC in higher education. Main findings of this study show
that ‘‘solution proposal’’ and ‘‘evaluation research’’ have been reported as two main research types adopted
by these studies. Moreover, pedagogy, language choice and students’ performance analysis are the most
frequently addressed aspects of IPC; whereas, curriculum contents, assessment design, and teaching/learning
through tools have appeared as less addressed aspects of IPC. Furthermore, a taxonomy of IPC has been
presented based on the studied literature. Lastly, general considerations and future research directions have
been presented for the practitioners and researchers in this area.

INDEX TERMS Introductory programming, higher education, programming education curriculum,

language choice, systematic review.

I. INTRODUCTION the limitations of learning the syntax of a programming

Introductory programming generally refers to course intro- language, according to this report. Dropout rates are still
duced to undergraduate students with little or no program- high despite progressing in methods/tools for teaching
ming experience. Introductory programming courses (IPCs) and learning IPC [3]. Moreover, suitability of a language,
are included in various undergraduate degree programs. as IPC, needs to be evaluated in order to analyse challenges
Program design, data structures, syntax, problem solving, faced by novice students. Impact of language choice in
programming logic and design techniques [1] are the basic learning IPC is evaluated by developing a framework in
concepts IPC aims to cover. Final report of Curriculum a study by [4]. Despite having several published reviews
Guidelines for Undergraduate Degree Programs in Computer in IPC domain, studies either cover papers up to 2017 or
Science [2] says that introductory courses differ across focus more on learning methods and less on curricula and
institutions to help students with significant variance in assessment design for IPC as mentioned in Table 1. This
pre-requisite. Some introductory courses are developed to table provides comparison among existing reviews based
deliver a broader introduction to computing concepts without on five major perspectives: quality assessment scoring,
curriculum, teaching and learning, assessment and targeted
The associate editor coordinating the review of this manuscript and digital repositories. We have included comparison for only
approving it for publication was Luca Ardito . reviews published in quality journals (excluding conferences

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME 8, 2020 125961
 E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

TABLE 1. Comparison with related works.

and workshops) in our study as more matured research gets standards and language choice as IPC, and focus more
published in journals. This comparison help us build the need on teaching methods and tools than on student problems.
of this survey. A more recent systematic review on challenges faced by
This systematic literature review (SLR) provides a detailed novice programmers in learning introductory programming
examination of problems faced by novice programmers in course evaluated studies till year 2016 conducted by [3].
learning introductory programming education and research Authors focus on analyzing three stages of computational
trends in IPC while covering all five major perspectives thinking (problem formulation, solution expression and
shown in Table 1. Based on the systematic review criteria, solution execution and evaluation). This is the only other
60 research papers have been finalized for further review survey to our knowledge that thoroughly presents key issues
and analysis. The selected papers are empirically and for the research road map on introductory programming
qualitatively evaluated through multiple aspects. The novelty learning and teaching in higher education. However, this
of our systematic literature review is that it provides a new study does not focus on other factors playing important role in
classification criteria, IPC research targeting channels, IPC learning IPC such as: curricula standards and language choice
curricula [2], [49], programming language choice [4], [24], as IPC.
teaching and learning approaches [29], [45], assessment Another SLR presented by [5] investigates the effective-
design [19], [21] and tools [23], [57], and approaches ness of using robots as teaching tools for IPC. Authors
to address challenges faced by novice students after val- discuss different technologies appearing helpful for learners
idating programming studies empirically. This SLR will to overcome existing barriers in this context. This SLR
help educators in development of standardized IPC learning discovers that most frequently adopted programming by
environment together with curricula, teaching and learning educators is JAVA for those who use robots as a teaching
tools, and effective assessment tools and design. tool. According to this study, high quality and large scale
This article is arranged as follows: Section II discusses research is still required in order to discover true efficacy
about existing relevant surveys, and provides a motivation of robots as tool for teaching programming. However, other
for this SLR. Section III presents adopted methodology to tools and methodologies of learning IPC are not discussed in
conduct this survey together with objectives and research this survey.
questions. Answers to these research questions have been Models of learning styles for IPC are recently reviewed
described and analyzed in Section IV. Section V presents by [6]. Total of five learning approaches appeared in this
synthesis of reviewed literature by describing a taxonomy review: online self-paced, face-to-face instructor, online col-
in this domain. Lastly, the article has been concluded laboration, face-to-face collaboration and online instructor-
in Section VI. led. Effectiveness of various learning approaches have been
evaluated on basis of students’ performance when blended
II. LITERATURE REVIEW together. Impact of five blended learning models on the
Most of the surveys and systematic reviews on introductory learning experience of novice programmers have been
programming do not cover publication channels, curricula reviewed in this survey which are: flipped, mixed, flex,

125962 VOLUME 8, 2020

E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

supplemental and online practicing. Mixed model appeared • Research objectives

to be the most appropriate learning model according to this • Specifying research questions(RQs)
review for enhancing students’ performance. This review, • Organizing searches of databases
however, focused only of learning models not on curricula, • Studies selection
and assessment for IPC. • Screening relevant studies
There have been surveys on other aspects of programming • Data extraction
languages. One of these surveys is review of literature • Results synthesising
by [7] which investigated the factors that contribute to • Finalizing the review report
the difficulties faced by students in learning introductory It is important to formulate primary RQs (see Table 2) in
programming. Another survey on program visualization tools order to achieve following core objectives:
for learning introductory programming education is done
by [8]. However, none of them focus on channels publishing a) RQ1 attempts to address our objective to develop a
literature on IPC and problems faced by novice students library of articles related to the challenges faced by
novice while learning IPC focusing on curricula and
required to achieve desired solutions.
language choice, and make this dataset available to other
Our review distinguishes itself from the above reviews
by focusing only on the publication channels related to researchers. Moreover, to identify more significant work
IPC, closely examining the IPC curricula, and identifying that provides direction to investigate students’ problems
the approaches addressing challenges faced by novice in programming education. Answer of RQ1 will discover
students. Furthermore, we conduct a more balanced and gaps in terms of research approaches and challenges in
the state-of-the-art.
comprehensive approach than all of the above reviews: we
b) Another objective to identify complexities faced by
select tools in a systematic way following strict criteria, and
we code them using a predetermined labelling. novices based on the selection of any particular language
and curriculum while learning IPC will be achieved
during assessment of RQ2.
III. RESEARCH METHODOLOGY
c) The objective to classify existing solutions addressing
Guidelines for systematic reviews presented in software
problems in teaching/learning an IPC for novice pro-
engineering research by [9] and [10] are followed by our
grammers, and identify their solutions will be addressed
survey. Based on these guidelines, a search protocol was
by answer of RQ3.
listed after finalizing research questions to reduce possibility
d) RQ4 attempts to achieve another objective: to identify
of any research bias. According to this protocol, we have
effectiveness of existing tools developed for teaching
included three main stages in our research methodology: plan,
and learning IPC.
conduct and report of review, presented in next sections.
e) RQ5 tends to determine the recently developed method-
ologies to analyze and validate students’ performance in
learning IPC in terms of assessment tools, design and
students’ performance analysis.

B. REVIEW CONDUCT
The process of conducting this review has been articulated
in four steps presented below. In first step, relevant primary
studies have been searched from most commonly used
digital libraries. Selection of studies based on pre-defined
inclusion/exclusion criteria has been performed during sec-
ond step. We have designed quality assessment criteria to
further enhance quality of our review described in third step.
Backward snowballing is then performed to extract important
candidate papers during final fourth step.
FIGURE 1. Research strategy.

1) AUTOMATED SEARCH IN DIGITAL LIBRARIES

A systematic research has been carried out to filter irrele-
A. REVIEW PLAN vant studies and extract appropriate information. Therefore,
Adequate search strategy have been developed to find automatic and manual search techniques have been followed
all relevant studies. Figures1 and 2 show the research while exploring the search terms. Multiple digital libraries
methodology, which demonstrate search process for relevant were explored during this process, and only those repositories
publications, definition of a classification scheme, and have been selected for our search process that are commonly
mapping of publications. A highly structured process has accessed by other systematic literature surveys globally.
been followed in this review that involved: These widely used public venues happen to be relevant for

VOLUME 8, 2020 125963

 E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

FIGURE 2. Search strategy.

TABLE 2. Research Questions (RQs).

our SLR too, therefore considered. In addition google scholar, • Central and Eastern European Online Library (https://
that covers other such venues which are not directly explored, www.ceeol.com/)
also included in our survey. Therefore, following eleven • Google Scholar(https://scholar.google.com/)
digital venues possibly covering almost all relevant research The objective of manual search is to collect more literature
have been selected as primary search sources for automatic relevant to introductory programming education curriculum
search: and approaches domain. Extracted information can be
• ACM Digital Library (http://dl.acm.org) more relevant for limited search terms therefore following
• IEEE eXplore (http://ieeexplore.ieee.org) conditions were applied to limit our search terms:
• PLOS ONE (https://journals.plos.org/plosone/) • Based on formulated RQs, determine primary keywords.
• ScienceDirect (https://www.sciencedirect.com) • Identification of secondary keywords and synonyms for
• SpringerLink (https://link.springer.com/) additional keywords.
• WileyOnlineLibrary (https://onlinelibrary.wiley.com/) • ‘AND’ and ‘OR’ Boolean operators have been incorpo-
• arXiv (https://arxiv.org/search/cs) rated with keywords to develop a search string.
• AIS eLibrary (https://aisel.aisnet.org/) Possible arrangements of search string used can be noted
• IGI Global (https://www.igi-global.com/search/) from Figure 3, while a sample query has been presented

125964 VOLUME 8, 2020

E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

FIGURE 3. Search strings used to describe work included in our knowledge base.

2) SELECTION BASED ON INCLUSION/EXCLUSION CRITERIA

1) Inclusion criteria:
a) Papers included in review must be in the domain of
introductory Programming Education.
b) Papers must target the research questions.
Listing 1. Search query. c) Papers published in journals or conferences are
included in review.
d) Papers discussing IPC focusing on teaching or
in Listing 1. Primary keywords were selected as key learning tools. Example: Intelligent tutoring sys-
identifiers for research of programming education. Primary tem, board games etc.
keywords were chosen along with any of secondary or 2) Exclusion criteria:
additional keywords. Combination of keywords, Boolean
a) Remove papers written in non-english.
operators and wildcard have developed a final search query
b) Remove papers that do not discuss IPC in higher
mentioned as:
education.
Above search query appeared to be restrictive during
c) Remove the papers published before 2011.
initial search process. It was observed that above search
d) Remove papers discussing the IPC in school
string could not help inclusion of articles for only IPC
education.
curriculum.
e) Papers written by same research group in this
Table 3 enlists final search strings used to explore eleven
subject matter were removed (most recent was kept
digital libraries with specific applied filters by applying
in this case).
limited keywords. Only titles were searched for ACM
journals, ScienceDirect and PLOS ONE during automatic
search. Other digital libraries were explored with ‘‘all fields’’ 3) SELECTION BASED ON QUALITY ASSESSMENT
setting, as these do not allow a more specific search Selection of relevant studies on the basis of quality
configuration. Search string being too restrictive failed to assessment (QA) is considered as most important step for
find relevant articles for IGI Global and CEEOL journals, conducting any review. As the primary studies vary in design
therefore final search string designed for this library contains therefore quantitative, qualitative, and mixed-method critical
less number of keywords shown in Table 3. appraisal tool used by [11] and [12] are followed to perform

VOLUME 8, 2020 125965

 E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

TABLE 3. Search strategy for digital libraries.

TABLE 4. Possible ratings for recognized and stable publication source.

QA in our review. In order to enhance our study, we have ranking lists [14] into account. Possible scores for
carried out QA by designing a questionnaire to assess the publications from recognized and stable sources are
quality of selected papers. The QA of our study was carried shown in Table 4.
out by three authors and each study is scored based on the A final score has been calculated for each study after adding
following criteria: scores of above questions: (a number between 0 to 8). Articles
a) The study has awarded score (1) if it contributes towards achieving scores 4 or more have been included in finalized
three identified aspects in IPC, otherwise scored (0). results.
b) If clear solutions for identified challenges in IPC have
been provided by the study: ‘‘Yes (2)’’, ‘‘Limited (1)’’, 4) SELECTION BASED ON SNOWBALLING
and ‘‘No (0)’’ were the possible scores. After performing quality assessment, we conducted back-
c) Score (1) is awarded to studies which presents empirical ward snowballing [15] through reference list of each finalized
results otherwise scored (0). study to extract papers. Only those important candidate
d) The studies were rated by taking computer science papers are selected which passed through inclusion/exclusion
conference rankings [13], and the journal and country criteria. Once the paper is found, inclusion/exclusion of that

125966 VOLUME 8, 2020

E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

TABLE 5. Selection phases and results.

paper has been decided after reading its abstract and then number of papers while filtering/inspection phases helped
other parts of paper. After having examined selected papers reduce this number to 60 articles.
thoroughly we identified five more studies [16]–[19], [20]
and totally added up to 60 primary studies. IV. ASSESSMENT AND DISCUSSION
OF RESEARCH QUESTIONS
C. REVIEW REPORT In this section, we analyzed the finalized 60 primary studies
Overview of selected studies is provided in this section. based on our research questions.

A. ASSESSMENT OF RQ1: WHICH ARE RELEVANT

1) OVERVIEW OF INTERMEDIATE SELECTION
PUBLICATION CHANNELS FOR IPC RESEARCH? WHICH
PROCESS OUTCOME
CHANNEL TYPES AND GEOGRAPHICAL AREAS
IPC is an extremely active field, therefore our review
TARGETING IPC RESEARCH?
methodology had to empirically and systematically draw
The analysis of IPC learning tools, methods, contents,
relevant studies from all related digital libraries. The next
assessments, and language choice is a key challenge for
stage of our systematic review is to select the papers that will
researchers for the development of international standards for
form the knowledge base for this review. Around 500k papers
novice programmers. For this purpose, identification of high
are left after removing papers older than year 2011. However,
quality publication venues and scientometric analysis based
only one report published in year 2010 related to curriculum
on meta information in the domain of IPC is required. In this
has also been included in this review to build strong basis for
section, an insightful knowledge of publication sources,
curricula comparison.
types, year and geographical distribution, publication channel
After building a knowledge base from eleven (11) digital
wise distribution of selected studies for the analysis of IPC
libraries, authors examined title, abstract, and corresponding
research is presented.
full paper if required of each search result. Papers less than
After inspection phase, maximum of 20 studies have been
four pages long and irrelevant papers were eliminated in this
finalized from ACM digital library and nearly equal number
process.
is selected from IEEExplore as shown in Table 5, proving
To ascertain the relevance and contribution, in the field of
these publication sources as world’s largest professional
IPC, accepted publications have been read thoroughly during
societies publishing more than 50 scholarly peer-reviewed
inspection phase. To achieve the core goal of this study,
journals in dozens of computing and information technology
we build a systematic knowledge base of articles based on
disciplines.
their contributions.
Figure 4 represents the number of publications w.r.t. year of
publication. Maximum of 10 publications have been selected
2) OVERVIEW OF SELECTED STUDIES from year 2018 out of 60 showing more interest of researchers
Significant results of primary search, filtering and inspection towards development in teaching and learning IPC in that
phases, covering eleven digital libraries, are presented year. However, less interest towards IPC research has been
in Table 5. The automated search resulted in a very big observed in most recent year resulting less improvement in

VOLUME 8, 2020 125967

 E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

it is clearly noted that QA scores are in the range of

4-8 discarding studies scored less than 4. IPC developers
may find this QA helpful to choose related studies while
addressing IPC challenges. Studies published in Q1 journals
mostly scored highest while studies scoring total of 4 are
from less recognized journal but highly relevant to the subject
matter. Exclusion of studies from less stable publication
channels might miss any relevant studies, which in turn can
compromise the quality standards of this SLR, therefore have
been included. Most of such studies scored four (4) making
a total of 16 studies out of 60. Whereas, 15 out 60 studies
scored highest (i.e, an eight 8) showing all QA criteria met
by these studies to their maximum.
FIGURE 4. Introductory programming publications identified by our
Overall classification results and QA of finalized studies
search. have been presented in Table 6. Finalized studies have been
classified based on five factors: research type, empirical type
and methodology. We have categorized types for research
as: SLR, solution proposal, evaluation research, experience
paper, framework, review or curicula. Selection based on
these defined research types helped us to build taxonomy
presented in section V. All studies other then review papers
have been empirically validated their results by performing
surveys, statistical analysis, experiments or case studies
increasing their quality standards and awarded one score
each. Only 12 papers out of 60 have not presented empirical
FIGURE 5. Percentage of publication type.
results indicated by zero score for category (c) of quality
assessment criteria, including eight those studies which
are SLRs (no proposed methodology hence no need to be
empirically validated), other review papers. Only 17 papers
score zero for category (d) of quality assessment criteria, rest
of them score higher indicating competent sources. Four (4)
was the lowest score any study has been awarded. In addition,
only formally executed SLRs have been included in our
selection, methodologies adopted by selected studies other
than SLRs are also listed in Table 6.
Furthermore, we have identified sources of finalized stud-
ies, their publication channels and total number/percentage
of studies per publication source mentioned in Table 8.
FIGURE 6. Percentage of distribution of publications. About 14% of finalized studies have been published in
(Q1) journal named: ‘‘ACM Transaction on Computing
Education’’, whereas proceedings of ‘‘IEEE Frontier in
IPC curriculum/assessment analysis in relevance to students Education Conference’’ (Rank A) appeared to be second
and market needs. highest publication source contributing 8% into our selected
It is observed from Figure 5 that highest number of studies studies. In addition to that, Table 9 presents contribution and
have been selected from recognized journals and second approaches developed by such studies whose research types
highest number of studies have been picked from good rank are either ‘solution proposal’ or ‘framework’.
conferences, whereas few relevant studies have been selected
from workshops. Publication type for curriculum contents B. ASSESSMENT OF RQ2: WHAT ARE WORLD WIDE
included in our SLR has been categorized as report, which ACCEPTED STANDARDS FOR IPC CURRICULUM AND
are 3% of the total number. WHAT FACTORS AFFECT LANGUAGE CHOICE FOR IPC?
Figure 6 shows geographical distribution of selected Appropriate content and language choice are basic parame-
studies. It is observed that majority of publications i.e, ters while designing curriculum for IPC. These parameters
21 out of 60, have been published from different countries play an important role in teaching and learning IPC. Fol-
of America. lowing sub-sections describe existing studies in the domain
QA score for each finalized study awarded according of language choice and content selection for teaching and
to criteria defined in section III.B.3, shown in Table 7, learning IPC.

125968 VOLUME 8, 2020

E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

TABLE 6. Classification.

VOLUME 8, 2020 125969

 E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

TABLE 6. (Continued.) Classification.

TABLE 7. Quality assessment score. • platform independent

• ease of installation
• compiled strongly typed languages (More errors have
been captured at compile time in them therefore less
vulnerable to runtime failures)
• pedagogical benefits
Another study by [36] has also assessed impact of dif-
ferent programming languages on software standards and
developers productivity by performing statistical analysis on
development of projects in C and C++ languages. They
have concluded that use of C++ considered to produce
better software quality and minimize maintenance efforts as
1) LANGUAGE CHOICE compared to using C language. Furthermore, [50] reported
Many programming languages had been taught as IPC that, C# is rarely adopted as IPC in CS1 & CS2 courses
reported by one SLR [46] and several studies [24], [30], whereas often demanded in industry job ads. According to
[35], [51], [52], [56], [59], [60], [67], [69] to assess best same study, Java and C++ are also demanded by industry
choice for an introductory programming language. Various frequently and often taught in CS1 & CS2. Findings by
languages used as IPC have been evaluated with the help of another study [25] revealed that teaching of a syntactically
a comprehensive framework proposed by [4] and categorized simple language (Python) as IPC accelerate students’ under-
using technical and environmental feature sets. Based on this standing of programming concepts, in comparison to teaching
framework researchers can develop new IPCs as it provides a syntactically complex one (Java). In addition, significant
quantitative score computation for programming languages. improvement has been found in students’ performance
Studies suggest selection of a language for IPC is based on when problem solving has been taught before programming
the following factors: concepts. It is further noticed that choice of programming
• presence of important features (repetition structures and paradigm also helps in selecting programming language for
functional decomposition features) IPC. According to our assessment of selected studies while
• industry relevance and/or marketability to students investigating language choice and curriculum contents, that
• online help and accessibility of a community is elaborated well in Table 10, it is found that different
• availability of libraries and capability of being extended programming languages appeared as choice for an IPC

125970 VOLUME 8, 2020

E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

TABLE 8. Publication source.

based on multiple criterion. Both Java and Python have language in data science with strong supportive libraries.
been well regarded first programming languages. Whereby, However, it is clear from the Table 10 that it is hard to
dominance of Java language can be clearly noticed where clearly rate any programming language as the unanimous
institutions select programming language based on industry choice for first programming language. On the other hand,
relevance. the most comprehensive and reasonably generic criteria for
Whereas Python initially appeared as the first choice the selection of appropriate first programming language has
based on pedagogical benefits, but now it is a popular been presented by [4].

VOLUME 8, 2020 125971

 E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

TABLE 9. Solutions proposed by selected studies.

2) CONTENT According to (P27) of curriculum [2], students need to

Surveys/reports on content design while developing curricu- gain problem solving skills by applying knowledge they
lum of IPC have been reported by [1], [2], [18], [53] [49]. have learned more than just write code. They should to
New IPC curriculum CS0.5 has been developed and taught be able to develop and enhance a system based on its
in study [18] resulting improved pass rates of students in functionality, performance and usability while performing its
comparison to CS1. Curriculum developed in [53] and [49] quantitative and qualitative assessment. Much focus of intro-
clearly presented topic wise content comparison of all ductory programming courses on ‘‘Software Development
volumes. Fundamentals’’ topics along with a subset of ‘‘Programming

125972 VOLUME 8, 2020

E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

TABLE 10. Curriculum: Findings of the reviewed studies.

VOLUME 8, 2020 125973

 E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

Languages’’ topics (P31). Some courses use functional pro- Results from these studies highlight significance to measure
gramming while others adopt object-oriented programming the effect of curricular changes.
or platform-based development. Findings and evaluation
methods of other studies reviewing curriculum presented 2) LEARNING
in Table 10 indicating there are only few studies present at the Existing studies investigated learning requirements and
moment addressing standards for programming objectives for pointed out need of basic reasoning and mental skills for
majors and non-majors, topic level comparison and program learning IPC. Surveys/literature targeting learning an IPC
by design curriculum. According to the reviewed studies it through methods/models, notional machine and games are
is concluded that ACM/IEEE CS Curriculum considered as evaluated as follows:
widely accepted standard.
i) Methods/Models: Different types and models of learn-
C. ASSESSMENT OF RQ3: WHICH TEACHING AND ing styles have been reviewed in an SLR [37] conclud-
LEARNING APPROACHES HAVE BEEN REPORTED TO ing learning style can not be used as an instrument to
ADDRESS PROBLEMS FOR NOVICE STUDENTS? predict student success. Students have been introduced
This systematic study aims to examine the existing knowl- to programming using different introductory program-
edge in IPC from finalized 60 papers. In order to gain the ming languages in another study [29] and performance
overview, we investigate teachers’ and students’ perspective has been analyzed statistically. Another analysis of
of IPC trends and approaches to address novice problems. programming language education has been conducted
Teaching/pedagogy an IPC refers to the techniques, tools, by a study [34] for a pre-introductory CS course.
approaches and methods developed for teaching program- According to the results of this survey, students rec-
ming in an effective way. These developments in teaching ognized that education programming language based
methodologies contributed in finding solutions for a lot of course framework helped their understanding of basic
challenges related to the nature of programming. Whereas, programming concepts and algorithm design. Objec-
learning/cognition requirements is the students’ ability to tive of both of these studies is to possibly improve the
think and act in their academic life which help them acquire selection of suitable programming language for novice
programming skills and overcome difficulties in learning by programmers. Another recent study [70], explores the
adopting any specific model, notional machine or games. effect of incremental mindsets in individuals and found
Teaching and learning an IPC by applying pedagogical an increases in effort during programming activities but
skills or through tools are summarized and discussed in this not in performance.
section. Pedagogy enables instructors to impart knowledge ii) Notional Machine:
and skills in such ways where students understand, remember An informal literature review to examine knowledge
and apply these skills. In our SLR, pedagogy is divided into bodies in conceptual relation to a ‘‘notional machine’’
two sub-levels i.e, teaching and learning. Existing surveys has been presented in study [39]. Notional machine
targeting teaching and learning IPC are evaluated in the serves as an abstract computer for the execution of
following sub-sections: particular kind of programs and fulfills the purpose
of understanding program execution stages. One or
1) TEACHING more programming languages/paradigms with one
Impact of various teaching approaches on students’ perfor- specific programming environment are possibly asso-
mance for IPC has been surveyed by several studies [16], [20], ciated with notional machine. several aspects on the
[27], [31], [38], [42], [45], [47], [64], [65], [70]. Different functionality of notional machines in introductory
pedagogical approaches are compared in terms of passing programming education (IPE) have been discussed
rate percentages by [27] where as some pedagogical and together in this study that concludes notional machines
motivational strategies have been surveyed in [31], [38], [65] as a prominent challenge in IPE.
to promote students’ motivation. Significant factors to predict iii) Games/Robots:
students’ attitude towards computer programming have been A game-based learning (GBL) model is presented by
identified as three variables i.e, (performance in program- [41] and [68] developed a robot-based programming
ming courses, perceived learning and programming self- tool. Major contributions and approaches used for
efficacy) by [20]. ‘‘Action research insights and outcomes the developments in domain of IPC are presented
ADRI approach’’, ‘‘active learning technique’’, ‘‘innovative in Table 9.
programming environment approach’’, ‘‘models for teaching Table 11 elaborates the criterion, evaluation methods,
programming’’ and ‘‘program by design approach’’ are participant details and findings of selected studies addressing
various approaches discussed, compared and validated in IPC teaching and learning methods and techniques. Four
studies [16], [42], [45], [47], [64]. Another study [71] evaluation criterion were identified for teaching aspect during
proposed agile process for teaching an IPC and concluded review that includes seven studies. These studies looked
positive impact on students’ learning. These studies examined at factors effecting students’ learning, pedagogical and
various factors that contribute to students’ failure in IPC. motivational strategies, use of active learning techniques and

125974 VOLUME 8, 2020

E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

TABLE 11. Teaching and Learning: Findings of the reviewed studies.

application based teaching techniques. Approaches devel- D. ASSESSMENT OF RQ4: HOW EFFECTIVE IS TEACHING
oped considering pedagogical and motivational strategies IPC THROUGH TOOLS IN IMPROVING THE LEARNING
appeared to be more effective towards students’ learning and EXPERIENCE OF NOVICE PROGRAMMERS?
programming behaviour. On the other hand, active learning Teaching and learning an IPC through tools is divided into
and application-based teaching techniques showed promising following categories:
results when used during an IPC. It is important to note
that four studies contributed in students’ cognitive level 1) IMPLEMENTED TOOLS
meeting two evaluation criterion. Findings of these studies GUI based tools have been developed for teaching and
suggesting adoption of such learning models that consider the learning an IPC and empirically validated in studies [22],
characteristics of the learners. [58], [63] with the aim of improving problem solving ability

VOLUME 8, 2020 125975

 E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

of novice programmers. These tools allow the programmers the course. Repeaters favor fixed programming aptitude and
to interactively develop executable flow charts. Introduction have lower programming efficacy which increases further
of abstract programming concepts visually allow students possibility of failure.
to make transition itno a complex programming language
easier, concluding visual tools are relevant in attaining better
2) ASSESSMENT TOOL
results.
A web-based students’ response platform has been developed
2) FRAMEWORK
in study [57] for instructors to ask free-form short response
questions to students, while impact of different teaching
A development framework has been designed by [54], called
approaches and languages on students’ learning an IPC
Splashkit, for motivating and engaging students in introduc-
has been assessed by study [26] using an assessment tool.
tory programming for tertiary education. This is an open-
objective of both of the studies is to assess teaching
source, cross-platform development and language-agnostic
and learning an IPC. Programming language learning is
framework that supports a wide range of programming
assessed in another study [23] based on peer code review
education approaches. Objects-first/ objects-later curriculum
model EduPCR2. The EduPCR system and its PCR-based
design, a range of fully-featured game engines like pro-
assessment process have significantly improved student
gramming tools, databases, programming language choices,
learning outcomes in many areas including programming
and web-servers together with easy API design tech-
skills.
niques are the basic programming education approaches
Custom-based assessment tools, self-assessment tools and
included in this framework. Objective of this study to
tools on giving student feedback have been reviewed in
empower students for developments of important software
another recent SLR [38]. This includes ProgTest, UML
engineering skill that is reading API documentation. Stu-
testing, industrial based testing suite, Bluefix, NoobLab and
dents were able to design games when taught using this
PRAISE. Another recent study [72] build a comprehensive
framework.
automated programming assessment system named: Edgar,
As can be seen in Table 11 four studies have been
that deals with various programming languages and can be
categorized addressing learning of an IPC using tools based
deployed on all major operating systems.
on two evaluation criterion. Proposed approaches proved to
be effective compared to traditional learning styles as these
helped students for problem solving improvement and to 3) ASSESSMENT DESIGN
develop a wide range of applications. It is concluded that Various factors have been reviewed in studies [17] [55] to
teaching with SplashKit considered as more beneficial as improve student academic self-efficacy and problem solving
this framework is empirically evaluated on large scale of and in learning programming by applying Spearmans’ rank
participants for several years. correlation coefficient technique. It was identified in [55]
that performance in formative assessment and problem-
E. ASSESSMENT OF RQ5: WHICH METHODOLOGIES HAVE solving-skill are weakly correlated. These studies aim to
BEEN ADOPTED TO ANALYZE AND VALIDATE discover correlation between various assessment method,
PERFORMANCE OF NOVICE PROGRAMMERS? student’s participation and their final performance.
Surveys performed to assess teaching and learning process of Table 12 further elaborates findings and evaluation
IPC are categorized into following sub-levels. methods of each reviewed study which are evaluated on
criterion like: students’ learning and programming behaviour,
1) STUDENTS’ PERFORMANCE ANALYSIS auto recognition of required features in students’ code,
Several methodologies have been adopted to analyze and students’ course performance, students’ programming errors
validate students’ performance in IPC by various studies investigation and correlation among perceived and academic
[19], [21], [28], [32], [33], [40], [43], [44], [48], [61], [62], performance.
[66]. Objective of these studies is to predict factors effecting First most addressed evaluation criteria was students’
students’ performance in an IPC by observing programming course performance, which was used to evaluate nine studies.
errors made by students using multiple testing techniques: Most of these studies gather data through assessment which
ADRI model based on six categories of Blooms’ Taxonomy, is then statistically analyzed. These studies attempt to
regression and t-tests, formative and summative approach, predict students’ pass rates as well as evaluated different
and bayesian network classifier. Few studies suggested that factors effecting students’ performance. It is concluded that
for many low-performance students learning fails due to such assessment tasks/tools considered more effective which
cognitive overload, while others identified students’ main tend to assess students’ problem solving skills and logical
problem was to divide activity into functions and classes errors in their code. Moreover, considering factors such
and to find errors in ones’ own programs. One survey as: structural design of assessment, time management, auto
observed that implicit theories of programming-aptitude feature detection and balanced weights of labs, quizzes and,
and programming-efficacy are interrelated and positively midterm/final exams may significantly improve methodolo-
correlated with effort, performance, and previous failures in gies for assessing students’ performance in an IPC.

125976 VOLUME 8, 2020

E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

TABLE 12. Assessment: Findings of the reviewed studies.

V. DISCUSSION AND FUTURE DIRECTIONS B. GENERAL OBSERVATIONS AND FUTURE DIRECTION

This section summarizes and discusses the results related to Several observations possibly can be made in the findings
the systematic literature review. of this SLR. Various RQs were developed in order to
determine methods/approaches/tools while teaching an IPC
A. TAXONOMIC HIERARCHY
and to provide an exclusive overview of subject matter. Many
The goal of this systematic literature review was to examine
trends and findings can be noted as a result in regards to
the current knowledge in IPC by selecting 60 papers.
the challenges for teaching and learning IPC. These include
To achieve this goal, We build taxonomic hierarchy of
following observations together with future directions:
selected studies shown in Figure 7, excluding those stud-
ies (SLRs) which have not validated their methodologies
empirically. We have investigated developments and chal- 1) CURRICULA
lenges for the aspects such as: IPC curriculum, teaching and The motivational factor for big proportion of selected
learning and assessment. However, these aspects are then studies (33% ) involves designing curriculum contents and
further divided into many sub-levels showing depth of each selection of appropriate programming language for IPC.
aspect and their role in the performance improvement of Early programming curricula attempts focused mainly on
novice programmers. the relatively simple challenge. Various factors for the

VOLUME 8, 2020 125977

 E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

FIGURE 7. Taxonomy of introductory programming education perspectives.

selection of language for IPC have been discussed by many 3) ASSESSMENT

researchers, concluding that using a syntactically simple The vast majority of literature assumed that forma-
language facilitated student’s learning of programming tive/summative assessment approaches are enough to
concepts. Therefore, Python as first programming language measure student’s performance. Little has been done to focus
considered as first choice of instructors, whereas, Java on assessment design. Code conventions are important to
leads where industry relevancy becomes core requirement. programmers to improve the readability and maintainability
Curriculum developers, on the other hand, have been focused of code which needs to be assessed while measuring perfor-
on the development of course contents for computer science mance of novice programmers. We have noticed a language
as major. Little has been done to tackle challenges in IPC dependant web-based assessment tool [23] that assesses
course contents for non-majors. Moreover, Lack of curricula students’ performance for object oriented programming in
standardization appears as a limitation as the industry cannot Java course and C programming for freshmen. Whereas,
rely on teachers’ subjective attitude towards programming. a language-independent assessment tool [26] has been found
Students might be able to learn IPC well if they have been during this review which assesses students’ grasp of all
taught IPC starting from school years instead of graduation fundamental and object-oriented concepts. Another domain
level and by including algorithm visualization tools in independent practical system designed in [57] for collecting
curricula. In addition, coding standards and year wise focus and labeling student responses to open ended questions has
of each study must be considered while developing IPC also been identified by our study. Associated rubric must be
curricula in future. included while designing assessments containing functional
correctness scheme, technical correctness scheme, high
2) TEACHING AND LEARNING
quality code sample files and code convention restrictions.
A large number of publications that we have explored
Tester files must also be included while assessing novice
focus on methods/tools designed for teaching and learning
programmers. To enhance learning an IPC, web based
IPC. Where most of the work has clearly help students
assessments and strong assessments need to be developed.
learn programming through building tools. Comparatively
less attention has been paid to analyzing and evaluating
those learning tools, according to our observations. No/less
distribution of these learning tools beyond the institution VI. CONCLUSION
where they were designed also been observed. Students This study has been conducted to build an understanding
can improve their problem solving skills and learn C++ of research trends in the field of Introductory Programming
programming language using tool developed in [22] whereas Education. To ensure thorough coverage of challenges
multiple programming languages are supported by another and their solutions we followed a systematic literature
recent learning tool: SplashKit [54]. It is further noticed that review. We have performed search using as many terms
web-based programming learning tools were developed in as we know associated with IPC then results have been
early years while trend recently shifts towards serious games evaluated accordingly. We concluded our search in June 2020,
using mobile phones.. In future, more modern tools should be so studies conducted after that date would have not been
developed for learning IPC such as serious games. included. Eleven main digital repositories were explored for

125978 VOLUME 8, 2020

E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

this search and around 60 articles appeared to contribute even [13] (2018). CORE Conference Portal. Accessed: Jan. 6, 2020. [Online].
better students’ performance out of 500k searched articles. Available: http://portal.core.edu.au/conf-ranks/
[14] (2018). Scimago Journal & Country Rank. Accessed: Jan. 6, 2020.
The results show that more than half of the selected [Online]. Available: https://www.scimagojr.com/
studies appeared in recognized journals and only a few had [15] C. Wohlin, ‘‘Guidelines for snowballing in systematic literature studies
published in workshops or as reports. ‘‘Solution proposal’’ and a replication in software engineering,’’ in Proc. 18th Int. Conf. Eval.
Assessment Softw. Eng. EASE, 2014, p. 38.
and ‘‘evaluation research’’ have been reported as two main [16] V. Aleksić and M. Ivanović, ‘‘Introductory programming subject in
research types adopted by these studies. Majority of selected European higher education,’’ Informat. Edu., vol. 15, no. 2, pp. 163–182,
studies were evidence based and possibly helping the 2016.
[17] R. A. Alturki, ‘‘Measuring and improving student performance in an
instructors to gain maximum benefits of IPC teaching and introductory programming course,’’ Informat. Edu., vol. 15, no. 2,
learning. Pedagogy, language choice and students’ perfor- pp. 183–204, Nov. 2016.
mance analysis were found as most frequently addressed [18] J. Q. Dawson, M. Allen, A. Campbell, and A. Valair, ‘‘Designing an
introductory programming course to improve non-Majors’ experiences,’’
aspects of IPC, whereas curricula contents, assessment in Proc. 49th ACM Tech. Symp. Comput. Sci. Edu., Feb. 2018, pp. 26–31.
tool/design and teaching/learning through tools have been [19] E. D. Canedo, G. A. Santos, and L. L. Leite, ‘‘An assessment of the
appeared as less addressed aspects of IPC. teaching-learning methodologies used in the introductory programming
courses at a brazilian university,’’ Informat. Edu., vol. 17, no. 1, pp. 45–59,
Limitations of any SLR are mainly related to search strat- Apr. 2018.
egy, inaccurate extracted data or misclassification. However, [20] M. D. Gurer, I. Cetin, and E. Top, ‘‘Factors affecting students’ attitudes
our search strategy reduced the risk of selection bias by con- toward computer programming,’’ Informat. Edu., vol. 18, no. 2, p. 281,
2019.
ducting this study with different keywords from all common [21] U. Nikula, O. Gotel, and J. Kasurinen, ‘‘A motivation guided holistic
digital repositories. By applying rigorous inclusion/exclusion rehabilitation of the first programming course,’’ ACM Trans. Comput. Edu.,
criteria and asking two independent reviewers for assessment vol. 11, no. 4, p. 24, 2011.
[22] D. Hooshyar, R. B. Ahmad, M. Yousefi, F. D. Yusop, and S.-J. Horng,
of all extractions, other two risks were addressed. ‘‘A flowchart-based intelligent tutoring system for improving problem-
For future research on IPC, more attention should be paid solving skills of novice programmers,’’ J. Comput. Assist. Learn., vol. 31,
to curricula design for non-majors, tools design; particularly no. 4, pp. 345–361, Aug. 2015.
[23] Y. Wang, H. Li, Y. Feng, Y. Jiang, and Y. Liu, ‘‘Assessment of programming
serious games, and web-based assessment design with rubric. language learning based on peer code review model: Implementation and
More evaluation research should be carried out in order to experience report,’’ Comput. Edu., vol. 59, no. 2, pp. 412–422, Sep. 2012.
evaluate existing IPC curriculum contents. [24] A. Stefik and S. Siebert, ‘‘An empirical investigation into programming
language syntax,’’ ACM Trans. Comput. Edu., vol. 13, no. 4, p. 19, 2013.
[25] T. Koulouri, S. Lauria, and R. D. Macredie, ‘‘Teaching introductory
REFERENCES programming: A quantitative evaluation of different approaches,’’ ACM
[1] H. Topi, J. S. Valacich, R. T. Wright, K. M. Kaiser, J. Nunamaker Jr, Trans. Comput. Edu., vol. 14, no. 4, p. 26, 2015.
J. C. Sipior, and G. De Vreede, ‘‘Curriculum guidelines for undergraduate [26] W. M. Kunkle and R. B. Allen, ‘‘The impact of different teaching
degree programs in information systems,’’ in Proc. ACM/AIS Task Force, approaches and languages on student learning of introductory program-
2010, pp. 56–57. ming concepts,’’ ACM Trans. Comput. Edu., vol. 16, no. 1, pp. 1–26,
[2] M. Sahami et al., Computer Science Curricula 2013: Curriculum Feb. 2016.
Guidelines for Undergraduate Degree Programs in Computer Science. [27] G. Silva-Maceda, P. D. Arjona-Villicana, and F. E. Castillo-Barrera, ‘‘More
New York, NY, USA: Association for Computing Machinery, 2013. time or better tools? A large-scale retrospective comparison of pedagogical
[3] R. P. Medeiros, G. L. Ramalho, and T. P. Falc ao, ‘‘A systematic literature approaches to teach programming,’’ IEEE Trans. Educ., vol. 59, no. 4,
review on teaching and learning introductory programming in higher pp. 274–281, Nov. 2016.
education,’’ IEEE Trans. Educ., vol. 62, no. 2, pp. 77–90, May 2019. [28] M. J. Scott and G. Ghinea, ‘‘On the domain-specificity of mindsets: The
[4] M. S. Farooq, S. A. Khan, F. Ahmad, S. Islam, and A. Abid, ‘‘An evaluation relationship between aptitude beliefs and programming practice,’’ IEEE
framework and comparative analysis of the widely used first programming Trans. Educ., vol. 57, no. 3, pp. 169–174, Aug. 2014.
languages,’’ PLoS ONE, vol. 9, no. 2, Feb. 2014, Art. no. e88941. [29] S. Xinogalos, T. Pitner, M. Ivanović, and M. Savić, ‘‘Students’ perspective
[5] L. Major, T. Kyriacou, and O. P. Brereton, ‘‘Systematic literature review: on the first programming language: C-like or pascal-like languages?’’ Edu.
Teaching novices programming using robots,’’ IET Softw., vol. 6, no. 6, Inf. Technol., vol. 23, no. 1, pp. 287–302, Jan. 2018.
pp. 502–513, 2012. [30] S. Nanz and C. A. Furia, ‘‘A comparative study of programming languages
[6] A. Alammary, ‘‘Blended learning models for introductory programming in rosetta code,’’ in Proc. IEEE/ACM 37th IEEE Int. Conf. Softw. Eng.,
courses: A systematic review,’’ PLoS ONE, vol. 14, no. 9, Sep. 2019, May 2015, pp. 778–788.
Art. no. e0221765. [31] A. Gomes and A. Mendes, ‘‘A teacher’s view about introductory program-
[7] Y. Qian and J. Lehman, ‘‘Students’ misconceptions and other difficulties ming teaching and learning: Difficulties, strategies and motivations,’’ in
in introductory programming: A literature review,’’ ACM Trans. Comput. Proc. IEEE Frontiers Edu. Conf. (FIE) Proc., Oct. 2014, pp. 1–8.
Edu., vol. 18, no. 1, pp. 1–24, Dec. 2017. [32] D. McCall and M. Kolling, ‘‘Meaningful categorisation of novice
[8] J. Sorva, V. Karavirta, and L. Malmi, ‘‘A review of generic program programmer errors,’’ in Proc. IEEE Frontiers Edu. Conf. (FIE), Oct. 2014,
visualization systems for introductory programming education,’’ ACM pp. 1–8.
Trans. Comput. Edu., vol. 13, no. 4, p. 15, 2013. [33] F. B. Tek, K. S. Benli, and E. Deveci, ‘‘Implicit theories and self-efficacy in
[9] P. Brereton, B. A. Kitchenham, D. Budgen, M. Turner, and M. Khalil, an introductory programming course,’’ IEEE Trans. Educ., vol. 61, no. 3,
‘‘Lessons from applying the systematic literature review process within the pp. 218–225, Aug. 2018.
software engineering domain,’’ J. Syst. Softw., vol. 80, no. 4, pp. 571–583, [34] I. Yoon, J. Kim, and W. Lee, ‘‘The analysis and application of an
Apr. 2007. educational programming language (RUR-PLE) for a pre-introductory
[10] B. Kitchenham, Procedures for Performing Systematic Reviews, vol. 33. computer science course,’’ Cluster Comput., vol. 19, no. 1, pp. 529–546,
Keele, U.K.: Keele Univ., 2004, pp. 1–26. Mar. 2016.
[11] A. Fernandez, E. Insfran, and S. Abrahāo, ‘‘Usability evaluation methods [35] O. Ezenwoye, ‘‘What language?—The choice of an introductory program-
for the Web: A systematic mapping study,’’ Inf. Softw. Technol., vol. 53, ming language,’’ in Proc. IEEE Frontiers Edu. Conf. (FIE), Oct. 2018,
no. 8, pp. 789–817, Aug. 2011. pp. 1–8.
[12] S. Ouhbi, A. Idri, J. L. Fernández-Alemán, and A. Toval, ‘‘Requirements [36] P. Bhattacharya and I. Neamtiu, ‘‘Assessing programming language impact
engineering education: A systematic mapping study,’’ Requirements Eng., on development and maintenance: A study on c and c++,’’ in Proc. 33rd
vol. 20, no. 2, pp. 119–138, Jun. 2015. Int. Conf. Softw. Eng. ICSE, 2011, pp. 171–180.

VOLUME 8, 2020 125979

 E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

[37] M. Carelli Oliveira Maia, D. Serey, and J. Figueiredo, ‘‘Learning styles [59] M. Ateeq, H. Habib, A. Umer, and M. U. Rehman, ‘‘C++ or Python? Which
in programming education: A systematic mapping study,’’ in Proc. IEEE one to begin with: A Learner’s perspective,’’ in Proc. Int. Conf. Teach.
Frontiers Edu. Conf. (FIE), Oct. 2017, pp. 1–7. Learn. Comput. Eng., Apr. 2014, pp. 64–69.
[38] A. Luxton-Reilly, J. Sheard, C. Szabo, Simon, I. Albluwi, B. A. Becker, [60] D. Krpan and I. Bilobrk, ‘‘Introductory programming languages in higher
M. Giannakos, A. N. Kumar, L. Ott, J. Paterson, and M. J. Scott, education,’’ in Proc. 34th Int. Conv. MIPRO, May 2011, pp. 1331–1336.
‘‘Introductory programming: A systematic literature review,’’ in Proc. [61] M. Corney, R. Lister, and D. Teague, ‘‘Early relational reasoning and the
Companion 23rd Annu. ACM Conf. Innov. Technol. Comput. Sci. Edu. novice programmer: Swapping as the hello world of relational reasoning,’’
ITiCSE Companion, 2018, pp. 55–106. in Proc. 13th Australas. Comput. Edu. Conf., Darlinghurst, Australia:
[39] J. Sorva, ‘‘Notional machines and introductory programming education,’’ Australian Computer Society, vol. 114, 2011, pp. 95–104.
ACM Trans. Comput. Edu., vol. 13, no. 2, pp. 1–31, Jun. 2013. [62] R. Mason and G. Cooper, ‘‘Why the bottom 10% just can’t do it: Mental
[40] K. Benda, A. Bruckman, and M. Guzdial, ‘‘When life and learning do not effort measures and implication for introductory programming courses,’’ in
fit: Challenges of workload and communication in introductory computer Proc. 14th Austral. Comput. Edu. Conf. (ACE), Melbourne, VIC, Australia,
science online,’’ ACM Trans. Comput. Edu., vol. 12, no. 4, p. 15, 2012. vol. 123, 2012, pp. 187–196.
[41] C. Kazimoglu, M. Kiernan, L. Bacon, and L. MacKinnon, ‘‘Understanding [63] E. de Jesus, ‘‘Teaching computer programming with structured program-
computational thinking before programming: Developing guidelines for ming language and flowcharts,’’ in Proc. Workshop Open Source Design
the design of games to learn introductory programming through game- Commun. OSDOC, 2011, pp. 45–48.
play,’’ Int. J. Game-Based Learn., vol. 1, no. 3, pp. 30–52, 2011. [64] V. K. Proulx, ‘‘Introductory computing: The design discipline,’’ in Proc.
[42] S. I. Malik, ‘‘Improvements in introductory programming course: Action Int. Conf. Informat. Schools, Situation, Evol., Perspect. Springer, 2011,
research insights and outcomes,’’ Syst. Pract. Action Res., vol. 31, no. 6, pp. 177–188.
pp. 637–656, Dec. 2018. [65] G. Venugopal-Wairagade, ‘‘Study of a pedagogy adopted to generate
[43] N. C. C. Brown and A. Altadmri, ‘‘Investigating novice programming interest in students taking a programming course,’’ in Proc. Int. Conf.
mistakes: Educator beliefs vs. Student data,’’ in Proc. 10th Annu. Conf. Learn. Teach. Comput. Eng. (LaTICE), Mar. 2016, pp. 141–146.
Int. Comput. Edu. Res. ICER, 2014, pp. 43–50. [66] Á. Matthíasdóttir and H. J. Geirsson, ‘‘The novice problem in computer
[44] A. Vihavainen, ‘‘Predicting Students’ performance in an introductory science,’’ in Proc. 12th Int. Conf. Comput. Syst. Technol. CompSysTech,
programming course using data from Students’ own programming 2011, pp. 570–576.
process,’’ in Proc. IEEE 13th Int. Conf. Adv. Learn. Technol., Jul. 2013, [67] E. Murphy, T. Crick, and J. H. Davenport, ‘‘An analysis of introductory
pp. 498–499. programming courses at UK universities,’’ 2016, arXiv:1609.06622.
[45] S. Dasuki and A. Quaye, ‘‘Undergraduate Students’ failure in program- [Online]. Available: http://arxiv.org/abs/1609.06622
ming courses in institutions of higher education in developing countries: A [68] P. Bachiller-Burgos, I. Barbecho, L. V. Calderita, P. Bustos, and
nigerian perspective,’’ Electron. J. Inf. Syst. Developing Countries, vol. 76, L. J. Manso, ‘‘LearnBlock: A robot-agnostic educational programming
no. 1, pp. 1–18, Sep. 2016. tool,’’ IEEE Access, vol. 8, pp. 30012–30026, 2020.
[46] T. Crow, A. Luxton-Reilly, and B. Wuensche, ‘‘Intelligent tutoring systems [69] M. S. Farooq, S. A. Khan, and A. Abid, ‘‘A framework for the assessment
for programming education: A systematic review,’’ in Proc. 20th Australas. of a first programming language,’’ J. Basic Appl. Sci. Res., vol. 2, no. 8,
Comput. Edu. Conf. ACE, 2018, pp. 53–62. pp. 8144–8149, 2012.
[47] J. L. Duffany, ‘‘Application of active learning techniques to the teaching of [70] J. G. C. Rangel, M. King, and K. Muldner, ‘‘An incremental mindset
introductory programming,’’ IEEE Revista Iberoamericana de Tecnologias intervention increases effort during programming activities but not
del Aprendizaje, vol. 12, no. 1, pp. 62–69, Feb. 2017. performance,’’ ACM Trans. Comput. Edu., vol. 20, no. 2, pp. 1–18,
[48] S. I. Malik, ‘‘Assessing the teaching and learning process of an introductory May 2020.
programming course with Bloom’s taxonomy and assurance of learning [71] B. Isong, ‘‘A methodology for teaching computer programming: First year
(AOL),’’ Int. J. Inf. Commun. Technol. Edu., vol. 15, no. 2, pp. 130–145, students’ perspective,’’ Int. J. Modern Edu. Comput. Sci., vol. 6, no. 9,
Apr. 2019. p. 15, 2014.
[49] M. Sabin et al., Information Technology Curricula 2017: Curriculum [72] I. Mekterovic, L. Brkic, B. Milasinovic, and M. Baranovic, ‘‘Building
Guidelines for Baccalaureate Degree Programs in Information Technol- a comprehensive automated programming assessment system,’’ IEEE
ogy. New York, NY, USA: Association for Computing Machinery, 2017. Access, vol. 8, pp. 81154–81172, 2020.
[50] J. Polack-Wahl, S. Davies, and K. Anewalt, ‘‘A snapshot of current
languages used in industry,’’ in Proc. Frontiers Edu. Conf. Proc., Oct. 2012,
pp. 1–6.
[51] W. Farag, S. Ali, and D. Deb, ‘‘Does language choice influence the
effectiveness of online introductory programming courses?’’ in Proc. 13th
Annu. ACM SIGITE Conf. Inf. Technol. Edu. - SIGITE, 2013, pp. 165–170.
[52] R. Mason, G. Cooper, and M. de Raadt, ‘‘Trends in introductory
programming courses in australian universities: Languages, environments
and pedagogy,’’ in Proc. 14th Australas. Comput. Edu. Conf., Darlinghurst,
Australia, Australian Computer Society, vol. 123, 2012, pp. 33–42.
[53] L. Marshall, ‘‘A topic-level comparison of the ACM/IEEE CS curriculum
volumes,’’ in Proc. Annu. Conf. Southern Afr. Comput. Lecturers’ Assoc.
Springer, 2017, pp. 309–324.
[54] J. Renzella, A. Cummaudo, A. Cain, J. Grundy, and J. Meyers,
‘‘SplashKit: A development framework for motivating and engaging
students in introductory programming,’’ in Proc. IEEE Int. Conf. Teaching,
Assessment, Learn. Eng. (TALE), Dec. 2018, pp. 40–47. ERUM MEHMOOD was born in Pakistan. She
[55] A. K. Veerasamy, D. D’Souza, R. Lindén, and M.-J. Laakso, ‘‘Relationship received the M.Phil. degree in computer science
between perceived problem-solving skills and academic performance of from NCBA&E, Lahore, Pakistan, in 2017. She
novice learners in introductory programming courses,’’ J. Comput. Assist.
is currently pursuing the Ph.D. degree with
Learn., vol. 35, no. 2, pp. 246–255, Apr. 2019.
the University of Management and Technology,
[56] J. Heliotis and R. Zanibbi, ‘‘Moving away from programming and towards
Lahore.
computer science in the cs first year,’’ J. Comput. Sci. Colleges, vol. 26,
no. 3, pp. 115–125, 2011. She is also working as a Lecturer of computer
[57] C. Heiner, ‘‘Mining student responses to learn answer models: A case study science with the Government Degree College
using data from an introductory programming course,’’ J. Comput. Sci. Lahore, Pakistan. Her research interests include
Colleges, vol. 29, no. 2, pp. 17–25, 2013. programming language education and design, big
[58] D. Gudmundsen, L. Olivieri, and N. Sarawagi, ‘‘Reducing the learning data analytics, stream processing, and ETL and real-time data warehousing.
curve in an introductory programming course using visual logic,’’ J. Her M.Phil. dissertation is in the area of stream processing for real-time data
Comput. Sci. Colleges, vol. 27, no. 6, pp. 10–12, 2012. warehousing.

125980 VOLUME 8, 2020

E. Mehmood et al.: Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course

ADNAN ABID (Member, IEEE) was born in NAEEM A. NAWAZ received the B.Sc. degree
Gujranwala, Pakistan, in 1979. He received the from the University of Punjab, Pakistan, in 1997,
B.S. degree from the National University of Com- the M.Sc. degree in computer sciences from
puter and Emerging Science, Pakistan, in 2001, Hamdard University, Pakistan, in 2000, the M.S.
the M.S. degree in information technology from degree in computer engineering from Mid Sweden
the National University of Science and Technol- University, Sweden, in 2008, and the Ph.D. degree
ogy, Pakistan, in 2007, and the Ph.D. degree from International Islamic University Malaysia,
in computer science from the Politecnico Di in 2018. He received different diploma (DCS) and
Milano, Italy, in 2012. He spent one year in certifications. He is currently a Lecturer with the
EPFL, Switzerland, to complete his M.S. thesis. Department of Computer Science (CFY), Umm
He is currently an Associate Professor with the Department of Computer Al-Qura University, Mecca. He has published research articles in renowned
Science, University of Management and Technology, Pakistan. He has almost journals and conferences. He is also serving as a reviewer for many journals,
70 publications in different international journals and conferences. He has conferences, and books. His teaching and research interests include WSN,
served as a reviewer in many international conferences and journals. His the IoT, crowd management, computer networks, and programing language.
research interests include computer science education, information retrieval,
and data management. He is a member of the IEEE Education Society and
the IEEE Education Society. He is also an Associate Editor of IEEE ACCESS
journal.

MUHAMMAD SHOAIB FAROOQ (Member,

IEEE) was born in Lahore, Pakistan. He received
the M.Sc. degree from Quaid e Azam University,
Pakistan, in 1995, the M.S. degree in computer
science from the Government College University,
in 2007, and the Ph.D. degree from Abdul Wali
Khan University, Pakistan, in 2015. He possesses
more than 23 years of teaching experience in
the field of computer science. He is currently
an Associate Professor with the Department of
Computer Science, University of Management and Technology, Pakistan.
His research interests include theory of programming languages, big data,
the IoT, and computer science education. He is a member of the IEEE
Systems, Man, and Cybernetics Society.

VOLUME 8, 2020 125981