Accepted Manuscript

A validity and reliability study of the Computational Thinking Scales (CTS)

Özgen Korkmaz, Recep Çakir, M. Yaşar Özden

PII: S0747-5632(17)30005-5

DOI: 10.1016/j.chb.2017.01.005

Reference: CHB 4700

To appear in: Computers in Human Behavior

Received Date: 09 October 2015

Revised Date: 18 October 2016

Accepted Date: 01 January 2017

Please cite this article as: Özgen Korkmaz, Recep Çakir, M. Yaşar Özden, A validity and reliability
study of the Computational Thinking Scales (CTS), Computers in Human Behavior (2017), doi:
10.1016/j.chb.2017.01.005

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to
our customers we are providing this early version of the manuscript. The manuscript will undergo
copyediting, typesetting, and review of the resulting proof before it is published in its final form.
Please note that during the production process errors may be discovered which could affect the
content, and all legal disclaimers that apply to the journal pertain.
 ACCEPTED MANUSCRIPT

Highlights

► We aimed to develop computational thinking scale. ► A 29 item computational

thinking scale was composed. ► Five factor were emerged for computational thinking
scale ► Factors are termed Creativity, Cooperativity, Algoritmic- Critical Thinking and
Problem Solving ► Reliability and validity of the scales are psychometrically good.
 ACCEPTED MANUSCRIPT

A validity and reliability study of the Computational Thinking Scales

(CTS)

Özgen KORKMAZ

Amasya University, Faculty of Technology, Department of Computer Engineering,

Amasya, TURKEY

Recep ÇAKIR

Amasya University, Faculty of Education, Department of Computer Education and

Instructional Technology, Amasya, TURKEY

M. Yaşar ÖZDEN

Eastern Mediterranean University, Faculty of Education, Department of Computer

Education and Instructional Technology, Famagusta, TRNC, Mersin-10, Turkey

It is possible to define Computational Thinking briefly as having the knowledge,

skill and attitudes necessary to be able to use the computers in the solution of the
life problems for production purposes. In this study, a scale has been developed
for the purpose of determining the levels of computational thinking skills (CTS)
of the students. CTS is a five-point likert type scale and consists of 29 items that
could be collected under five factors. The study group of this work consists of
726 students educated at the levels of associate degree and undergraduate degree
with formal education in Amasya University for the first application. For the
second application 580 students who were educated in pedagogical formation
education via distance education in Amasya University. The validity and
reliability of the scale have been studied by conducting exploratory factor
analysis, confirmatory factor analysis, item distinctiveness analyses, internal
consistency coefficients and constancy analyses. As a result of the conducted
analyses, it has been concluded that the scale is a valid and reliable measurement
tool that could measure the computational thinking skills of the students. In

addition; the digital age individuals are expected to have the

computational thinking skill, and at what degree they have these skills,

the revelation of whether the levels they have are sufficient or not are a
 ACCEPTED MANUSCRIPT

requirement. Within this frame, it could be said that the scale could make

significant contributions to the literature.

Key Words:Computer-mediated communication; valuation methodologies;

pedagogical issues; programming and programming languages; teaching/learning
strategies

Correspondence: AssocProf.Dr. Özgen KORKMAZ, ozgenkorkmaz@gmail.com;

ozgen.korkmaz@amasya.edu.tr Phone (Cell): +905053192785
 ACCEPTED MANUSCRIPT

A validity and reliability study of the Computational Thinking Scales

(CTS)

It is possible to define Computational Thinking briefly as having the

knowledge, skill and attitudes necessary to be able to use the computers in the
solution of the life problems for production purposes. In this study, a scale has
been developed for the purpose of determining the levels of computational
thinking skills (CTS) of the students. CTS is a five-point likert type scale and
consists of 29 items that could be collected under five factors. The study group
of this work consists of 726 students educated at the levels of associate degree
and undergraduate degree with formal education in Amasya University for the
first application. For the second application 580 students who were educated in
pedagogical formation education via distance education in Amasya University.
The validity and reliability of the scale have been studied by conducting
exploratory factor analysis, confirmatory factor analysis, item distinctiveness
analyses, internal consistency coefficients and constancy analyses. As a result
of the conducted analyses, it has been concluded that the scale is a valid and
reliable measurement tool that could measure the computational thinking skills
of the students. In addition; the digital age individuals are expected to

have the computational thinking skill, and at what degree they have

these skills, the revelation of whether the levels they have are sufficient

or not are a requirement. Within this frame, it could be said that the

scale could make significant contributions to the literature.

Key Words: Computer-mediated communication; valuation methodologies;

pedagogical issues; programming and programming languages;
teaching/learning strategies

1. Introduction
It is possible to define Computational Thinking briefly as having the

knowledge, skill and attitudes necessary to be able to use the computers in the

solution of the life problems for production purposes (Özden, 2015). Computational

thinking is a method of problem solving, system designing and also a method of

1
 ACCEPTED MANUSCRIPT

understanding the human behaviors by drawing attention to the basic concepts of the

science of computer. Computational thinking also covers the acquainted concepts

such as analysis, data demonstration and modeling and also the ideas that are less

known such as binary search, repetition and parallelization. When the fact that

computational thinking has a border and general frame is taken into consideration, it

is a valid basic skill not only for the computers, but also for everybody and it is

considered that it shall take place in the basic skills (reading, writing and arithmetic)

used by everyone in the near future (Wing 2006). Computational thinking is a

fundamental skill for everyone, not just for computer scientists. Wing (2006)

emphasizes that computational thinking should also be added to the reading, writing

and arithmetical processes for the analytical skills of every child. According to the

researchers (Wing, 2006); the computational thinking makes use of extraction and

decomposition when it is compared to the great complex systems or processes. It

helps in the selection of a convenient representation to solve a problem or modeling in

the parts related to the problem. Moreover; the digital age individuals are expected to

have the computational thinking skill, but no proof has been encountered in the

literature regarding at what degree they have these skills and the revelation of whether

the levels they have are sufficient. Within this frame, it could be said that the scale

could make significant contributions to the literature.

According to Bundy (2007), computational thinking affects the studies in almost all

the disciplines in both the human sciences and the natural sciences. The researchers

make use of the cognitive metaphors for the purpose of enriching the theories such as

protein science and mind-body problem. The science of computer has made the

researchers gain the skills of asking new questions such as the ones requiring the data

process with substantial amount and forming new solutions.

2
 ACCEPTED MANUSCRIPT

Today, there is almost no environment in which there is no computer and no computer

is used. The daily usage is mostly limited to communication, surfing in the Internet or

office applications. Whereas, the concept of computational thinking is far deeper than

them and forces the ordinary thinking way to change. Such that, it is claimed that a

person who would like to understand the 21st century is firstly obliged to understand

the computational thinking (Bundy, 2007).

On the basis of the aforementioned explanations, there are limited numbers of studies

in the literature regarding this skill that has such an important for the individuals to

have for meeting the requirements of the digital age. Moreover; the digital age

individuals are expected to have the computational thinking skill, but no proof has

been encountered in the literature regarding the determination of at what degree they

have these skills and the revelation of whether the levels they have are sufficient.

Within this frame, it could be said that the scale could make significant contributions

to the literature.

Wing (2008) argued that CT complements thinking in mathematics and engineering

with a focus on designing systems that help to solve complex problems humans face

(Wing 2008; Lu and Fletscher 2009). The core CT concepts include, abstractions (the

mental tools of computing, necessary to solve the problem), layers (problems need to

be solved on different levels) and relationships between layers and abstractions (Wing

2008). The idea of abstraction and students’ ability to deal with different levels of

abstractions, as well as to think algorithmically and to understand the consequences of

scale (big data), are fundamental to CT (Denning 2009; Lu and Fletscher 2009).Aho

(2012) further argued that CT involves “thought processes involved in formulating

problems so their solutions can be represented as computational steps and algorithms”

3
 ACCEPTED MANUSCRIPT

(p. 832). Denning (2009) argued CT has a long history in computer science dating

back to 1950s when it was known as algorithmic thinking meaning “a mental

orientation to formulating problems as conversions of some input to an output and

looking for algorithms to perform the conversions” (p. 28). Some computer science

educators have also argued that programming is not essential in the teaching of

computational thinking (e.g. Yadav et al. 2011; Lu and Fletscher 2009). Lu and

Fletscher (2009) even suggested that an emphasis on programming might deter

students from becoming interested in computer science. In sum, computational

thinking is a conceptual way to “systematically, correctly, and efficiently process

information and tasks” to solve complex problems (Lu & Fletcher, p. 261).

Many educators and especially the experts in the educational technology field have

emphasized that Computational thinking is so significant in terms of the skills of 21st

century (VoogtFisser, Good, Mishra, &Yadav, 2015). When the definitions given in

the literature are examined, the focus regarding the computational thinking is that it

contains continuity by getting help from the computing and computers in the solution

of the complex problems (e.g. Barr and Stephenson 2011; Grover and Pea 2013; Lee

et al. 2011; Sengupta et al. 2013; Wolz et al. 2011), it is very distinguishing in many

levels in the abstraction and design-oriented thinking (Sengupta et al. 2013) and it

brings the algorithmic thinking to the forefront (Mishra and Yadav, 2013).

It is possible to see in the literature some scales that measure the sub-dimensions of

the computational thinking skills as independent from one another (Korkmaz, 2012;

Korkmaz&Yeşil, 2010; Aksoy, 2004; Kökdemir, 2003). This research is limited with

self-report based scale and other assessment techniques were not taken into

consideration.In this frame; in the literature, no scale whose validity and reliability

4
 ACCEPTED MANUSCRIPT

have been proven has been found to measure the levels regarding especially the

computational thinking skills. The purpose of this study is to develop a scale for the

purpose of determining the computational thinking skills of the students by filling this

space in the literature. It is considered that this scale shall make significant to measure

CT

2. Theoretical Framework

2.1. Computational Thinking

Even if the CT concept was used by Papert (1980) in the literature for the first

time, the first description was done by Wing (2008) but it was not countervail the

description fully. However, what the CT covers has been a subject of question.

Computational thinking that is a kind of analytical thinking according to Wing (2008)

makes use of the common points with the mathematical thinking at the stage of

problem solving, with the engineering while designing and assessing a complex

system and with the scientific thinking in understanding the concepts such as

calculability, mind, brain and human behaviors. According to ISTE (2015), this skill

does not take the place of creativity, logical thinking and critical thinking; but it

increases the capacity of a computer in the solution of a problem of a human being by

taking benefit of the human creativity and critical thinking by putting these skills forth

while revealing the problem solving ways in a way that the computers could help.

Similarly, Curzon (2015) defines the computational thinking as a basic skill that

comes to the meaning of problem solving for the human beings and points out that it

is necessary to understand what the problem is before thinking of the solutions while

solving a problem according to a certain point of view. CT; Formulating problems,

Logically organizing and analysing data, Representing data through abstractions such

5
 ACCEPTED MANUSCRIPT

as models and simulations, Automating solutions through algorithmic thinking,

Identifying, analysing, and implementing possible solutions with the goal of

achieving the most efficient and effective combination of steps and resources and

Generalizing and transferring this problem solving process to a wide variety of

problems was defined as one of the processes to solve the problem that was arisen

(CSTA & ISTE, 2011). But later on, ISTE (2015), computational thinking covers the

skills of creativity, algorithmic thinking, critical thinking, problem solving,

establishing communication and establishing cooperation (ISTE). Barr, Harrison and

Conery (2011) emphasized that the computational thinking is a problem solving

process including the following properties: The formulation of the problem in a way

that computers of other tools can help for its solution; Arrangement and analysis of

the data in a logical way; Presentation, modeling and simulation of the data by means

of abstraction; Automation of the solutions with the help of algorithmic thinking;

Application of the possible solutions; Conversion of the problem solving process into

problem variability and extensification. Smilarly, Google (2016) described the CT as

a problem solving process that includes a number of characteristics and dispositions

and it also expressed that the Ct covers the skills such as Pattern Recognition,

Abstraction and Algorithm Design.

Mannila et al.(2014), Riley and Hunt (2014) and Syslove and Kwiatkowska (2013)

emphasized that the CT is a set of thinking skills. When considering that ISTE is a

high level skill above the problem solving skill and the other skills and their

subcomponents mentioned above, it can be said that ISTE (2015) is more convenient

component than what the CT covers. For the purpose, in this research, the

components of the ISTE discussed in detail below are accepted as straightforward.

6
 ACCEPTED MANUSCRIPT

According to ISTE (2015), as a requirement of the age of informatics, the computer

people should learn where, how and when they shall make use of the digital tools for

the solution of problem.

In addition, the conducted studies have expressed that the students of 21st century

have the skills of novelty, creativity, research, cooperation, problem solving, critical

thinking, technology, social skills, cognitive skills, communication skills and self-

management skills (Günüç, Odabaşı&Kuzu,2013). In recent study; Anshari, Alas,

Yunus, Sabtu & Hamid (2016) highlighted that students of 21st century generally use

internet to access resources by using mobile devices instead of traditional techniques.

At the same time; it is also an important issue how to be in contact with the people

who may be helpful in the computer-based solutions. The concept of computational

thinking could be used to be able to explain these proficiencies as a whole. Because,

the students develop their own thinking way when they realize that the computers

could produce more effective automatical solutions while solving the problems. ISTE

(2015) emphasizes that the purpose of computational thinking in education is not

bringing the students to the position of leader in computer science, but applying their

computational thinking skills in also other courses like a habit. While the most

important skill owned by a human in problem solving is mind, the development of

this skill by means of computer and other digital tools has become one of the basic

parts of our work (Barr et.al., 2011). It is also possible to say that the students already

learn some of the sub-structures of computational thinking in the contents of many

different courses. According to Barr et.al. (2011), all the students should be sure that

they have all the sub-elements belonging to the computational thinking and the

ownership of these skills by the students and being able to transfer them to other

7
 ACCEPTED MANUSCRIPT

problem situations are seen as an important requirement.

In summary, according to ISTE (2015), computational thinking is the extension of the

problem solving skills of a person and the development of the creativity and critical

thinking skills of the people by re-focusing. The students make use of the

computational thinking while using the algorithms to solve a problem and while

solving the problem with calculation. The students establish connection with the

computational thinking while analyzing a text or designing complex communications.

In addition, they also make use of the computational thinking while analyzing very

broad data groups and expressing the pattern of the study in the scientific researches.

ISTE (2015) defines computational thinking as the common reflection of creativity,

algorithmic thinking, critical thinking, problem solving, cooperative thinking and the

communication skills. These skills are the ones that are mostly discussed in the

literature. However; when these skills are taken into consideration as together, they

explain a brand-new thinking skill that is called as computational thinking. When

considered within this frame; it shall be beneficial to explain other skills used to

define this thinking skill to be able to understand the computational thinking

correctly. These skills could be shortly explained as follows:

2.2. Creativity:
Craft (2003) has explained creativity as a skill that is not related to art and that

is life-long, and defined it as “the capacity of expressing oneself and using mind and

imagination”. Creativity is a concept that has always been existent in the life of

human being since the past and that covers the different viewpoints of the people. It

literally means to generate and form. Creative thinking is one of the prominent

concepts of our age. It has found a place for itself in many areas such as politics,

economy, art, technology and science (Aksoy, 2004). At the same time, creativity is

8
 ACCEPTED MANUSCRIPT

the skill of being able to reveal a non-existent product, being able to imagine or being

able to carry out a work in was different than those seen by everyone and being able

to develop new ideas. Being able to find different solutions for the events and

experiences faced in the daily life and having different viewpoints from those of

others are related to the richness of the creativity side of a person. Creative thinking

has a great importance in the developing societies. Gaining different viewpoints to

this development by following the improving science and art is possible with creative

thinking. According to Aksoy (2004); creativity is to reveal new relations and form

new compositions from one or more concepts in the mind for the purpose of

observing the events from new viewpoints. All the imagined things are the new

compositions of ideas, products, colors and words. Creativity results in the scientific

inventions, new products, art and literature meeting the needs of the humankind.

According to Cropley (1997) creative thinking has three basic elements.

 Novelty: It is a new product that is revealed as a result of an idea or a behavior

that is separated from its likes with certain lines.

 Impressiveness: It could be a material that brings income or that is beneficial

as well as it could be the aesthetic, artistic or divine things.

 Ethical Conformity: The term “creativity” is not used to define the selfishness,

subversion, crime, agonizing things or pleasures.

Creative thinking is not a way of thinking all alone. It also covers the thinking

structures inside itself such as critical problem solving. An individual having the

property of creative thinking also has the properties of critical thinking and problem

solving. Creative thinking starts with the self-recognition of a person. Developing

genuine ideas different from the ordinary ones and finding these ways are the results

9
 ACCEPTED MANUSCRIPT

of the problem solving skill and creative thinking. As a summary, programming is a

process that makes the problem numerical and helps realize the solution. In this

context, the student discovers his creativity and finds methods for solution. Thus, it

can be said that creativity is one of the important components of CT.

2.3. Algorithmic Thinking:

Algorithmic thinking is the skill of understanding, applying, assessing and

producing the algorithms (Brown, 2015). When it is considered that the daily life is

surrounded with the algorithms, it could be concluded that it shall be an important

gain to develop this skill of the individuals. Algorithmic thinking is deemed as one of

the key elements to be able to be an individual in line with the age of informatics

defined by NRC (US National Research Council). According to NRC union;

algorithmic thinking has been defined as follows, “General concept of algorithmic

thinking covers the subjects of generalizing and parameterize the functional analysis,

repetition, basic data organizations (record, order, listing) and the subjects of

algorithm and program, upper and lower designs and correction. In addition; it is not

specifically necessary to understand or make use of math in some algorithmic

thinking types.” (Fluent 1999). It is necessary to have the skills of understanding and

assessing the algorithms to be algorithmic thinker. While some people find it easy to

reach the solutions by using certain instructions, others could define it as challenging.

Because every step should be taken in the correct order and without any skipping, it

could be said that algorithmic thinking is a process demanding patience. In addition to

patience, an individual thinking algorithmically should be meticulous and determined.

Therefore; many people give up without being able to complete the steps (Brown,

2015). Another requirement of algorithmic thinking is the skill of assessment. This

also determines whether an algorithm shall really form a solution to a given problem.

10
 ACCEPTED MANUSCRIPT

The final requirement of algorithmic thinking is the skill of being able to produce new

algorithms. It is a hard process in a given work to form a certain order and writing the

step instructions that is always right. While it is easy to produce algorithms for easy

problem statuses, the algorithm to be written gets complex as the problem status gets

complex (Brown, 2015). Consequently; an individual that could think algorithmically

could think in a detailed and purposeful way in the issue of the solution methods

while producing a solution in any subject. As a result, solving a problem can be

realized by placing the proceedings in sequence. For the purpose, it can be claimed

that one of the important components of CT is Algorithmic Thinking.

2.4. Critical Thinking:

According to Halpern (1996); critical thinking has been defined as “the use of

cognitive skills or strategies that increase the possibility of the desired behaviors”.

When the literature is examined, it could be observed that one of the most criticized

issues of our educational system is rote learning that is the result of the traditional

understanding. The individuals that are not questioning shall be insufficient in

meeting the desired qualified human power in the age of information in which there is

always a change and development. The need for critical thinking occurs at this exact

moment. Critical thinking is one of the subjects that catch the attention of the

researchers recently. ÇoklukBökeoğlu and Yılmaz (2015) have expressed that the

individuals to keep up with the change is possible by means of using the information

efficiently as a result of the information explosion and they have emphasized that the

individuals that are making efficient use of information are those that are flexible,

creative, questioning, researching, analyzing, who could assess the events in many

ways, who could make selections, who are open for innovations, who know

themselves well, in other words, who could think critically. When the literature is

11
 ACCEPTED MANUSCRIPT

examined, it could be seen that there are many different definitions on the critical

thinking. In the most general meaning, critical thinking is one of the high level

thinking skills. Kazancı (1989) has expressed that critical thinking is the whole of the

attitude, information and skill processes that are used in the justification and

assessment of a problem status according to the scientific, cultural and social standard

scales in terms of consistency and validity. Critical thinking could be defined as the

active, regular and functional process that is carried out to be able to make better use

of the understanding and presentation skills of the individual’s or others’ ideas and

thoughts (Chaffe, 1994:imp. Kökdemir, 2003). According to Facione; critical thinking

is to make a judgment and reach a decision within the direction of a purpose with the

explanation of the proofs, concepts, methods, scales and contexts as well as the

interpretation, analysis, assessment and conclusions (imp: Özdemir, 2005). Ennis

(1985) has mentioned the three structures of critical thinking as the judgment,

development of information and questioning, and has defined critical thinking as the

reflective and logical thinking focused on deciding what shall be done and what shall

be believed. Critical thinkinghas been seen as skill of the individual for determining

the assumptions, hidden belief, values and attitudes. Smith has expressed that the

critical thinking is the judgment focused on accepting or rejecting the claims. On the

other hand; Paul has stated that critical thinking is the processes of an individual for

shaping and assessing his/her own idea. According to Mayhew; critical thinking is the

process of questioning “how” and “why” (Seferoğlu&Akbıyık, 2006). According to

Kökdemir (2003); there are the skills such as being able to catch the difference

between the reality and asserted claims, being able to test the reliabilities of the

resources belonging to the attained information, being able to debug the unrelated

information from the proofs, being aware of the prejudice and cognitive errors, being

12
 ACCEPTED MANUSCRIPT

aware of the inconsistent judgments, being able to ask efficient questions, being able

to make efficient use of verbal and written language and meta-cognitions etc. in

which the individual is aware of his/her own thoughts among the skills covered by the

process of critical thinking. To be able to make an inter-disciplinary definition of

critical thinking; in a study conducted by 46 theorists from the countries of the United

States of America and Canada by American Psychology Association (APA) in 1990,

critical thinking is defined as “the individual to make analyst and assessment-oriented

conscious judgments and express these judgments to reach a decision as to that s/he

shall do or believe” (Evancho, 2000). As a conclusion, a problem can be solved by

using different methods; in another words, when a problem is solved through critical

thinking, critical thinking can be considered as an important component in CT.

2.5. Problem Solving:

The obstacle on the way that someone finds to reach the intended purpose is

called Problem. If someone meets with some obstacles while endeavoring to reach a

certain purpose or intellection, it means there is a problem for that person (Aksoy,

2004). The word “problem” is the general name of many troubles we encounter in

life. The difficulties, distress and hardships in social life are called problem. In the

area of education, it is known as the problem to be intended to be found as numeral

depending on some values and the solution for it (Aksoy, 2004). Overcoming the

problems encountered in the future life is one of the priority targets of our education.

The processes necessary for the solution stage of the problem should be gathered and

used in the solution of the problem (Soylu&Soylu, 2006). When programming

process is considered as a main problem solving process, problem solving skill cannot

be ignored in a macro thinking skill such as CT.

13
 ACCEPTED MANUSCRIPT

2.6. Cooperativity:

Cooperative learning is a learning method in which learning of both individual

and group members of the small groups are tried to be maximized(Veenman,

Benthum, Bootsma, Dieren, & Kemp, 2002). ÇaycıBaşaran and Demir (2007) have

defined the cooperative learning as rewarding the set success and set members

helping each other to learn an academic subject with different methods by forming

small sets in accordance with a common purpose. It has been known that cooperative

learning is a method that is preferred and accepted as efficient in all levels (Johnson,

Johnson,& Smith, 2007). Cooperative learning has a preferable place among the

learning methods because of its contributions such as contribution to academic

success, sharing information and establishing social relations (Korkmaz, 2012; Nam,

2014). That the individuals in the 21st century cooperating together for a deliberate

purpose with different skills on solving complex problems is inevitable. In this

context, cooperativity can be said to be one of the decisive skills of CT.

2.7. Communication Skills:

Human beings have tried to meet their needs of transmitting their feelings,

thoughts, dreams and hopes by speaking and writing, tried to make them

understandable by means of reading and writing and the phenomenon called as

communication has occurred depending on this common sharing requirement

(Çetinkaya, 2011). Communication in basic meaning could be defined as the feeling,

thought and information sharing of the individuals with one another (Karatekin,

Sönmez, & Kuş, 2012). Communication is a process that occurs with the two people

sharing their feelings, thoughts and information to understand each other (Üstün,

2005). People get into communication and interaction with themselves and with their

environment during their lives. There is a need for efficient communication skills to

14
 ACCEPTED MANUSCRIPT

express themselves, being able to impress their environment and being able to provide

the changes they desire (Gökçe, AtanurBaşkan, 2012). Efficient communication skills

could facilitate the human relations and professional relations(Korkut, 2005). It could

be considered that people should establish communication to continue their lives and

to interact with other people.

Consequently; based on the aforementioned skills and the approach of ISTE (2015),

computational thinking could be defined as the skills of being able to develop creative

solutions for the problem with an algorithmic approach by handling a problem by the

individuals that could establish healthy communication in a cooperative environment.

Within this frame; whether a person has computational thinking skills could be

defined by examining the aforementioned skills. In CT, it is expected that the

members of the project should communicate well when the cooperativity is desired to

work to solve the problem. This communication can be provided based on the

communication skills of the members. Hence, cooperativity can be accepted as one of

the components of the CT. As a result, these mentioned sub-dimensions constitute

CT. Each sub-dimension that constitutes CT covers the fundamental thinking and

problem solving skills discussed for a long time. However, when these skills

discussed one by one for years are gathered, they lead to new and more powerful

skills. When CT is described, these skills that are in relation with each other directly

are used. On the other hand, what CT is a new is a new concept discussed in field

software. That is why, there are limited researches related to CT in literature.

15
 ACCEPTED MANUSCRIPT

2. Method

2.1.Sample
The sample group of this study consists of 726 students educated at the levels

of associate degree and undergraduate degree with formal education in Amasya

University, Turkey, in 2014-2015 spring termfor the first application and 580 students

educated in pedagogical formation education via distance education in Amasya

University for the second application. The sample of this study was selected randomly

from voluntary students.While other validity and reliability analyses have been

carried out together with the exploratory factor analysis on the data collected with 1st

Application, confirmatory factor analysis has been carried out on the data collected

with the 2nd Application. The distribution of the study group according to their

universities, classes and genders is summarized in Table 1.

(Insert Table 1)

2.2. Development process of the scale

The scale development process has started firstly by literature review and the

formation of item pool. The item pool has been formed in 3 stages. At the first stage,

the literature has been reviewed and the scales aiming to measure the similar

properties have been examined. Although computational thinking is discussed in the

literature within the scope of 21st century skills opened for discussion lately, this skill

has been explained with 6 basic skills that have been clarified in details above by

ISTE (2015). Within this frame; computational thinking has been defined by using

these basic skills in this study.

Within this frame, suitable items have been selected from the Creativity scale

called “How Creative Are You?” developed by Whetton and Cameron (2002) and

adapted by Aksoy (2004) for the purpose of determining the creativity of the students

16
 ACCEPTED MANUSCRIPT

of the faculty of education, from “Problem Solving Scale” developed in 1982 by

Heppner and Peterson and whose reliability study was conducted by Taylan (1990)

for our country for the purpose of measuring the problem solving skills, from the

“Cooperative Learning Attitude Scale” prepared by Korkmaz (2012), from “The Scale

of California Critical Thinking Tendency” translated into Turkish by Kökdemir(2003)

while forming the items and from the scale of “Logical-Mathematical Thinking”

developed by Korkmaz and Yeşil (2010); and they have been added to the item pool.

In the selection of the items, firstly, two experts of educational technology have made

separate selections and after that, the selected items have been compared and

agreement has been reached.

While forming the items in the sub-heading of Communication skills, the questions in

the item tool formed by 2 instructors and 4 researchers being expert in the field of

instructional technologies have been benefited. At the second stage for the

algorithmic thinking, interviews have been made with 13 students who are continuing

their education in the senior year of the department of computer and instructional

technologies in the faculty of education, these interviews have been examined by 2

experts and 4 researchers and the ideas attained from the interviews have been

converted into items of scale. Finally, all the items have been examined by the

researchers and decision has been reached as to the convenient items. In the item pool

of the scale of Computational Thinking attained at the end of the conducted studies;

there are 8 items for the communication skills, 20 items for the algorithmic thinking,

12 items for Critical Thinking, 8 items for Cooperative Learning, 13 items for

Creativity and 13 items for problem solving skills. The item pool consists of 74 items

in total. After that, the help of an expert of Turkish Language has been taken and the

expressions hard to understand in the items and the misstatements have been

17
 ACCEPTED MANUSCRIPT

controlled and necessary corrections have been made.

Five-point options have been placed as opposed to the formed items for the purpose

of determining the levels of the students expressed in the items. These options have

been arranged and scored as “(1) never”, “(2) rarely”, (3) sometimes”, “(4) generally”

and “(5) always”. The formed draft scale has been examined by 13 students whose

comments have been taken beforehand, it has been questioned whether the students

have had hardship in understanding the items and how they perceive each item, the

items not understood or determined to be understood differently have been examined

again and the draft scale has been finalized.

2. 3.Data analysis
Within the frame of the statistical analyses on the data collected with the scale,

firstly the KMO and Bartlett analyses have been conducted for the purpose of

determining the construct validity of the scale and it has been detected whether factor

analysis could be conducted or not. The fact that the KMO value is above 0,90 is

interpreted as the fact that the data set is in the excellent level for conducting factor

analysis (Russell, 2002). Also; According to the Bartlett values known as the unit

matrix of the correlation it tests, it is understood that the zero hypothesis is rejected in

0,05 meaningfulness level (Büyüköztürk, 2002; Eroğlu, 2008).

With regard to the attained values, exploratory and confirmatory factor analyses have

been conducted on the data; the status of the scale in distribution to factors has been

examined via basic components analysis and the factor loads have been examined

with the use of Varimax steep rotation technique. Factor analysis is used for the

purpose of revealing whether the items in a scale are distributed into less number of

factors or not (Balcı, 2009). Basic components analysis is a technique frequently used

18
 ACCEPTED MANUSCRIPT

as a factorizing technique (Büyüköztürk, 2002). Within this frame; as a result of the

Basic Components Analysis used in the factor analysis, it is necessary that the items

whose factor loads are below 0,40 and the items in which there is not at least 0,100

difference between their loads in two factors, in other words, the items whose load is

distributed into two factors should be dismissed (Büyüköztürk, 2002). However, the

fact that the factor loads of the items taking place in the scale are higher than 0,30 and

the fact that at least 40% of the general variance is explained are seen as sufficient in

terms of behavior sciences (Büyüköztürk, 2002; Eroğlu, 2008; Kline, 1994; Scherer at

al., 1988). However; the fact that the factor loads are 0,50 and above is accepted as

very good (Büyüköztürk, 2002). The basic criterion in the assessment of the results of

the factor analysis is the factor loads (Balcı, 2009; Gorsuch, 1983; Eroğlu, 2008)).

The fact that the factor loads are high is seen as an indicator that the variable may take

place below the mentioned factor (Büyüköztürk, 2002). In addition, it is expressed

that the calculation of the common factor variance is significant especially in terms of

multi-factor patterns and is defined as the common variance caused by the factors on

each variable as a result of the factor analysis (Çokluket.al., 2010). In the event that

the common factor variance is below 0,20, there are comments as to the fact that this

item should be excluded from the scale (Çokluk et.al., 2010).

The scale form attained with the exploratory factor analysis has been applied to a new

study group except for the one to which the first application has been made; and the

confirmatory factor analysis has been conducted on the attained values. The

confirmatory factor analysis is based on the principle of taking the formulas (items

and factors) between the observable and non-observable variables as a hypothesis and

testing them (Pohlmann, 2004). In other words, confirmatory factor analysis is a

structural equation model that is concerned with the measurement models of the

19
 ACCEPTED MANUSCRIPT

relations between latent variables and observed measurements. Each factor is

explained in terms of the relations between them and the observed variables (items)

(Yılmaz and Çelik, 2009; Raykov and Marcoulides, 2006). Maximum likelihood

technique has been used in the confirmatory factor analysis. Generally in the

structural equation model, it is suggested to report more than one conformity value

(Thompson, 2000). For this reason, in this study, five conformity values have been

reported. Within this frame, the fact that the values observed in the scale model

attained as a result of the confirmatory factor analysis is between the range of χ2/d<3;

0<RMSEA<0.05; 0≤S-RMR≤0.05; 0.97≤NNFI≤1; 0.97≤CFI≤1; 0.95≤GFI≤1;

0.95≤AGFI≤1 and 0.95≤IFI≤1 indicates the excellent conformity, the fact that they

are between the range of χ2/d<5; 0.06≤RMSEA<0.08; 0.06≤S-RMR≤0.08;

0.90≤NNFI≤0.96; 0.90≤CFI≤0.96; 0.90≤GFI≤0.96; 0.90≤AGFI≤0.96 and

0.90≤IFI≤0.96 indicates the acceptable conformity (Kline, 2005; Şimsek, 2007).

The validity property of the scale has been determined by testing the item

distinctiveness powers of the items remaining in the scale as a result of the factor

analysis with independent sample t test; and the item-total correlations with Pearson’s

r test. Finding a correlation between the points attained from each item and the point

attained from the factor to which the item belongs to is used as a criterion in terms of

understanding the level of servicing the general purpose of the factor (Balcı, 2009).

Another value that is observable in terms of testing the level of an item in servicing

the general purpose is the corrected correlations. The fact that the corrected

correlation coefficients are higher than 0,20 means that an item could serve the

purpose of the related factor in a meaningful level (Tavşancıl, 2010). These

coefficients are the validity coefficients of each item and they express the consistency

of the scale as a whole, in other words, the level of the scale in servicing the general

20
 ACCEPTED MANUSCRIPT

purpose (Parasuraman, Zeithaml&Berry, 1988: imp. Yüksel, 2009;

Carminesi&Zeller, 1982). The property of distinctiveness is accepted as one of the

significant proofs used in the determination of the validity of a scale (Büyüköztürk,

2008). Another way to test the distinctiveness of a scale is to observe the

differentiation between the lower 27% and upper 27% groups after sequencing the

raw points attained from an item in a gradual way.

Internal consistency and constancy tests have been conducted to determine the

reliability of the scale. Cronbach alpha reliability coefficient, the correlation value

between the two equal-half, Sperman-Brown formula and Guttmannsplit-half

reliability formula have been used in the determination of the internal consistency

level. The fact that the reliability coefficient is 0,70 and above is accepted as an

indicator of the reliability of a scale (Büyüköztürk, 2002; Gorsuch, 1983). The

constancy level of the scale has been calculated by determining the correlation

between two application results conducted with 5-week intervals. As it is known, a

reliable measurement tools should be able to conduct consistent measurements (Balcı,

2009). In addition; reliability is related to the properties of constancy, consistency and

preciseness. For this reason, these values determined as constancy coefficient are

assessed as a proof of whether the reliability of the scale is high or not (Hovardaoğlu,

2000). The reliability coefficient expressing the consistency degree rises as it gets

close to 1,00 and decreases as it gets close to 0,00 (Gorsuch, 1983). As it is known;

generally for the correlation coefficients, the level of 0,00 – 0,30 expresses the

existence of a low correlation, the level of 0,30 – 0,70 expresses the existence of a

medium correlation and the level of 0,70 – 1,00 expresses the existence of a high

correlation (Büyüköztürk, 2002).

21
 ACCEPTED MANUSCRIPT

3. Results

3.1.Findings Regarding the Validity of the Scale

Within the frame of the validity of Computational Thinking Scale (CTS); the

construct validity, item-total correlations, corrected correlations and item

distinctivenesses have been examined and the findings have been presented below:

3.1.1. Construct Validity

Findings Regarding the Exploratory Factor Analysis: Firstly; the tests of

Kaiser-Meyer-Oklin (KMO) and Bartlett have been conducted on the data for the

purpose of testing the construct validity of CTS and the results have been determined

as KMO= 0,914; Bartlett test value χ2= 15886,208; sd=2701 (p=0,000). Within the

frame of these values, it has been understood that factor analysis could be conducted

on the scale consisting of 74 items

At the first stage, basic components analysis has been conducted to determine whether

the scale is one-dimensional or not. After that, Varimax steep rotation technique has

been used according to the basic components. Within this frame; after 21 items whose

item load is below 0,40 at the end of the analyses repeated with multiple stages and 24

items whose load is distributed to different factors, namely a total of 45 items have

been excluded from the scale, factor analysis has been conducted again on the

remaining items. As explained before; computational thinking skill could be

explained with the basic skills such as problem solving, critical thinking, algorithmic

thinking, communication and cooperative learning. For this reason, while preparing

the item pool, it has been expressed that benefit has been taken from some items

taking place in “How Creative Are You?”, “Problem Solving Scale”, “Scale of

Cooperative Learning Attitude”, “Scale of California Critical Thinking” and

22
 ACCEPTED MANUSCRIPT

“Logical-Mathematical Mind Scale”. However, because these skills are so connected

to each other, lots of similar items may take place in the mentioned scales. As a result

of the conducted analyses, these similar items have been distributed to more than one

factor naturally. For this reason, such amount of items (45 items) has been excluded

from the item pool obligatorily. For the purpose of sustaining the content validity

possible to be distorted due to the excluded items; the attained item pool has been

examined again by two educational technologists and two experts of guidance and

psychological counseling. Other analyses could be proceeded after the field experts

have specified that the remaining items are sufficient in terms of measuring the

related skills.

It has been seen that a total of 29 items remaining in the scale as a result of these

processes have been categorized under five factors. With its final condition; it has

been determined that the KMO value of the scale is 0,880 and the Bartlett values are

χ2=7727,897; sd=406; p<0,001. It has been seen that the factor loads of the 29 items

remaining in the scale are between 0,475 and 0,785 without exposing to any rotation;

on the contrary, these loads are between ,494 and ,842 when they are exposed to

rotation after the varimax steep rotation technique. On the other hand; it has been

determined that the items and factors taken to the scope of the scale explain 56,12%

of the total variance. In the next step, the contents of the items in the factors have

been examined and the factors have been named. Because the skill levels have been

taken into consideration to define the computational thinking skill while forming the

item pool, this situation has been taken into account also while naming the factors

occurring. The occurring factors coincide with the sub-skills determined while

forming the item pool at the beginning to a large extent. Within this frame; 8 items

have been collected under the factor called as “Creativity”, 6 under the factor called

23
 ACCEPTED MANUSCRIPT

as “Algorithmic Thinking”, 4 under the factor called as “Critical Thinking” and 6

under the final factor called as “Problem Solving”.

This situation is also seen in the slope accumulation graphic (Graphic 1) drawn

according to the eigenvalues. In Graphic 1, it means that there is high-accelerated

decrease in the first five factors; these five factors have a significant contribution due

to this; on the contrary, the decrease in the other factors has become horizontal, in

other words, their contributions to the variances are close to each other (Büyüköztürk,

2002; Eroğlu, 2008).

(Insert Figure 1)

As a result of these conducted processes, the findings regarding the item loads of the

total 29 items remaining in the scale according to the factors and the amounts of the

factors in explaining the eigenvalues and variance are presented in Table 2.

(Insert Table 2)

As it could be seen in Table 2; the “Creativity” factor of the scale includes 8 items

and the factor loads vary between 0,548 and 0,708. The eigenvalue of this factor

within the general scale is 7,19; and its amount of contribution to the general variance

is 13,5%. “Algorithmic Thinking” factor includes 6 items. The factor loads of the

items are between 0,666 and 0,827. The eigenvalue of the factor within the general

scale is 3,19; and its amount of contribution to the general variance is 13,1%.

“Cooperativeness” factor includes 4 items. The factor loads of the items are between

0,685 and 0,842. The eigenvalue of the factor within the general scale is 2,54; and its

amount of contribution to the general variance is 10,7%. “Critical Thinking” factor

includes 5 items. The factor loads of the items are between 0,533 and 0,764. The

eigenvalue of the factor within the general scale is 1,80; and its amount of

24
 ACCEPTED MANUSCRIPT

contribution to the general variance is 10,1%. “Problem Solving” factor includes 6

items. The factor loads of the items are between 0,494 and 0,720. The eigenvalue of

the factor within the general scale is 1,34; and its amount of contribution to the

general variance is 8,7%. In addition, because all of the items collected under this

factor are reverse, they should be coded reversely while coding.

Findings Regarding the Confirmatory Factor Analysis: Confirmatory factor analysis

has been conducted on the data collected from 580 students except for the sample in

which the data used for the exploratory factor analysis has been collected regarding

the confirmation of the factor structures of the scale detected to be consisting of 5

factors as a result of the exploratory factor analysis.

The estimate values produced for each item as a result of the confirmatory factor

analysis conducted with the use of the maximum likelihood technique without making

any limitation are presented in Table 3.

(Insert Table 3)

As it could be seen in Table 3, it is seen that 4 of the estimate values are far from

0,70. However; these four items have not been excluded from the scale with the fear

that it may negatively affect the content validity. Within this frame; it has been

observed that the estimate values of the items are between 0,470 and 0,861.

When the values of the goodness of fit are examined, they have been found as

χ2(sd=362, N=580)= 1169,932, p<.001, CMIN/DF=3,232, RMSEA= 0,062, S-RMR=

0,044, GFI= 0,91, AGFI= 0,90, CFI= 0,95 and IFI= 0,97. According to these values,

it could be said that the observed fit values show an acceptable goodness (Kline,

2005; Şimsek, 2007). In other words, this attained model reveals that the factors are

confirmed by the data. The factorial model of the scale and its values regarding the

25
 ACCEPTED MANUSCRIPT

factor-item relation are given in Figure 2.

(Insert Figure 2)

3.1.2. Item Factor Total and Corrected Correlations

In this section; the correlations between the points attained from each item in

the factors and the points attained from the factors have been calculated according to

the item total correlation and corrected item correlation method and the level of each

item in servicing the general purpose has been tested. The item-factor correlation

values attained for each item are given in Table 4 and the corrected correlation values

are given in Table 5.

(Insert Table 4)

As it could be seen in Table 4; the item test correlation coefficients vary between

0,671 and 0,732 for the first factor, between 0,717 and 0,833 for the second factor,

between ,788 and ,889 for the third factor, between ,681 and ,809 for the fourth factor

and between ,632 and ,677 for the final factor. Each item is in a meaningful and

positive relation with the factor in general (p<0,000). According to this, it could be

said that each item serves the general purpose of both the factor it is in and also the

scale.

(Insert Table 5)

As it could be seen in Table 5, the corrected correlation coefficients with the factor of

each item they belong to in the scale vary between ,506 and ,610 for the first factor,

between ,599 and ,748 for the second factor, between ,643 and 783 for the third

factor, between ,467 and ,663 for the fourth factor and between ,415 and ,496 for the

final factor. It could be said that these results support the above results; and according

26
 ACCEPTED MANUSCRIPT

to this, each item serves the general purpose of both the factor it is in and also the

scale.

3.2.3. Item Distinctiveness

The distinctiveness power of the items taking place in the scale has been

calculated. For this purpose; firstly, the raw points attained from each item have been

gradually sequenced; after that, the lower and upper groups of 196 people each

forming the lower 27% and upper 27% groups have been determined. Independent

groups t-test values have been calculated on the total points taking place in the

groups. The t values regarding the distinctiveness powers and findings regarding the

meaningfulness levels are presented in Table 6.

(Insert Table 6)

In Table 6; it is seen that the independent sample t test values regarding the 29 items

in the scale and the total point vary between -3,197 and 16,287. T value for the

general of the scale has been determined as -37,105. The level of each determined

difference is meaningful (p<0,001). According to this; it could be said that the

distinctiveness of both the whole and each item of the scale is high. However; it is

also seen that the distinctiveness level of the Problem solving factor is lower when

compared to other factors.

3.2. Findings Regarding the Reliability of the Scale

Internal consistency and constancy analyses have been conducted on the data

for the purpose of calculating the reliability of the scale. The conducted processes and

findings are presented below:

27
 ACCEPTED MANUSCRIPT

3.2.1.Internal Consistency Level

The reliability analysis of the scale according to the factors and as a whole has

been calculated with the use of the Cronbach Alpha reliability coefficient, the

correlation value between two equal-half, Sperman-Brown formula and

Guttmannsplit-half reliability formula. The values of the reliability analysis regarding

the general of the scale and regarding each factor are summarized in Table 7:

(Insert Table 7)

As it could be seen in Table 7; the two Split Half correlations of the scale consisting

of a total of 29 items and the five factors has been determined as ,344; Sperman

Brown reliability coefficient as 0,512; GuttmannSplit-Half value as ,498; Cronbach

Alpha reliability coefficient as ,822. On the other hand; it is seen that the split half

correlations regarding the factors vary between ,406 and ,713; Sperman Brown

values between ,578 and ,832; GuttmannSplit-Half values between ,578 and ,832 and

Cronbach’s Alpha values between 0,727 and 0,869. According to this, it could be said

that both each factor and the scale in general could conduct consistent measurements.

3.2.2.Constancy Level
The constancy level of the scale has been detected with the use of test re-test

method. The final form of the scale consisting of 29 items has been applied to 51

students on which application has been carried out after three weeks again. The

relation between the points attained at the end of both applications has been taken into

consideration both in terms of each item and the whole of the scale. In this way; the

property of both each item taking place in the scale and the whole of the scale in

conducting consistent measurements has been tested and the findings have been

summarized in Table 8.

28
 ACCEPTED MANUSCRIPT

(Insert Table 8)

In Table 8; it is seen that the correlation coefficients of each item forming the scale

attained with the method of test re-test vary between 0,317 and 0,671 and each

relation is meaningful and positive. The correlation coefficients of the factors forming

the scale attained with test re-test method vary between ,371 and ,613. The correlation

regarding the total point is ,512 and it is seen that each relation is meaningful and

positive. According to this, it could be said that the scale could make consistent

measurements.

4. Discussion
In this study, a scale has been developed for the purpose of determining the

computational thinking skill levels of the students. CTS is a five-point likert type

scale and consists of 29 items that could be collected under five factors. Each one of

the items taking place in the factors has been scaled as never (1), rarely (2),

sometimes (3), generally (4), always (5). The validity of the scale has been examined

with two different methods. These are the methods of (1) factor analysis and (2)

testing of validity via the distinctiveness properties.

When the factor loads of the items in the factors, eigenvalues of the factors and the

explained variance rates are taken into consideration, it could be said that the scale

has construct validity. Confirmatory factor analysis has been conducted to verify the

factor structures of the scale detected to be consisting of 5 factors as a result of the

exploratory factor analysis. As a result of the conducted confirmatory factor analysis;

the observed values of the scale model reveal that the data show conformity, in other

words, this attained model has been verified by the data.

29
 ACCEPTED MANUSCRIPT

Total item correlations and corrected correlations have been calculated on the data in

order to identify at what level each one of the items in the scale measure the factor it

belongs to and the properties to be measured. According to the attained values, it has

been identified that each item and each factor in the scale serve the purpose of

measuring the property to be measured and the whole of the scale at meaningful level.

Additionally, the item distinctiveness levels have been investigated by examining t

values related to the difference between upper 27% and lower 27% groups and it has

been determined that both the whole of the scale and each item have high

distinctiveness; in other words, each item is distinctive at the required level. The

internal consistency coefficients of the scale have been calculated by using two split

half correlations, Cronbach Alpha, Sperman-Brown formula and Guttmannsplit-half

reliability formula. Within the scope of these calculated values, it has been

determined that the scale can make reliable measurements. The constancy level of the

scale according to time has been investigated by using test re-test method. Test re-test

method has been calculated within the scope of sub factors of the scale and for each

item and it has been identified that each factor and each item can make decisive

measurements as constancy according to time.

Although it has been expected to emerge a structure having six factors according to

ISTE (2015) as a result of the analysis related to construct validity, a structure with

five factors has emerged. When the items coming under factors have been examined,

it has been seen that communication skills and a great part of the related items have

come under the factors of critical thinking, problem solving and cooperative learning.

It has been decided that the situation is acceptable thinking that the communication

skills have a fundamental qualification in the occurrence of the skills mentioned

above and the structure with five factors has been maintained. Thus, communication

30
 ACCEPTED MANUSCRIPT

can be simply defined as people sharing emotions, thoughts and information

(Karatekin, Sönmez, & Kuş, 2012). In other words, communication is a process

occurring by people’s sharing their emotions, thoughts and information with each

other in order to understand each other (Üstün, 2005). In this context, it will not be

wrong to say that communication is the basis of cooperative learning, critical thinking

and problem solving skills. The remaining five factors can be explained shortly as:

Creativity:Craft (2003) has explained the creativeness as an ability that is not related

only to art and that continues lifelong and defined it as self expression, the capacity to

use intelligence and imagination. According to Aksoy (2004), creativity is to

introduce new relations and form new combinations from one or more concepts in the

mind for the purpose of looking at the events from different aspects. Everything that

is created is the new combinations of the ideas, products, colors and the words.

Creativeness is ended up with scientific discoveries, new products, art and literature.

Algorithmic Thinking: Algorithmic thinking is the ability to think, understand the

algorithms, apply, evaluate and produce (Brown, 2015). Critical Thinking: According

to Halpern (1996), critical thinking has been defined as “using cognitive skills or

strategies increasing the possibility of the intended behaviors”. Problem Solving: The

obstacle on the way that someone finds to reach the intended purpose is called

Problem. If someone meets with some obstacles while endeavoring to reach a certain

purpose or intellection, it means there is a problem for that person (Aksoy, 2004).

Cooperativity: Cooperative learning is a learning method in which learning of both

individual and group members of the small groups are tried to be

maximized(Veenman, Benthum, Bootsma, Dieren, & Kemp, 2002).

5. Conslusion

31
 ACCEPTED MANUSCRIPT

It could be said that statistically CTS is a valid and reliable scale that could be used

in the identification of the computational thinking levels of students. Since there has

not been encountered a valid and reliable measurement tool aiming at measuring the

computational thinking levels as a whole in the literature, it could be thought that this

measurement tool could make important contributions to literature. However, the

validity and the reability of the scale should be done to use it for different groups and

purposes. It should be considered that each scale can be valid and reliable for its own

group. Besides, the validity and reliability studies of the measurement tool have been

conducted by the participation of 1306 participants consisting of the students studying

at different associate, undergraduate and post graduate degrees and graduated from

undergraduate degrees and having pedagogical formation. That the participants are

from different education levels and age groups has given an opportunity to use the

scale in different groups. However, it is suggested that the validity and reliability

analysis are done again when it is wanted to be used in the groups in the scope of the

study. On the other hand, five factor point obtained from the scale could be used

separately; it could also be used as total point.

However; this study is limited to the associate students and the students studying in

higher levels. Adaptation study should be conducted in the event that this study is

desired to be applied to the high school students or the students in lower grades. In

addition; computational thinking is limited to the sub-skills expressed in the definition

made by ISTE (2015) in this scale and to the basic items taking place in the scales

developed separately for these skills

32
 ACCEPTED MANUSCRIPT

6. References

Aksoy, B. (2004). Coğrafya öğretiminde problem dayalı öğrenme yaklaşımı [The

Problem-Based Learning Approach in Geography Teaching ]. (Unpublished
master’s thesis).Gazi University, Institute of Education Sciences, Ankara.
Anshari, M., Alas, Y., Yunus, N.M., Sabtu, N.I. & Hamid, M.S. (2016). Online
learning: trends, issues, and challenges in big data era. Journal of E-Learning
and Knowledge Society.12 (1), 121-134.
Balcı, A. (2009). Sosyal bilimlerde araştırma: Yöntem, teknik ve ilkeler [Research in
social science: Methods, techniques and principles]. Ankara: PegemA Pub.
Barr, D., Harrison, J. &Conery, L.(2011).Computational Thinking: A Dijital Age
Skill for Everyone, Available at: http://files.eric.ed.gov/fulltext/EJ918910.pdf
Brown, W. (2015).Introduction to Algorithmic Thinking. Available at:
www.cs4fn.com/algoritmicthinking.php
Bundy, A. (2007). Computational thinking is pervasive. Available at:
http://www.inf.ed.ac.uk/publications/online/1245.pdf
Büyüköztürk, Ş. (2002). Sosyal bilimler için very analizi el kitabı [Data analysis for
social sciences hand book]. Ankara: PegemA Pub.
Carmines, E.G., & Zeller, R.A. (1982).Reliability and validity assessment. 5th ed.
Beverly Hills: Sage Publications Inc.
Çaycı, B., Demir, M., Başaran, M., &Demir, M. (2007).Concept Teaching With
Cooperative Learning On Social Studies Lesson.Kastamonu Education
Journal, 19(2), 619-630.
Çetinkaya, Z. (2011). Identifying Turkish Pre-Service Teachers’ Views Related To
Communication Skills. Kastamonu Education Journal, 19(2), 567-576.
Çokluk, Ö.,Şekercioğlu, G. &Büyüköztürk, Ş. (2010). Sosyal bilimler için çok
değişkenli istatistik: Spss ve Lisrel uygulamaları [Multivariable statistics for
social sciences: Spss and Lisrel applications]. Ankara: PegemA pub.
Çokluk Bökeoğlu, Ö. &Yılmaz,K. (2005). The Relationship Between Attitudes Of
University Students Towards Critical Thinking And Research Anxieties.
Educational Administration: Theory and Practice, 41, 47-67.
Craft, A. (2003). Creative thinking in theearlyyears of education. EarlyYears, 23(2),
143– 54.

33
 ACCEPTED MANUSCRIPT

Cropley, A.J. (1997). Fostering creativity in the classroom: general principles.The

Creativity Research Handbook.Runco, M.A. (eds). Volume One. Hampton
Press. Cresskill, pp. 83-114. New Jersey.
CSTA & ISTE (2011).Operational Definition of Computational Thinking for K–12
Education. Available at:
http://csta.acm.org/Curriculum/sub/CurrFiles/CompThinkingFlyer.pdf.
Curzon, P.(2015). Computational Thinking: Searching to Speak. Available at:
http://teachinglondoncomputing.org/free-workshops/computational-thinking-
searching-to-speak/
Ennis, R.(1985). Goals for critical thinking curriculum. A. Costa (Ed.), Developing
Minds ;A Resource Book for Teaching Thinking: Alexandria, VA:
Association for Supervision and Curriculum Development. s 54-57
Ennis, R., (1985).Goals for critical thinking curriculum. A. Costa (Ed.), Developing
Minds ;A Resource Book for Teaching Thinking: Alexandria, VA:
Association for Supervision and Curriculum Development. s 54-57.
Eroğlu, A. (2008). Faktör analizi [Factor analyses]. In: Kalaycı, Ş. (ed), SPSS
Uygulamalı Çok Değişkenli İstatistik Teknikleri [Statistics Techniques with
Multi Variable in SPSS Applications], Ankara: Asil Publishers, 321-331.
Evancho, R. S. (2000). Critical thinking skills and dispositions of the undergraduate
baccalaureate nursing student.Southem Connecticut State University,
Connecticut, unpublished master's thesis.
Gökçe, D., &Atanur Başkan, G. (2012). Educational supervisors communication
Skills. H.U. Journal of Education, 42: 200-211.
Google (2016).Computational Thinking for Educators. Available at:
https://computationalthinkingcourse.withgoogle.com/unit?lesson=8&unit=1
Gorsuch, R. L. (1983). Factor analysis. Hillsdale: Lawrence Erlbaum Associates.
Günüç, S. Odabaşı, F., Kuzu A. (2013). 21. Yüzyıl öğrenci özelliklerinin öğretmen
adayları tarafından tanımlanması: Bir twitter uygulaması. [The defining
characteristics of students of the 21st century by student teachers: A twitter
activity]. journal of theory and practice in education, 9(4): 436-455
Halpern, D. F. (1996). Thoughts and Knowledge: An Introduction to Critical
Thinking, New Jersey-London: Lawrence Erlbaum Associates.
Hovardaoğlu, S, (2000).Davranış bilimleri için araştırma teknikleri [Research
techniques for behavioral science]. Ankara: Ve-Ga Pub.

34
 ACCEPTED MANUSCRIPT

ISTE (2015). CT Leadership toolkit. Available at http://www.iste.org/docs/ct-

documents/ct-leadershipt-toolkit.pdf?sfvrsn=4.
Johnson, D. W., Johnson, R. T. & Smith, K. (2007). The state of cooperative learning
in postsecondary and professional settings. Educational Psychology Review,
19(1), 15–29. Doi: 10.1007/s10648-006-9038-8
Karatekin, K., Sönmez, Ö. & Kuş, Z. (2012). Investigation Of Primary School
Students "Communication Skills According To Several Variables.
International Periodical For The Languages, Literature and History of Turkish
or Turkic, 7(3), 1695-1708.
Kline, P. (1994). An easy guide to factor analysis. London and New York: Routledge.
Kline, R.B. (2005). Principles and practice of structural equation modeling, 2nded,
New York: Guilford Press.
Kökdemir, D. (2003). Belirsizlik durumlarinda karar verme ve problem çözme
[Making decisions and problem solving in uncertainty].Unpublished PhD
thesis. Ankara: Ankara University, Institute of Social Sciences, Department of
Social Psychology.
Korkmaz, Ö. (2012).A validity and reliability study of the Online Cooperative
Learning Attitude Scale.Computers& Education, 59, 1162-1169.
Korkut, F. (2005). Communication Skills Training Program For Adults. H.U. Journal
of Education 28:143-149.
Mannila, L., Dagiene, V., Demo, B., Grgurina, N., Mirolo, C., Rolandsson, L.&
Settle, A. (2014). Computational thinking in K-9 education. In: Proceedings of
the Working Group Reports of the 2014 on Innovation & Technology in
Computer Science Education Conference, ITiCSE-WGR 2014, 1–29. ACM,
New York.
Nam, C. W. (2014). The effects of trust and constructive controversy on student
achievement and attitude in online cooperative learning
environments.Computers in Human Behavior, 37, 237-248
Özdemir, S. M. (2005). Assessing university students’ critical thinking skills for some
variables .Gazi University, The Journal of Turkish Educational Science.3 (3),
1 -17.
Özden, M. Y. (2015). Computational Thinking.http://myozden.blogspot.com.tr/2015/06/
computational-thinking-bilgisayarca.html

35
 ACCEPTED MANUSCRIPT

Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. New York,
NY, USA: Basic Books.
Pohlmann, J.T. (2004). Use and Interpretation of Factor Analysis in The Journal of
Educational Research: 1992-2002. The Journal of Educational Research,
98(1), 14-23
Raykov, T., & Marcoulides, G.A. (2006).A first course structural equation modeling.
New Jersey: Lawrence Erlbaum Assocation Inc. Publishers, pp 4.
Riley, D. D.& Hunt, K. A. (2014). Computational thinking for the modern problem
Solver. Boca Raton, FL : CRC Press.
Russell, D. W. (2002). In search of underlying dimensions:the use (and abuse) of
factor analysis. Personality and Social Psychology Bulletin, 28, 1629–1646.
Scherer, R.F., Wiebe F.A., Luther, D. C. & Adams J. S. (1988). Dimensionality of
coping: Factor stability using the ways of coping questionnaire, Psychological
Reports 62(3), 763-770. PubMed PMID: 3406294.
Seferoğlu S.S & Akbıyık C. (2006).Teaching critical thinking. H.U. Journal of
Education. 30:193-200.
Şimşek, Ö.F. (2007).Yapısal eşitlik modellemesine giriş [Introduction to structural
equation modeling]. Ankara: Ekinoks Pub., 18-71.
Soylu, Y. &Soylu, C. (2006). The Role of Problem Solving in Mathematics Lessons
For Success.Inönü University Educational Journal, 7(11), 97-111.
Syslo, M. M. & Kwiatkowska, A. B. (2013). Informatics for All High School Students
: A Computational Thinking Approach. In: Diethelm, I., Mittermeir, R.T.
(eds.) ISSEP 2013. LNCS, 7780, 43–56. Springer, Heidelberg
Tavşancıl E. (2010). Tutumların ölçülmesi ve SPSS ile very analizi [Measurement of
attitudes, and data analysis with SPSS], 4th ed. Ankara: Nobel Pub., 93-124.
Thompson, B. (2000). Ten commandments of structural equation modeling. In
Grimm, L. G., Yarnold, P. R. (Eds.), Reading and understanding more
multivariate statistics (pp. 261–284). Washington, DC: American
Psychological Association.
Üstün, B. (2005).Because Communication Has More Changed.The Journey of
Atatürk University Nursing School, 8(2), 88-94.
Veenman, S., Benthum, N., Bootsma, D., Dieren, J. & Kemp, N. (2002).Cooperative
Learning and Teacher Education. Teaching and Teacher Education, 18, 87-
103.

36
 ACCEPTED MANUSCRIPT

Wing, J. M. (2008). Computational thinking and thinking about computing. Phil.

Trans. R. Soc. A, 366: 3717–3725 doi:10.1098/rsta.2008.0118
Wing, J. M. 2006 Computational thinking. Commun. ACM 49:33-35.
Yaşar, M. C. & Aral, N. (2010).The Effects of Early Childhood Education on
Creative Thinking Skills. Theoretical of educational journey, 3(2), 201-209.
Yeşil, R. & Korkmaz, Ö. (2010). Reliability and validity analysis of the multiple
intelligence perception scale. Education. Project Innovation Inc. Volume 131,
Issue1. 8-32.
Yılmaz, V. & Çelik, E. (2009).Lisrelile yapısal eşitlik modellemesi I [Structural
equation modeling with Lisrel I]. Ankara: PegemA pub, pp 53.
Yüksel, S. (2009).Informal contacts of education faculty students and correlate to
their academic achievement. Ahi Evran University Journal of Kirsehir
Education Faculty. 10(2): 119-127.

37
 ACCEPTED MANUSCRIPT

Figure caption page

Figure 1. Screen plot graphic (eigenvalues according to the factors).

Figure 2. Confirmatory factor analysis diagram of the scale

38
 ACCEPTED MANUSCRIPT

Tables

Table 1.The distribution of the study group according to the University, Class and
gender
I. Implementation II. Implementation
Department Female
Male Total Female Male Total
Pri. Math Ed. 30 10 40
Science Ed. 51 21 72
CEIT 34 52 86
Pri. Clas. Ed 2 3 5
Türkish Ed. 32 13 45
Psch. Cons and 75 27 102
Guid.
Soc. Sci. Ed.. 43 46 89
Comp. Prog. 32 60 92
Mech .Eng. 8 45 53
Elec-Elect Eng. 19 81 100
Math 48 50 98
Biology 42 11 53
Geograph 3 7 10
Liteature 53 34 87
Philosoph 27 3 30
Phisics 11 16 27
Theology 71 45 116
Chemistry 13 10 23
Healty Sci. 79 13 92
History 26 18 44
Total 358 326 684 373 207 580
*. 42 students have not specified their department or gender in the first application.

Table 2. Factor analysis results of the scale as per factors

Items Com. F1 F2 F3 F4 F5
Fact.
I59 I like the people who are sure of most of ,592 ,708
their decisions
I58 I like the people who are realistic and ,628 ,703
neutral
I69 I believe that I can solve most of the ,643 ,686
problems I face if I have sufficient amount
of time and if I show effort.
I70 I have a belief that I can solve the ,659 ,658
problems possible to occur when I
encounter with a new situation.
I67 I trust that I can apply the plan while ,533 ,629
making it to solve a problem of mine.
I57 Dreaming causes my most important ,586 ,609
projects to come to light.
I56 I trust my intuitions and feelings of ,593 ,579
“trueness” and “wrongness” when I
Creativity

approach the solution of a problem

I66 When I encounter with a problem, I stop ,476 ,548
before proceeding to another subject and
think over that problem.

39
 ACCEPTED MANUSCRIPT

I26 I can immediately establish the equity that ,731 ,827

will give the solution of a problem
I25 I think that I have a special interest in the ,719 ,824
mathematical processes
I28 I think that I learn better the instructions ,688 ,808
made with the help of mathematical
symbols and concepts
AlgoritmicThinking

I27 I believe that I can easily catch the relation ,491 ,719
between the figures
I22 I can mathematically express the solution ,578 ,673
ways of the problems I face in the daily
life.
I9 I can digitize a mathematical problem ,625 ,666
expressed verbally.
I42 I like experiencing cooperative learning ,785 ,842
together with my group friends.
I46 In the cooperative learning, I think that I ,693 ,818
attain/will attain more successful results
Cooperativity

because I am working in a group.

I41 I like solving problems related to group ,728 ,817
project together with my friends in
cooperative learning.
I45 More ideas occur in cooperative learning. ,632 ,685
I38 I am good at preparing regular plans ,723 ,764
regarding the solution of the complex
problems.
I37 It is fun to try to solve the complex ,747 ,726
problems.
Critical Thinking

I35 I am willing to learn challenging things. ,633 ,652

I39 I am proud of being able to think with a ,648 ,597
great precision.
I72 I make use of a systematic method while ,599 ,533
comparing the options at my hand and
while reaching a decision.
I21 I have problems in the demonstration of ,610 ,720
the solution of a problem in my mind.
I20 I have problems in the issue of where and ,660 ,675
how I should use the variables such as X
and Y in the solution of a problem.
I18 I cannot apply the solution ways I plan ,638 ,666
respectively and gradually.
I68 I cannot produce so many options while ,582 ,638
thinking of the possible solution ways
Problem Solving

regarding a problem.
I48 I cannot develop my own ideas in the ,674 ,532
environment of cooperative learning.
I47 It tires me to try to learn something ,604 ,494
together with my group friends in
cooperative learning.
Eigenvalue 7,19 3,39 2,54 1,80 1,34
Explainedvariance 13,5 13,1 10,7 10,1 8,7

Table 3.Standardized Regression Weights

I. No Estimate I. No Estimate
m59 <--- ,739 m38 <--- ,749
m58 <--- ,758 m37 <--- ,744
m69 <--- ,779 m35 <--- ,679

40
 ACCEPTED MANUSCRIPT

m70 <--- ,801 m39 <--- ,667

m67 <--- ,785 m72 <--- ,536
m57 <--- ,629 m21 <--- ,633
m56 <--- ,650 m20 <--- ,545
m66 <--- ,745 m18 <--- ,470
m26 <--- ,858 m68 <--- ,544
m25 <--- ,825 m48 <--- ,754
m28 <--- ,794 m47 <--- ,719
m27 <--- ,861 m42 <--- ,850
m22 <--- ,676 m46 <--- ,764
m9 <--- ,690 m41 <--- ,874

Table 4.Item-factor scores correlation analysis

F1 F2 F3 F4 F5
Creativity AlgorithmicThinking Cooperativity Critical Problem
Thinking Solving
I r I r I r I r I R
I59 ,678** I26 ,833** I42 ,889** I38 ,809** I21 ,677**
I58 ,674** I25 ,831** I46 ,837** I37 ,780** I20 ,639**
I69 ,732** I28 ,832** I41 ,861** I35 ,687** I18 ,633**
I70 ,722** I27 ,759** I45 ,788** I39 ,707** I68 ,632**
I67 ,722** I22 ,736** I72 ,681** I48 ,684**
I57 ,680** I9 ,717** I47 ,640**
I56 ,643**
I66 ,671**
N=726; **=p<, 001

Table 5.Item-factor scores corrected correlation analysis

F1 F2 F3 F4 F5
Creativity AlgorithmicThinking Cooperativity Critical Problem
Thinking Solving
I r I r I r I r I R
I59 ,546 I26 ,748 I42 ,783 I38 ,663 I21 ,496
I58 ,542 I25 ,733 I46 ,689 I37 ,629 I20 ,429
I69 ,610 I28 ,745 I41 ,731 I35 ,481 I18 ,415
I70 ,610 I27 ,604 I45 ,643 I39 ,532 I68 ,441
I67 ,608 I22 ,617 I72 ,467 I48 ,484
I57 ,528 I9 ,599 I47 ,443
I56 ,506
I66 ,537
N=726
Table 6.Item Discrimination Powers

F1 F2 F3 F4 F5
Creativity AlgorithmicThinking Cooperativity Critical Problem
Thinking Solving
I t I t I t I t I t
I59 -8,959 I26 -17,483 I42 -10,868 I38 -17,112 I21 -7,123
I58 -7,526 I25 -15,750 I46 -6,839 I37 -16,421 I20 5,486
I69 -11,515 I28 -16,287 I41 -10,582 I35 -11,646 I18 -4,610
I70 -13,939 I27 -10,677 I45 -7,757 I39 -13,032 I68 -3,197
I67 -13,461 I22 -14,293 I72 -14,597 I48 -4,309
I57 -9,661 I9 -12,106 I47 -3,234
I56 -15,075
I66 -11,526 FT -37,105
F1 -18,449 F2 -21,253 F3 -10,826 F4 -22,520 F5 -8,241

41
 ACCEPTED MANUSCRIPT

*df: 390; p<,001

Table 7.Reliability analysis results considering the whole of the scale and its factors.
Two congruent
Number Sperman Guttmann Cronbach’s
Factor halves
of items Brown Split-Half Alpha
correlation
Creativity 8 ,713 ,832 ,832 ,843
Algorithmic Thinking 6 ,756 ,861 ,,860 ,869
Cooperativity 4 ,835 ,910 ,908 ,865
Critical Thinking? 5 ,562 ,719 ,687 ,784
Problem Solving 6 ,406 ,578 ,578 ,727
Total 29 ,344 ,512 ,498 ,822

Table 8.Test-retest results of the items of the scale.

F1 F2 F3 F4 F5
Creativity AlgorithmicThinking Cooperativity Critical Problem
Thinking Solving
I r. I r I r I r I r
I59 ,612** I26 ,578** I42 ,591** I38 ,419* I21 ,412*
I58 ,578** I25 ,591** I46 ,609** I37 ,439** I20 ,398*
I69 ,621** I28 ,612** I41 ,601** I35 ,534** I18 ,432**
I70 ,453** I27 ,607** I45 ,634** I39 ,497** I68 ,476**
I67 ,498** I22 ,609** I72 ,501** I48 ,373*
I57 ,671** I9 I47 ,317*
I56 ,663**
I66 ,621** FT ,512**
F1 ,613** F2 ,598* F3 ,601* F4 ,503** F5 ,371*
N: 51; *=p<0,05: **=p<0,001

42
 ACCEPTED MANUSCRIPT

Figures

Figure 1. Screen plot graphic (eigenvalues according to the factors).

Figure 2. Confirmatory factor analysis diagram of the scale

43