Analysis in experimental studies

 In computing fields, in addition to the observation of subjects

before and after treatment, experimental studies refer also
specifically a research in a laboratory using tools and materials
like algorithms
 Laboratory Experiment
 Assumption
 You have selected (or developed) tools,
techniques/algorithms based on literature
 Inputs
 The problem,
 Data ,
 algorithms/techniques,

1
 various setting options
Cont…
 Procedure
 Plan number of experiments by specifying the data input,
algorithms to be used, experimental settings like test
options …
 Write /report the results of each experiments
 Set criteria and compare the experimental results
 Output
 Accuracy/performance etc.. of an experiment
 Lessons learned, new insights, new knowledge,
 Interpretation and Description of how it is going to be
used

2
Cont…

Experiments Algorithm Test option Attributes Instances

to be used

Exp-One Randomforest 10-fold 17 10000

validation
Exp-Two J48 10 –fold 17 10000
validation
Exp-Three PART 10 –fold 17 10000
validation
Exp-Four Randomforest 80/20

Exp-Five J48 80/20

Exp-Six PART 80/20

3
Cont…
 Simulation Analysis
 Is a different flavor of laboratory experiment
 Simulation analysis is a descriptive modeling technique.
 As such, simulation analysis does not provide the explicit problem
formulation and solution steps

 Consequently, one must specify in detail a procedure for the development

and use of simulation models to assure successful outcome from a
simulation study.

 While laboratory experiment focus on measuring the impact of

variables on a target issue, simulation is measuring the
effectiveness of a given model/system in an artificial
environment
 It also involves experiments in it
4
 Cont…
 Elements of Simulation Analysis
 Problem Formulation: questions for which answer are sought, the variables
involved and measures of system performance to be used
 Data Collection : assembling the information necessary to further refine our
understanding of the problem.
 Model Development: building and testing the model of the real system,
selecting simulation tool (programming language), coding the model
and debugging it
 Model Verification and Validation: establish that the model is an
appropriate accurate representation of the real system.
 Model Experimentation and Optimization: precision issues, how large
sample (simulation time) is necessary to estimate the performance of
the system.The design of effective experiments with which to answer
the question asked in the problem formulation.
 Implementation of Simulation result: acceptance of the result by the users and
improved decision making stemming from the analysis. 5
Cont….
Major Iterative Loops in a Simulation Study
Problem Formulation

Data Collection

Model development

Model Verification and Validation

Model Experimentation and Optimization

Implementation of Simulation Results

6
Validity and Reliability
 Construct Validity
 Concepts being studied are operationalized and measured
correctly
 Internal Validity
 Does the data support the conclusion?
 External Validity
 Establish the domain to which a study’s findings can be
generalized
 Can the result be generalized?
 Experimental Reliability
 Demonstrate that the study can be repeated with the same
results
 Is the method/conduct of the research systematic and logical?
Analysis in Design Researches
 Analysis through iterative design and evaluation process
 Suggest, Design, Construct and Evaluate an artifact as a
search process
 Following standards and principles is important (eg OOSE)
 Creativity has also significant place
 Iterations and improvement has to be reported
 Recall the iterative design cycles
 May need input from mainly qualitative type of analysis but
from quantitative also

 Mainly analysis as a deductive process of checking an

abductively identified/ suggested solutions through an
iterative design, development and evaluation process

8
Data collection
Analysis
9
Main perspectives in design deductive
analysis
 Structure of the artifact: the information space the artifact spans
 basis for deducing all required information about the artifact
 determines the configurational characteristics necessary to enable the
evaluation of the artifact
 Evaluation criteria: the dimensions of the information space which
are relevant for determining the utility of the artifact
 can differ on the purpose of the evaluation
 Evaluation approach: the procedure how to practically test an artifact
 defines all roles concerned with the assessment and the way of
handling the evaluation
 result is a decision whether or not the artifact meets the evaluation
criteria based on the available information.
 Can be qualitative or quantitative

10
Methods
Structure Evaluation Evaluation approach
criteria
⚫process-based meta ⚫appropriateness ⚫laboratory research
model ⚫completeness ⚫field inquiries
⚫intended ⚫consistency ⚫surveys
applications ⚫case studies
⚫conditions of
⚫action research
applicability
⚫practice descriptions
⚫products and results
⚫interpretative
of the method
application research
⚫reference to
constructs
Models
Structure Evaluation criteria Evaluation approach
⚫ domain ⚫correcteness ⚫syntactical validation

⚫ scope, purpose ⚫completeness ⚫integrity checking

⚫ syntax and ⚫clarity ⚫sampling using

semantics ⚫flexibility selective matching of
⚫ terminology ⚫simplicity
data to actual external
⚫ intended
phenomena or trusted
⚫applicability
application surrogate
⚫implementability
⚫integration tests

⚫risk and cost analysis

⚫user surveys
Instantiations
Structure Evaluation Evaluation approach
criteria
⚫executable implementation in ⚫functionality ⚫Code inspection
a programming language ⚫usability ⚫Testing
⚫reference to a design model ⚫reliability ⚫Code analysis
⚫reference to a requirement ⚫performance ⚫Verification
specification ⚫supportability ⚫Accptance study
⚫reference to the
⚫Usability analysis
documentation
⚫reference to quality
management documents
⚫reference to configuration
management documents
⚫reference to project

⚫management documents
Validity and reliability in deign research
 Pragmatic Utility through scientific methods
 Does it solve the initial problem?
 Was it correct and repeatable
 Is it acceptable by the respective beneficiaries
 Rigorous evaluation

14
 Interpretation and Discussion

15
Interpretation and Discussion
 In strict sense
 Interpretation is telling what the result means
 Discussion is telling how good/better or related the result is
compared to other works.
 Explain the results in light of
 Previous literatures and theories.
 Own RQ and/or objectives
 No clear distinction in case of qualitative
 Involves reporting results of demonstration and evaluation in
case of design research
 May be required to collect data and analysis with the same
procedures as we have seen before
16
 Interpreting the Data
 Interpreting the data means several things, in particular,
it means:

 Relating the findings to the original research problem

and to the specific research questions and hypotheses.
 Researchers must eventually come full circle to their
starting point – why they conducted a research study in
the first place and what they hoped to discover – and
relate their results to their initial concerns and questions
Cont…
 Relating the findings to preexisting literature,
concepts, theories, and research studies.
 To be useful, research findings must in some way be
connected to the larger picture – to what people
already know or believe about the topic in question.

 Perhaps the new findings confirm a current

theoretical perspective.

18
Cont…
 Determining whether the findings have practical
significance as well as statistical significance.
 Statistical significance is one thing; practical
significance – whether findings are actually useful – is
something else altogether.

 Identifying limitations of the study. Finally,

interpreting the data involves outlining the weaknesses
of the study that yielded them.
 No research study can be perfect, and its imperfections
inevitably cast at least a hint of doubt on its findings.
Good researchers know – and they also report – the
weaknesses along with the strengths of their research
Example: Types of results and their discussion
may take two forms in design science research

20
TOOLS OF RESEARCH- in relation to data
collection and analysis
 Tools are chosen to facilitate research tasks
 Some researchers need special tools indigenous to particular discipline
 The concern here is with the general tools of research that the
majority of researchers, regardless of discipline and situation,
typically need to derive meaningful and insightful conclusions from
the data they collect
 Be careful not to equate tools of research with the methodology of
research
 There are six general tools of research:
1. The library and its resources
2. The computer and its software
3. Techniques of measurement
4. Statistics
5. The human mind
6. Language
1. The Library and Its Resources

 Scholar should know its principal resources and understand its

classification system, and find the shortest route to the
information it contains.
 Both physical and online databases
 You learn the library by using the library
 Feel the mood of research in a library
 Libraries have manual : Learn where the various holdings are
located
 Catalogue is the heart of the library – books , films, filmstrip, tapes,
phonograph records, microfilm, maps, pictures, slides, CDs, …
2. The Computer and its Software – Tool of
Research
 Software tools
 Weka, NLTK, Ns-3 and other networking monitoring tools, etc…
 Be familiar with at least few selected tools in the area of research
you are engaged
 Use appropriate tools with logical settings.
 Taking advantage of the Internet
 Web site: journals, publishers, organizations, individuals, etc.
 Search engine: googlescholar , yahoo, Alta Vista, etc.
 E-Mail (Asking questions to authors, experts, etc., Facilitate
collaboration among people, Attached file (reports, etc.)
 News (List servers: E-discussion group., Many groups with
particular interests)
3. Measurement – Tool of Research

 Researchers strive for objectivity: not influenced by own

perceptions, impressions, and biases.
 Therefore, must identify systematic way of measuring a
phenomenon
 In most case numerically
 But also qualitatively
 Old proverb – if it exists, then it can be measured
 If it is researchable, then data must be measurable
4. Statistics as a Tool of Research
 All tools are MORE suitable for some purposes than for others.
 Example : screw driver was designed to INSERT and remove screws –
BUT people often used it for punching holes, scratch away paints,
etc…..misuse……
 So, too, with statistics- Statistics can be a powerful tool when used
correctly (for specific kind of data & research questions)
 BUT can be misleading when applied in other contexts.
 More useful in some academic disciplines than in others.
 REMEMBER, the statistical values obtained are never the end of research
nor the final answer to research problem.
 The final question is “What do the data indicate” not what is their
numerical configuration.
 Statistics give information about the data BUT a conscientious
researcher is not satisfied until the MEANING of this information is
revealed.
5. The Human Mind – Tool of Research
 Statistics can tell us the center, the spread, relationship of data BUT
cannot interpret and arrive at a logical conclusion or meaning.
 Only mind can do.
 Mind is the most important tool.
 Nothing equals its powers of comprehension, integrative reasoning
and insight
 Strategies to make use of the human mind to better understand
include :
1. Deductive logic
2. Inductive reasoning
3. Scientific method
4. Critical thinking
5. Collaboration with others
Cont…
 Critical Thinking
 Good researchers engage in critical thinking.
 Involves evaluating information or arguments in terms of
their accuracy and worth
 As an example
 During LR don’t just accept research findings and theories at
face value.
 Scrutinize for faulty assumptions, questionable logic,
weaknesses in methodology, inappropriate statistical analyses,
and unwarranted conclusions.
Cont…
 Critical Thinking: Take a variety of forms, depending on the
context.
 Verbal reasoning – Understanding and evaluating the persuasive
techniques found in oral and written language.
 Argument analysis – Discriminating between reasons that do and
do not support a particular conclusion.
 Decision making – Identifying and judging several alternatives and
selecting the best alternative.
 Critical analysis of prior research.: Evaluating the value of data and
research results in terms of the methods used to obtain them and
their potential relevance to particular conclusion.
Cont…
 Collaboration with Others
 More heads are better than one.
 A researcher has certain perspectives, assumptions, and
theoretical biases – not to mention holes in knowledge
about subject matter – that limit research approaches of a
project.
 Need to bring colleagues who have perspectives,
backgrounds, and areas of expertise somewhat different –
more cognitive resources to tackle research problem and
how to find meaning.
6. Language as a Tool of Research

 Human kind’s greatest achievements – facilitate

communication and think effectively.
 Can think more clearly and efficiently when can represent
thoughts with specific WORDS and PHRASES
 Good words, even a simple one, can
1. Reduce world’s complexity
2. Facilitate generalization and inference drawing in new
situation.
3. Allow abstraction of the environment
4. Enhance the power of thought
Exercise
 Relating research problems, objectives and results
 Problem statements should lead to defining a clear
objectives
 Objectives should lead to clear results and interpretations

 Read the following problem descriptions and

1. Craft a candidate general objectives for both.
2. Suggest possible/logical result of the studies and relevant
interpretations expected

31
32
33
34
Review questions
 Explain the central concept in qualitative analysis
 Describe the difference between survey experiment and
laboratory experiment methods from IT research perspective
 Explain the essence of analysis in design science research
 What is the purpose of factor analysis?
 Describe knowledge contribution of a typical Design science
research
 How do you interpret your research findings?
 Enumerate at least three major challenges in data collection
phase of a research.

35
 Make sure that your data
collection and analysis is
logical!!!

36