Guided by the theory of interpretive validity argument, this study investigated the plausibility ... more Guided by the theory of interpretive validity argument, this study investigated the plausibility and accuracy of five sets of warrants which were deemed crucial to the validity of a self-assessment (SA) scale designed and used in a local EFL context. Methodologically, this study utilized both the Rasch measurement theory and structural equation modeling (SEM) to examine the five warrants and their respective rebuttals. Results from Rasch analysis indicated that the scale could reliably distinguish students at different proficiency levels. Among the 26 can-do statements in the SA scale, only one statement failed to fit the expectations of the Rasch model. Furthermore, each category was found to function as intended, though the first category was somewhat underused. Confirmatory factor analysis of the SA data supported the tenability of the Higher-Order Factor model which is consistent with the current view of L2 ability. Structural regression analysis revealed that the association between students' self-assessments and their scores on a standardized proficiency test was moderately strong. The multiple strands of evidence generated by various quantitative analyses of the SA data generally supported the validity of the SA scale. Future research, however, is warranted to examine other inferences in the validity argument structure, particularly in relation to the utility of the SA scale in English teaching and learning.
This study investigated test candidates’ attitudes to the Fudan
English Test (FET), a university-... more This study investigated test candidates’ attitudes to the Fudan English Test (FET), a university-based English proficiency test, and explored the relationships between test candidates’ attitudes, test taker characteristics, and test performance. The participants in this study were 157 students who had just attempted the FET. A questionnaire and an interview guide were designed to collect the data. Exploratory factor analysis extracted five attitudinal factors. Descriptive statistics at the factor level indicated that test candidates had the most positive attitudes to test administration and least positive attitudes to the mode of the computer-based speaking test. Factorial MANOVA showed that test candidates’ gender and academic backgrounds had no significant effect on their reported attitudes. Stepwise regression analyses showed that two attitudinal factors, attitude to test washback and attitude to test information explained 9.4% of the test score variance. As well as presenting the FET provider with additional evidence about the quality of the test, this study also lends empirical support to the argument that attitudinal factors need to be taken into serious consideration in test development and validation.
Despite the steadily increasing application of Rasch modeling in human
sciences, few attempts hav... more Despite the steadily increasing application of Rasch modeling in human sciences, few attempts have been made to use the Rasch measurement model to analyze Likert-type scale survey data in the field of language testing. This study investigated test takers’ attitude toward the Versant English Test (VET), a fully automated spoken English test using the Rating Scale Model (RSM) in Rasch measurement. Based on previous research, attitude in this study was conceptualized as a psychological construct consisting of three components: beliefs, opinions, and emotions. A 21-item questionnaire was designed to collect the data from 125 VET test takers. The collected data were then subjected to Rasch analysis including item statistics, reliability and separation indexes, category structure, Rasch factor analysis, and differential item functioning. The results indicated that the questionnaire was essentially unidimensional, tapping into a single attitudinal construct. The principle of measurement invariance held across the relevant subsamples. The findings of this study have implications for the VET provider in light of the further improvement and promotion of the test. Furthermore, this study also has methodological implications for researchers in the field of language testing.
Motivated by the dearth of studies investigating language learners’ listening difficulties
in EFL... more Motivated by the dearth of studies investigating language learners’ listening difficulties in EFL context, this study examined and compared teacher and learner perceptions regarding the sources of listening difficulties experienced by low-proficiency Chinese learners of English. A 38-item questionnaire and semi-structured interviews were employed to elicit the participants’ beliefs. Findings show that in general the teachers and learners share the view that text- and processing-related variables have potentially the greatest impact on L2 listeners’ comprehension, indicating the importance of giving priority to developing weak listeners’ bottom-up perception abilities before higher-level processing skills should be attempted. Discrepancy in perceptions is also identified with respect to the challenges posed by different listening task types, opening up an important area where immediate pedagogical and research efforts are needed to bring the gap to closure.
This study investigated Chinese test takers’ attitudes towards the Versant English
Test (VET), an... more This study investigated Chinese test takers’ attitudes towards the Versant English Test (VET), an automated spoken English test developed by Pearson Knowledge Technologies. Based on previous attitudinal studies, ‘attitude’ in this study was conceptualized as a hypothetical psychological construct which consisted of three components: beliefs, opinions, and emotions. In data collection and analysis, this study adopted the mixed-methods sequential explanatory design. Quantitative data were collected through a questionnaire while qualitative data were collected through follow-up semi-structured interviews and focus-group discussions. The findings of this study indicate that test takers had overall positive attitudes towards the VET, believing that the VET largely reflected their spoken English ability. Test takers’ gender and background had no significant relationship with their reported attitude. Despite the satisfactory face validity of the VET, this study also found that test takers still preferred the more traditional speaking tests (e.g., the IELTS-style live interviews) because these tests were believed to be more authentic and tap into a wider range of English speaking competencies. The findings of this study have implications for the VET provider in light of the further improvement and promotion of the test. Meanwhile, the findings are also meaningful to other providers of spoken English tests of a similar nature.
Guided by the theory of interpretive validity argument, this study investigated the plausibility ... more Guided by the theory of interpretive validity argument, this study investigated the plausibility and accuracy of five sets of warrants which were deemed crucial to the validity of a self-assessment (SA) scale designed and used in a local EFL context. Methodologically, this study utilized both the Rasch measurement theory and structural equation modeling (SEM) to examine the five warrants and their respective rebuttals. Results from Rasch analysis indicated that the scale could reliably distinguish students at different proficiency levels. Among the 26 can-do statements in the SA scale, only one statement failed to fit the expectations of the Rasch model. Furthermore, each category was found to function as intended, though the first category was somewhat underused. Confirmatory factor analysis of the SA data supported the tenability of the Higher-Order Factor model which is consistent with the current view of L2 ability. Structural regression analysis revealed that the association between students' self-assessments and their scores on a standardized proficiency test was moderately strong. The multiple strands of evidence generated by various quantitative analyses of the SA data generally supported the validity of the SA scale. Future research, however, is warranted to examine other inferences in the validity argument structure, particularly in relation to the utility of the SA scale in English teaching and learning.
This study investigated test candidates’ attitudes to the Fudan
English Test (FET), a university-... more This study investigated test candidates’ attitudes to the Fudan English Test (FET), a university-based English proficiency test, and explored the relationships between test candidates’ attitudes, test taker characteristics, and test performance. The participants in this study were 157 students who had just attempted the FET. A questionnaire and an interview guide were designed to collect the data. Exploratory factor analysis extracted five attitudinal factors. Descriptive statistics at the factor level indicated that test candidates had the most positive attitudes to test administration and least positive attitudes to the mode of the computer-based speaking test. Factorial MANOVA showed that test candidates’ gender and academic backgrounds had no significant effect on their reported attitudes. Stepwise regression analyses showed that two attitudinal factors, attitude to test washback and attitude to test information explained 9.4% of the test score variance. As well as presenting the FET provider with additional evidence about the quality of the test, this study also lends empirical support to the argument that attitudinal factors need to be taken into serious consideration in test development and validation.
Despite the steadily increasing application of Rasch modeling in human
sciences, few attempts hav... more Despite the steadily increasing application of Rasch modeling in human sciences, few attempts have been made to use the Rasch measurement model to analyze Likert-type scale survey data in the field of language testing. This study investigated test takers’ attitude toward the Versant English Test (VET), a fully automated spoken English test using the Rating Scale Model (RSM) in Rasch measurement. Based on previous research, attitude in this study was conceptualized as a psychological construct consisting of three components: beliefs, opinions, and emotions. A 21-item questionnaire was designed to collect the data from 125 VET test takers. The collected data were then subjected to Rasch analysis including item statistics, reliability and separation indexes, category structure, Rasch factor analysis, and differential item functioning. The results indicated that the questionnaire was essentially unidimensional, tapping into a single attitudinal construct. The principle of measurement invariance held across the relevant subsamples. The findings of this study have implications for the VET provider in light of the further improvement and promotion of the test. Furthermore, this study also has methodological implications for researchers in the field of language testing.
Motivated by the dearth of studies investigating language learners’ listening difficulties
in EFL... more Motivated by the dearth of studies investigating language learners’ listening difficulties in EFL context, this study examined and compared teacher and learner perceptions regarding the sources of listening difficulties experienced by low-proficiency Chinese learners of English. A 38-item questionnaire and semi-structured interviews were employed to elicit the participants’ beliefs. Findings show that in general the teachers and learners share the view that text- and processing-related variables have potentially the greatest impact on L2 listeners’ comprehension, indicating the importance of giving priority to developing weak listeners’ bottom-up perception abilities before higher-level processing skills should be attempted. Discrepancy in perceptions is also identified with respect to the challenges posed by different listening task types, opening up an important area where immediate pedagogical and research efforts are needed to bring the gap to closure.
This study investigated Chinese test takers’ attitudes towards the Versant English
Test (VET), an... more This study investigated Chinese test takers’ attitudes towards the Versant English Test (VET), an automated spoken English test developed by Pearson Knowledge Technologies. Based on previous attitudinal studies, ‘attitude’ in this study was conceptualized as a hypothetical psychological construct which consisted of three components: beliefs, opinions, and emotions. In data collection and analysis, this study adopted the mixed-methods sequential explanatory design. Quantitative data were collected through a questionnaire while qualitative data were collected through follow-up semi-structured interviews and focus-group discussions. The findings of this study indicate that test takers had overall positive attitudes towards the VET, believing that the VET largely reflected their spoken English ability. Test takers’ gender and background had no significant relationship with their reported attitude. Despite the satisfactory face validity of the VET, this study also found that test takers still preferred the more traditional speaking tests (e.g., the IELTS-style live interviews) because these tests were believed to be more authentic and tap into a wider range of English speaking competencies. The findings of this study have implications for the VET provider in light of the further improvement and promotion of the test. Meanwhile, the findings are also meaningful to other providers of spoken English tests of a similar nature.
Uploads
Papers by Jinsong Fan
English Test (FET), a university-based English proficiency test,
and explored the relationships between test candidates’ attitudes,
test taker characteristics, and test performance. The participants
in this study were 157 students who had just attempted the FET. A
questionnaire and an interview guide were designed to collect the
data. Exploratory factor analysis extracted five attitudinal factors.
Descriptive statistics at the factor level indicated that test
candidates had the most positive attitudes to test administration
and least positive attitudes to the mode of the computer-based
speaking test. Factorial MANOVA showed that test candidates’
gender and academic backgrounds had no significant effect on
their reported attitudes. Stepwise regression analyses showed
that two attitudinal factors, attitude to test washback and attitude
to test information explained 9.4% of the test score variance. As
well as presenting the FET provider with additional evidence about
the quality of the test, this study also lends empirical support to
the argument that attitudinal factors need to be taken into serious
consideration in test development and validation.
sciences, few attempts have been made to use the Rasch measurement model to
analyze Likert-type scale survey data in the field of language testing. This study
investigated test takers’ attitude toward the Versant English Test (VET), a fully
automated spoken English test using the Rating Scale Model (RSM) in Rasch
measurement. Based on previous research, attitude in this study was conceptualized
as a psychological construct consisting of three components: beliefs, opinions, and
emotions. A 21-item questionnaire was designed to collect the data from 125 VET
test takers. The collected data were then subjected to Rasch analysis including item
statistics, reliability and separation indexes, category structure, Rasch factor analysis,
and differential item functioning. The results indicated that the questionnaire was
essentially unidimensional, tapping into a single attitudinal construct. The principle
of measurement invariance held across the relevant subsamples. The findings of this
study have implications for the VET provider in light of the further improvement and
promotion of the test. Furthermore, this study also has methodological implications
for researchers in the field of language testing.
in EFL context, this study examined and compared teacher and learner perceptions regarding
the sources of listening difficulties experienced by low-proficiency Chinese learners of
English. A 38-item questionnaire and semi-structured interviews were employed to elicit the
participants’ beliefs. Findings show that in general the teachers and learners share the view
that text- and processing-related variables have potentially the greatest impact on L2 listeners’
comprehension, indicating the importance of giving priority to developing weak listeners’
bottom-up perception abilities before higher-level processing skills should be attempted.
Discrepancy in perceptions is also identified with respect to the challenges posed by different listening task types, opening up an important area
where immediate pedagogical and research efforts
are needed to bring the gap to closure.
Test (VET), an automated spoken English test developed by Pearson Knowledge
Technologies. Based on previous attitudinal studies, ‘attitude’ in this study was
conceptualized as a hypothetical psychological construct which consisted of three
components: beliefs, opinions, and emotions. In data collection and analysis, this
study adopted the mixed-methods sequential explanatory design. Quantitative
data were collected through a questionnaire while qualitative data were collected
through follow-up semi-structured interviews and focus-group discussions. The
findings of this study indicate that test takers had overall positive attitudes towards
the VET, believing that the VET largely reflected their spoken English ability. Test
takers’ gender and background had no significant relationship with their reported
attitude. Despite the satisfactory face validity of the VET, this study also found that
test takers still preferred the more traditional speaking tests (e.g., the IELTS-style live
interviews) because these tests were believed to be more authentic and tap into a
wider range of English speaking competencies. The findings of this study have
implications for the VET provider in light of the further improvement and promotion
of the test. Meanwhile, the findings are also meaningful to other providers of spoken
English tests of a similar nature.
English Test (FET), a university-based English proficiency test,
and explored the relationships between test candidates’ attitudes,
test taker characteristics, and test performance. The participants
in this study were 157 students who had just attempted the FET. A
questionnaire and an interview guide were designed to collect the
data. Exploratory factor analysis extracted five attitudinal factors.
Descriptive statistics at the factor level indicated that test
candidates had the most positive attitudes to test administration
and least positive attitudes to the mode of the computer-based
speaking test. Factorial MANOVA showed that test candidates’
gender and academic backgrounds had no significant effect on
their reported attitudes. Stepwise regression analyses showed
that two attitudinal factors, attitude to test washback and attitude
to test information explained 9.4% of the test score variance. As
well as presenting the FET provider with additional evidence about
the quality of the test, this study also lends empirical support to
the argument that attitudinal factors need to be taken into serious
consideration in test development and validation.
sciences, few attempts have been made to use the Rasch measurement model to
analyze Likert-type scale survey data in the field of language testing. This study
investigated test takers’ attitude toward the Versant English Test (VET), a fully
automated spoken English test using the Rating Scale Model (RSM) in Rasch
measurement. Based on previous research, attitude in this study was conceptualized
as a psychological construct consisting of three components: beliefs, opinions, and
emotions. A 21-item questionnaire was designed to collect the data from 125 VET
test takers. The collected data were then subjected to Rasch analysis including item
statistics, reliability and separation indexes, category structure, Rasch factor analysis,
and differential item functioning. The results indicated that the questionnaire was
essentially unidimensional, tapping into a single attitudinal construct. The principle
of measurement invariance held across the relevant subsamples. The findings of this
study have implications for the VET provider in light of the further improvement and
promotion of the test. Furthermore, this study also has methodological implications
for researchers in the field of language testing.
in EFL context, this study examined and compared teacher and learner perceptions regarding
the sources of listening difficulties experienced by low-proficiency Chinese learners of
English. A 38-item questionnaire and semi-structured interviews were employed to elicit the
participants’ beliefs. Findings show that in general the teachers and learners share the view
that text- and processing-related variables have potentially the greatest impact on L2 listeners’
comprehension, indicating the importance of giving priority to developing weak listeners’
bottom-up perception abilities before higher-level processing skills should be attempted.
Discrepancy in perceptions is also identified with respect to the challenges posed by different listening task types, opening up an important area
where immediate pedagogical and research efforts
are needed to bring the gap to closure.
Test (VET), an automated spoken English test developed by Pearson Knowledge
Technologies. Based on previous attitudinal studies, ‘attitude’ in this study was
conceptualized as a hypothetical psychological construct which consisted of three
components: beliefs, opinions, and emotions. In data collection and analysis, this
study adopted the mixed-methods sequential explanatory design. Quantitative
data were collected through a questionnaire while qualitative data were collected
through follow-up semi-structured interviews and focus-group discussions. The
findings of this study indicate that test takers had overall positive attitudes towards
the VET, believing that the VET largely reflected their spoken English ability. Test
takers’ gender and background had no significant relationship with their reported
attitude. Despite the satisfactory face validity of the VET, this study also found that
test takers still preferred the more traditional speaking tests (e.g., the IELTS-style live
interviews) because these tests were believed to be more authentic and tap into a
wider range of English speaking competencies. The findings of this study have
implications for the VET provider in light of the further improvement and promotion
of the test. Meanwhile, the findings are also meaningful to other providers of spoken
English tests of a similar nature.