Artificial intelligence (AI) technologies have the potential to reduce the workload for the secon... more Artificial intelligence (AI) technologies have the potential to reduce the workload for the second language (L2) teachers and test developers. We propose two AI distractor-generating methods for creating Chinese vocabulary items: semantic similarity and visual similarity. Semantic similarity refers to antonyms and synonyms, while visual similarity refers to the phenomenon that two phrases share one or more characters in common. This study explores the construct validity of the two types of selected-response vocabulary tests (AI-generated items and human expert-designed items) and compares their item difficulty and item discrimination. Both quantitative and qualitative data were collected. Seventy-eight students from Beijing Language and Culture University were asked to respond to AI-generated and human expert-designed items respectively. Students' scores were analyzed using the two-parameter item response theory (2PL-IRT) model. Thirteen students were then invited to report their test taking strategies in the think-aloud section. The findings from the students' item responses revealed that the human expert-designed items were easier but had more discriminating power than the AI-generated items. The results of think-aloud data indicated that the AI-generated items and expert-designed items might assess different constructs, in which the former elicited test takers' bottom-up test-taking strategies while the latter seemed more likely to trigger test takers' rote memorization ability.
Despite an increasing number of studies on L1 Chinese students' L2 English writing in recent year... more Despite an increasing number of studies on L1 Chinese students' L2 English writing in recent years, little research is conducted on Chinese test takers' performance in high-stakes international tests, particularly in terms of how integrated and discrete assessment may impact on their writing. This book chapter, therefore, aims to fill this gap by comparing Chinese and non-Chinese test takers' responses in different writing tasks of PTE Academic in terms of scoring and use of recurrent word combinations. As an international test of academic English, PTE Academic includes both the traditional design of independent essay writing as well as two integrated tasks of summary writing from listening or reading input. Written samples from 500 Chinese test takers and another 500 test takers of other L1s were randomly chosen from PTE Academic and analysed. The results indicate that Chinese test takers outperformed their non-Chinese peers in the tasks of read-tosummarise and essay writing. Chinese test takers also tended to write longer responses with more occurrences of recurrent word combinations, and they used more stance expressions in the essays compared with their peers. A closer examination of highly frequent word combinations, however, suggests a relationship between task type and the language elicited, regardless of test takers' L1.
This in-depth study evaluated motivation, learning strategies and demographic factors among early... more This in-depth study evaluated motivation, learning strategies and demographic factors among earlylevel Chinese HSK test-takers, predicting differences at higher test levels when Chinese character knowledge is required. 71 UK-based HSK test-takers from beginner (1/2) to intermediate (3) levels were recruited; using quantitative survey and qualitative interviews, test-taker characteristics were analysed. However, findings revealed similar patterns at all three levels, and few effects of demographic factors. Higher levels of intrinsic than extrinsic motivation, and higher use of meaningfocused learning strategies were found across levels, although learners may start with structurebased priorities at beginner levels, especially in female learners. Including pinyin made beginnerlevel HSK exams seem more accessible, but commitment to learning characters was also evident at beginner stages. Noting limitations in the HSK exam format as a test of linguistic abilities, the findings are evaluated for wider pedagogical implications for effective early-stage L2 Mandarin Chinese language development.
This study explores the assessment literacy of Mandarin Chinese teachers in the Confucius Institu... more This study explores the assessment literacy of Mandarin Chinese teachers in the Confucius Institutes (CI) in the UK. Analyses of the data from questionnaires and interviews indicate that the levels of assessment literacy among CI Mandarin Chinese teachers differ from those of local Modern Foreign Language (MFL) teachers, particularly in terms of the assessment methods used and their understanding of assessment practices. The assessment literacy characteristics of CI Mandarin Chinese teachers also vary according to their years of teaching experience and their educational backgrounds. By exploring the reasons behind these differences, this study will help future CI Mandarin Chinese teachers be better prepared for their secondment in the UK education context and reduce anxiety over working in a different system, especially when it comes to assessing their students.
The Challenge Research has shown that Content and Language Integrated Learning (CLIL) programs ca... more The Challenge Research has shown that Content and Language Integrated Learning (CLIL) programs can improve and sustain learners' motivation. However, most CLIL studies focus on the teaching and learning of English as a second/foreign language. Can the implementation of CLIL pedagogy be as effective and motivating in a program where the target language is Mandarin Chinese?
This study examined relationships between the after-school literacy activities and test performan... more This study examined relationships between the after-school literacy activities and test performance of students who passed and students who failed the 2003 administration of the Ontario Secondary School Literacy Test. It included both students who have been in the Canadian school system for most of their school lives and mostly use English as a first language (L1), and students who have recently entered the Canadian school system and use English as a second language (L2). This study demonstrates the relatively small yet complex relationship between the test performance of these different groups of students on this literacy test and their after-school reading and writing activities. (Contains 4 tables and 5 footnotes.)
The learning and teaching of Mandarin Chinese as a second or foreign language (CSL/CFL) has been ... more The learning and teaching of Mandarin Chinese as a second or foreign language (CSL/CFL) has been gaining momentum around the world, leading to a shortage of qualified Chinese teachers. Thus more research is needed into the training and professional development of Chinese teachers. This study investigates the challenges and difficulties that native Chinese-speaking teachers of Mandarin have encountered in British schools and the professional development that they desire. A questionnaire survey and semi-structured interviews were used for data collection from native Chinese-speaking teachers of Mandarin in Confucius Institutes (Hanban teachers) in the UK. Results show that participants have encountered linguistic, pedagogical and socio-cultural challenges. Major challenges reported by the Hanban teachers include lack of English proficiency and intercultural competence, lack of knowledge about the local education system, and lack of effective pedagogical methods and classroom managemen...
This study explored the language learning motivation of a small group of five university students... more This study explored the language learning motivation of a small group of five university students in the UK through the constructs of Dornyei's L2 Motivational Self System and Henry's notion of 'the ideal multilingual self'. The five participants, all successful language learners at school, chose to specialise in modern languages at university and in addition to the foreign languages they had previously learned, decided to study Chinese. Interviews were conducted in the final year of the participants' university course and their motivational changes during the course of studying Chinese were explored. The results revealed that two students who achieved remarkable success in learning Chinese came to possess a strong desire to integrate in Chinese society, linked to a longer period spent studying in the target language country. We suggest that integrative orientation is a significant component in maintaining the ideal L2 self, especially when a psychological and physical distance exists in relation to the target language, its culture and the country.
This article describes an exploratory study of the relationships between the HSK (Level 1-3) test... more This article describes an exploratory study of the relationships between the HSK (Level 1-3) test design, test questions, and the performance of UK test takers in 2019. Using quantitative analysis of the 1,833 test taker scores and qualitative interviews with seven test takers, a series of mixed method inquiries were made on the quality of HSK test quality indices, including test difficulty, reliability, and test item discrimination. In addition, the challenges encountered by this group of test takers and the reasons behind these challenges were investigated. Based on the results, suggestions were made to improve the test design to promote more effective Chinese language teaching.
The learning and teaching of Mandarin Chinese as a second or foreign language (CSL/CFL) has been ... more The learning and teaching of Mandarin Chinese as a second or foreign language (CSL/CFL) has been gaining momentum around the world, leading to a shortage of qualified Chinese teachers. Thus more research is needed into the training and professional development of Chinese teachers. This study investigates the challenges and difficulties that native Chinese-speaking teachers of Mandarin have encountered in British schools and the professional development that they desire. A questionnaire survey and semi-structured interviews were used for data collection from native Chinese-speaking teachers of Mandarin in Confucius Institutes (Hanban teachers) in the UK. Results show that participants have encountered linguistic, pedagogical and socio-cultural challenges. Major challenges reported by the Hanban teachers include lack of English proficiency and intercultural competence, lack of knowledge about the local education system, and lack of effective pedagogical methods and classroom management skills to work with disruptive students, students with special educational needs, as well as teaching assistants. The study concludes that training and professional development tailored to the local context seems imperative for this group of teachers to facilitate their effective teaching and localisation.
This study examined the relationships between students' foreign language classroom anxiety and co... more This study examined the relationships between students' foreign language classroom anxiety and cognitive test anxiety and their College English Test Band 4 (CET-4) performance. A questionnaire was distributed to 921 Chinese university students to understand the nature and degree of of the examined relationships. Follow-up interviews with twelve students were used to shed further light on uncovering mechanisms of relationships found in the survey. Results revealed three factors of anxiety, explaining 43.14% of the total variance examined in the questionnaire items. Means, standard deviations, the internal consistency for each factor, and zero-order correlations among the three factors were calculated. Correlation and multiple regression of the anxiety factors and test scores were then conducted. Results confirmed that cognitive test anxiety factor was a significant predictor of language achievement. Interview results did not fully support the relationships found in the survey. Most students did not perceive themselves to be very anxious in their university settings, either in classrooms or in testing situations. However, they did express their anxiety toward English speaking skills in the classroom. The differential perspectives of anxiety revealed from both analyses indicate that a better understanding of language classroom anxiety and cognitive test anxiety can help students and teachers to optimize their foreign language learning and teaching practices.
This study investigates the feedback practices and perceptions of lecturers and students in a UK ... more This study investigates the feedback practices and perceptions of lecturers and students in a UK university setting. To assess how lecturers give feedback in practice, 47 pieces of lecturer-written feedback were categorised into a total of 571 analytical points. Analysing the feedback from lecturers’ perspectives in terms of the value of feedback, the role of feedback and the effectiveness of feedback helps in an understanding of the rationale for and effects of feedback provision. The results of feedback analyses from students’ perspectives show the impact of individual background and intercultural communication barriers on the effective reception of feedback. Differences were identified between students and lecturers’ views of feedback as an assessment for learning tool.
This project investigated potential Aptis test-taker perceptions of its validity and test practic... more This project investigated potential Aptis test-taker perceptions of its validity and test practicality in China. Eighty-three students from Wuhan University China participated in the study and took the Aptis test. A survey design with follow-up interviews was adopted. A questionnaire was used to investigate test-taker perceptions on aspects of their test-taking experience, test items, skills measured, and some practicality issues. Semi-structured interviews were carried out to further probe participant perceptions of the test. The study was designed to compare English skills measured in Aptis to skills measured by other English tests on the Chinese market. Gaining a better understanding of test-taker perceptions of this new test can provide valuable information, informing test development, as well as test preparation practices.
This research provided a comprehensive evaluation and validation of the listening section of a ne... more This research provided a comprehensive evaluation and validation of the listening section of a newly introduced computerized test, Pearson Test of English Academic (PTE Academic). PTE Academic contains 11 item types assessing academic listening skills either alone or in combination with other skills. First, task analysis helped identify skills important for listening comprehension in academic settings. Aspects analyzed included the purpose of assessment tasks, skills/constructs assessed, and task stimuli employed in PTE Academic. The findings indicated that modern technologies enabled PTE Academic, a computer-based test, to assess students' academic listening abilities in real time using the integration of multi-modal sources. The statistical validation consisted of two stages. Exploratory factor analysis was performed first with a sample of over 5,000 students who took PTE Academic, to examine the underlying listening constructs as measured by the scores on the 11 item types item on different listening skills; these scores were subjected to Rasch analysis using CONQUEST. Second, the difficulties of the item types were estimated and the effectiveness of these item types was evaluated by calculating the information function by item type. The study has implications for test developers and test users regarding the interpretation of student performance on listening assessments.
This chapter reports Chinese university students’ perceptions of the factors that influence their... more This chapter reports Chinese university students’ perceptions of the factors that influence their English learning and test performance in CET-4. The results showed that the interviewees had a variety of motivations for learning English, from broad reasons such as contributing to China’s globalization to specific personal reasons. Interviewees recognized the importance of English in their current study, further study, and future career. They were conscious of the balance between developing English language competence and maintaining Chinese culture. CET-4 was considered mainly as a positive influence as it provides them with a direction and a focus in learning English. Understanding students’ perspectives of the factors influencing their English learning and test performance is useful in shaping future English learning, teaching and assessment practices in China.
Artificial intelligence (AI) technologies have the potential to reduce the workload for the secon... more Artificial intelligence (AI) technologies have the potential to reduce the workload for the second language (L2) teachers and test developers. We propose two AI distractor-generating methods for creating Chinese vocabulary items: semantic similarity and visual similarity. Semantic similarity refers to antonyms and synonyms, while visual similarity refers to the phenomenon that two phrases share one or more characters in common. This study explores the construct validity of the two types of selected-response vocabulary tests (AI-generated items and human expert-designed items) and compares their item difficulty and item discrimination. Both quantitative and qualitative data were collected. Seventy-eight students from Beijing Language and Culture University were asked to respond to AI-generated and human expert-designed items respectively. Students' scores were analyzed using the two-parameter item response theory (2PL-IRT) model. Thirteen students were then invited to report their test taking strategies in the think-aloud section. The findings from the students' item responses revealed that the human expert-designed items were easier but had more discriminating power than the AI-generated items. The results of think-aloud data indicated that the AI-generated items and expert-designed items might assess different constructs, in which the former elicited test takers' bottom-up test-taking strategies while the latter seemed more likely to trigger test takers' rote memorization ability.
Despite an increasing number of studies on L1 Chinese students' L2 English writing in recent year... more Despite an increasing number of studies on L1 Chinese students' L2 English writing in recent years, little research is conducted on Chinese test takers' performance in high-stakes international tests, particularly in terms of how integrated and discrete assessment may impact on their writing. This book chapter, therefore, aims to fill this gap by comparing Chinese and non-Chinese test takers' responses in different writing tasks of PTE Academic in terms of scoring and use of recurrent word combinations. As an international test of academic English, PTE Academic includes both the traditional design of independent essay writing as well as two integrated tasks of summary writing from listening or reading input. Written samples from 500 Chinese test takers and another 500 test takers of other L1s were randomly chosen from PTE Academic and analysed. The results indicate that Chinese test takers outperformed their non-Chinese peers in the tasks of read-tosummarise and essay writing. Chinese test takers also tended to write longer responses with more occurrences of recurrent word combinations, and they used more stance expressions in the essays compared with their peers. A closer examination of highly frequent word combinations, however, suggests a relationship between task type and the language elicited, regardless of test takers' L1.
This in-depth study evaluated motivation, learning strategies and demographic factors among early... more This in-depth study evaluated motivation, learning strategies and demographic factors among earlylevel Chinese HSK test-takers, predicting differences at higher test levels when Chinese character knowledge is required. 71 UK-based HSK test-takers from beginner (1/2) to intermediate (3) levels were recruited; using quantitative survey and qualitative interviews, test-taker characteristics were analysed. However, findings revealed similar patterns at all three levels, and few effects of demographic factors. Higher levels of intrinsic than extrinsic motivation, and higher use of meaningfocused learning strategies were found across levels, although learners may start with structurebased priorities at beginner levels, especially in female learners. Including pinyin made beginnerlevel HSK exams seem more accessible, but commitment to learning characters was also evident at beginner stages. Noting limitations in the HSK exam format as a test of linguistic abilities, the findings are evaluated for wider pedagogical implications for effective early-stage L2 Mandarin Chinese language development.
This study explores the assessment literacy of Mandarin Chinese teachers in the Confucius Institu... more This study explores the assessment literacy of Mandarin Chinese teachers in the Confucius Institutes (CI) in the UK. Analyses of the data from questionnaires and interviews indicate that the levels of assessment literacy among CI Mandarin Chinese teachers differ from those of local Modern Foreign Language (MFL) teachers, particularly in terms of the assessment methods used and their understanding of assessment practices. The assessment literacy characteristics of CI Mandarin Chinese teachers also vary according to their years of teaching experience and their educational backgrounds. By exploring the reasons behind these differences, this study will help future CI Mandarin Chinese teachers be better prepared for their secondment in the UK education context and reduce anxiety over working in a different system, especially when it comes to assessing their students.
The Challenge Research has shown that Content and Language Integrated Learning (CLIL) programs ca... more The Challenge Research has shown that Content and Language Integrated Learning (CLIL) programs can improve and sustain learners' motivation. However, most CLIL studies focus on the teaching and learning of English as a second/foreign language. Can the implementation of CLIL pedagogy be as effective and motivating in a program where the target language is Mandarin Chinese?
This study examined relationships between the after-school literacy activities and test performan... more This study examined relationships between the after-school literacy activities and test performance of students who passed and students who failed the 2003 administration of the Ontario Secondary School Literacy Test. It included both students who have been in the Canadian school system for most of their school lives and mostly use English as a first language (L1), and students who have recently entered the Canadian school system and use English as a second language (L2). This study demonstrates the relatively small yet complex relationship between the test performance of these different groups of students on this literacy test and their after-school reading and writing activities. (Contains 4 tables and 5 footnotes.)
The learning and teaching of Mandarin Chinese as a second or foreign language (CSL/CFL) has been ... more The learning and teaching of Mandarin Chinese as a second or foreign language (CSL/CFL) has been gaining momentum around the world, leading to a shortage of qualified Chinese teachers. Thus more research is needed into the training and professional development of Chinese teachers. This study investigates the challenges and difficulties that native Chinese-speaking teachers of Mandarin have encountered in British schools and the professional development that they desire. A questionnaire survey and semi-structured interviews were used for data collection from native Chinese-speaking teachers of Mandarin in Confucius Institutes (Hanban teachers) in the UK. Results show that participants have encountered linguistic, pedagogical and socio-cultural challenges. Major challenges reported by the Hanban teachers include lack of English proficiency and intercultural competence, lack of knowledge about the local education system, and lack of effective pedagogical methods and classroom managemen...
This study explored the language learning motivation of a small group of five university students... more This study explored the language learning motivation of a small group of five university students in the UK through the constructs of Dornyei's L2 Motivational Self System and Henry's notion of 'the ideal multilingual self'. The five participants, all successful language learners at school, chose to specialise in modern languages at university and in addition to the foreign languages they had previously learned, decided to study Chinese. Interviews were conducted in the final year of the participants' university course and their motivational changes during the course of studying Chinese were explored. The results revealed that two students who achieved remarkable success in learning Chinese came to possess a strong desire to integrate in Chinese society, linked to a longer period spent studying in the target language country. We suggest that integrative orientation is a significant component in maintaining the ideal L2 self, especially when a psychological and physical distance exists in relation to the target language, its culture and the country.
This article describes an exploratory study of the relationships between the HSK (Level 1-3) test... more This article describes an exploratory study of the relationships between the HSK (Level 1-3) test design, test questions, and the performance of UK test takers in 2019. Using quantitative analysis of the 1,833 test taker scores and qualitative interviews with seven test takers, a series of mixed method inquiries were made on the quality of HSK test quality indices, including test difficulty, reliability, and test item discrimination. In addition, the challenges encountered by this group of test takers and the reasons behind these challenges were investigated. Based on the results, suggestions were made to improve the test design to promote more effective Chinese language teaching.
The learning and teaching of Mandarin Chinese as a second or foreign language (CSL/CFL) has been ... more The learning and teaching of Mandarin Chinese as a second or foreign language (CSL/CFL) has been gaining momentum around the world, leading to a shortage of qualified Chinese teachers. Thus more research is needed into the training and professional development of Chinese teachers. This study investigates the challenges and difficulties that native Chinese-speaking teachers of Mandarin have encountered in British schools and the professional development that they desire. A questionnaire survey and semi-structured interviews were used for data collection from native Chinese-speaking teachers of Mandarin in Confucius Institutes (Hanban teachers) in the UK. Results show that participants have encountered linguistic, pedagogical and socio-cultural challenges. Major challenges reported by the Hanban teachers include lack of English proficiency and intercultural competence, lack of knowledge about the local education system, and lack of effective pedagogical methods and classroom management skills to work with disruptive students, students with special educational needs, as well as teaching assistants. The study concludes that training and professional development tailored to the local context seems imperative for this group of teachers to facilitate their effective teaching and localisation.
This study examined the relationships between students' foreign language classroom anxiety and co... more This study examined the relationships between students' foreign language classroom anxiety and cognitive test anxiety and their College English Test Band 4 (CET-4) performance. A questionnaire was distributed to 921 Chinese university students to understand the nature and degree of of the examined relationships. Follow-up interviews with twelve students were used to shed further light on uncovering mechanisms of relationships found in the survey. Results revealed three factors of anxiety, explaining 43.14% of the total variance examined in the questionnaire items. Means, standard deviations, the internal consistency for each factor, and zero-order correlations among the three factors were calculated. Correlation and multiple regression of the anxiety factors and test scores were then conducted. Results confirmed that cognitive test anxiety factor was a significant predictor of language achievement. Interview results did not fully support the relationships found in the survey. Most students did not perceive themselves to be very anxious in their university settings, either in classrooms or in testing situations. However, they did express their anxiety toward English speaking skills in the classroom. The differential perspectives of anxiety revealed from both analyses indicate that a better understanding of language classroom anxiety and cognitive test anxiety can help students and teachers to optimize their foreign language learning and teaching practices.
This study investigates the feedback practices and perceptions of lecturers and students in a UK ... more This study investigates the feedback practices and perceptions of lecturers and students in a UK university setting. To assess how lecturers give feedback in practice, 47 pieces of lecturer-written feedback were categorised into a total of 571 analytical points. Analysing the feedback from lecturers’ perspectives in terms of the value of feedback, the role of feedback and the effectiveness of feedback helps in an understanding of the rationale for and effects of feedback provision. The results of feedback analyses from students’ perspectives show the impact of individual background and intercultural communication barriers on the effective reception of feedback. Differences were identified between students and lecturers’ views of feedback as an assessment for learning tool.
This project investigated potential Aptis test-taker perceptions of its validity and test practic... more This project investigated potential Aptis test-taker perceptions of its validity and test practicality in China. Eighty-three students from Wuhan University China participated in the study and took the Aptis test. A survey design with follow-up interviews was adopted. A questionnaire was used to investigate test-taker perceptions on aspects of their test-taking experience, test items, skills measured, and some practicality issues. Semi-structured interviews were carried out to further probe participant perceptions of the test. The study was designed to compare English skills measured in Aptis to skills measured by other English tests on the Chinese market. Gaining a better understanding of test-taker perceptions of this new test can provide valuable information, informing test development, as well as test preparation practices.
This research provided a comprehensive evaluation and validation of the listening section of a ne... more This research provided a comprehensive evaluation and validation of the listening section of a newly introduced computerized test, Pearson Test of English Academic (PTE Academic). PTE Academic contains 11 item types assessing academic listening skills either alone or in combination with other skills. First, task analysis helped identify skills important for listening comprehension in academic settings. Aspects analyzed included the purpose of assessment tasks, skills/constructs assessed, and task stimuli employed in PTE Academic. The findings indicated that modern technologies enabled PTE Academic, a computer-based test, to assess students' academic listening abilities in real time using the integration of multi-modal sources. The statistical validation consisted of two stages. Exploratory factor analysis was performed first with a sample of over 5,000 students who took PTE Academic, to examine the underlying listening constructs as measured by the scores on the 11 item types item on different listening skills; these scores were subjected to Rasch analysis using CONQUEST. Second, the difficulties of the item types were estimated and the effectiveness of these item types was evaluated by calculating the information function by item type. The study has implications for test developers and test users regarding the interpretation of student performance on listening assessments.
This chapter reports Chinese university students’ perceptions of the factors that influence their... more This chapter reports Chinese university students’ perceptions of the factors that influence their English learning and test performance in CET-4. The results showed that the interviewees had a variety of motivations for learning English, from broad reasons such as contributing to China’s globalization to specific personal reasons. Interviewees recognized the importance of English in their current study, further study, and future career. They were conscious of the balance between developing English language competence and maintaining Chinese culture. CET-4 was considered mainly as a positive influence as it provides them with a direction and a focus in learning English. Understanding students’ perspectives of the factors influencing their English learning and test performance is useful in shaping future English learning, teaching and assessment practices in China.
Uploads
Papers by Ying Zheng