Modern technology allows for surveying through different media (e.g. Internet, mobile phones, tab... more Modern technology allows for surveying through different media (e.g. Internet, mobile phones, tablets), which may influence the quality of collected data with additional effects due to the survey mode and should be pretested with the purpose of avoiding effects that would deteriorate the quality of collected data. In the present study, we analysed the technological development of surveying tools by testing the applicability of cognitive interviews on several different survey modes. We focused on the importance of the cognitive interviewer and the effect of reducing their degree of involvement on the quality of the interview results. We carried out personal interviews, interviews using voice-over-Internet protocol, interviews using programs for instant messaging, and web-based interviews; these enabled us to analyse the quality of each survey mode and recognize their advantages and deficiencies. Through the comparison of these modes and their assigned technique, we showed that the role of a cognitive interviewer is important for the quality of interviews regardless of the degree of their involvement in the survey process. However, the requirement for the pretesting situation to resemble the actual final process leads to a necessity to develop new, enhanced approaches to cognitive interviewing on trending survey modes.
Cognitive interviews are a useful tool for questionnaire pretesting. However, detailed informatio... more Cognitive interviews are a useful tool for questionnaire pretesting. However, detailed information on the proper way of performing them is scarce–researchers use this method without agreement on applicable techniques or specific rules on their realization. Additionally, users of cognitive interviewing techniques only seldom explain the pretesting results in their research, leaving it to readers to trust their outcomes. Follow-up observations on the success of the interviews and the overall experience are rarely, if ever, presented. We were interested in the learnability of the techniques when performed by non-experienced, newly instructed interviewers; moreover, we wanted to understand the most common problems they had experienced. During a five-year period, 120 methodology students performed 612 cognitive interviews and analyzed 17 different survey questionnaires. The precise documentation of their assignment served as a detailed database of qualitative and quantitative information on their experiences. The results present the first-time interviewers’ ability to accurately perform and analyze a cognitive interview. We show the most common mistakes and issues in all stages of the interviewing process, the influence of different interviewers’ and respondents’ characteristics, and the effect of the technique on the interview’s success.
Researchers improve the quality of survey questions pretesting questionnaires before using the it... more Researchers improve the quality of survey questions pretesting questionnaires before using the items on the chosen sample. One of the pretest methods that has emerged in the last 30 years is cognitive interviewing. Its purpose is to study respondent's mental processes while answering the survey questions. In the article, we provide an overview of the development of cognitive laboratory methods and of the techniques of cognitive interviewing; in addition, we analyze and evaluate the classifications of these methods. Following a systematic review, we assess the key disadvantages of the development and use of cognitive laboratory methods.
Survey Methods: Insights from the Field, Jan 25, 2013
With the decrease of landline phones in the last decade, telephone survey methodologists face a n... more With the decrease of landline phones in the last decade, telephone survey methodologists face a new challenge to overcome coverage bias. In this study we investigate coverage error for telephone surveys in Europe over time and compare two situations: classical surveys that rely on landline only with surveys that also include mobile phones. We analyzed Eurobarometer data, which are collected by means of face-to-face interviews and contain information on ownership of landline and mobile phones. We show that for the period 2000-2009, time has a significant effect on both mobile phone penetration and coverage bias. In addition, the countries’ development significantly affects the pace of these changes.
Modern technology allows for surveying through different media (e.g. Internet, mobile phones, tab... more Modern technology allows for surveying through different media (e.g. Internet, mobile phones, tablets), which may influence the quality of collected data with additional effects due to the survey mode and should be pretested with the purpose of avoiding effects that would deteriorate the quality of collected data. In the present study, we analysed the technological development of surveying tools by testing the applicability of cognitive interviews on several different survey modes. We focused on the importance of the cognitive interviewer and the effect of reducing their degree of involvement on the quality of the interview results. We carried out personal interviews, interviews using voice-over-Internet protocol, interviews using programs for instant messaging, and web-based interviews; these enabled us to analyse the quality of each survey mode and recognize their advantages and deficiencies. Through the comparison of these modes and their assigned technique, we showed that the role of a cognitive interviewer is important for the quality of interviews regardless of the degree of their involvement in the survey process. However, the requirement for the pretesting situation to resemble the actual final process leads to a necessity to develop new, enhanced approaches to cognitive interviewing on trending survey modes.
Cognitive interviews are a useful tool for questionnaire pretesting. However, detailed informatio... more Cognitive interviews are a useful tool for questionnaire pretesting. However, detailed information on the proper way of performing them is scarce–researchers use this method without agreement on applicable techniques or specific rules on their realization. Additionally, users of cognitive interviewing techniques only seldom explain the pretesting results in their research, leaving it to readers to trust their outcomes. Follow-up observations on the success of the interviews and the overall experience are rarely, if ever, presented. We were interested in the learnability of the techniques when performed by non-experienced, newly instructed interviewers; moreover, we wanted to understand the most common problems they had experienced. During a five-year period, 120 methodology students performed 612 cognitive interviews and analyzed 17 different survey questionnaires. The precise documentation of their assignment served as a detailed database of qualitative and quantitative information on their experiences. The results present the first-time interviewers’ ability to accurately perform and analyze a cognitive interview. We show the most common mistakes and issues in all stages of the interviewing process, the influence of different interviewers’ and respondents’ characteristics, and the effect of the technique on the interview’s success.
Researchers improve the quality of survey questions pretesting questionnaires before using the it... more Researchers improve the quality of survey questions pretesting questionnaires before using the items on the chosen sample. One of the pretest methods that has emerged in the last 30 years is cognitive interviewing. Its purpose is to study respondent's mental processes while answering the survey questions. In the article, we provide an overview of the development of cognitive laboratory methods and of the techniques of cognitive interviewing; in addition, we analyze and evaluate the classifications of these methods. Following a systematic review, we assess the key disadvantages of the development and use of cognitive laboratory methods.
Survey Methods: Insights from the Field, Jan 25, 2013
With the decrease of landline phones in the last decade, telephone survey methodologists face a n... more With the decrease of landline phones in the last decade, telephone survey methodologists face a new challenge to overcome coverage bias. In this study we investigate coverage error for telephone surveys in Europe over time and compare two situations: classical surveys that rely on landline only with surveys that also include mobile phones. We analyzed Eurobarometer data, which are collected by means of face-to-face interviews and contain information on ownership of landline and mobile phones. We show that for the period 2000-2009, time has a significant effect on both mobile phone penetration and coverage bias. In addition, the countries’ development significantly affects the pace of these changes.
Uploads
Papers by Anja Mohorko