Sample Size Planning and Questionnaire Length in Website-Evaluations

When planning studies to evaluate websites practitioners need to answer two central questions: (1) Which instrument should be used? (2) How many users should be queried? While the answer to the first question depends on the goal of the specific evaluation, the second question can also be answered in relation to fundamental psychometric relationships. The present study explores the relationship between the reliability of the questionnaire, and the required sample size. Based on questionnaires to assess website-content, we determine how many user have to be sampled to achieve a specific precision. Based on these results we present guidelines, that practitioners can use when planning their evaluation study.