Are Those Rose-Colored Glass you are Wearing? Student and Alumni Survey Responses

RESEARCH & PRACTICE IN ASSESSMENT Abstract Student surveys are often important elements of assessment in higher education, but alumni surveys can play a substantial role as well. However, little is known about how responses from these two groups compare to one another. Combining data from the Strategic National Arts Alumni Project (SNAAP) and the National Survey of Student Engagement (NSSE), this study examines self-reported college experiences and skill development of seniors and alumni who majored in the arts. Results suggest that alumni rate their overall experience higher, while students judge specific aspects of their institutional experience and their skill development more positively. Given these differences, it is recommended that institutions survey both students and alumni to achieve a more complete picture of the educational experience. AUTHORS Amber D. Dumford, Ph.D. Indiana University Angie L. Miller, Ph.D. Indiana University Are those Rose-Colored Glasses You are Wearing? Student and Alumni Survey Responses A s the economy slowly emerges from the struggle of recession and funding to higher education institutions continues to be cut, there is an increasing trend for requiring colleges and universities to show measures of their effectiveness (Kuh & Ewell, 2010). Using surveys to assess skill development and the quality of collegiate experiences has become commonplace (Kuh & Ikenberry, 2009; Porter, 2004), but much of that research uses current or graduating students to collect information. In fact, the vast majority (85%) of U.S. colleges and universities use some type of national student survey in their assessment plan (Kuh, Jankowski, Ikenberry, & Kinzie, 2014). Yet students are not the only source of information that could be of use for institutions that are determined to provide evidence of their value and success. Other stakeholders can contribute relevant assessment information as well. An increasing number of institutions are turning to alumni surveys, focus groups, and interviews to gain a unique perspective on learning and other outcomes (Borden & Kernel, 2013; Kuh et al., 2014). One important measure of institutional effectiveness is alumni success in the workplace (Cabrera, Weerts, & Zulick, 2005). Not only do those at higher education institutions have to show evidence of their effectiveness to state funding and accreditation CORRESPONDENCE agencies, but students are also aware that in the current economy their employment prospects may be constrained and they are concerned with getting the best return on their Email academic investment in the form of employability. With these things in mind, perhaps the anglmill@indiana.edu viewpoints of alumni who are already in the field or struggling to enter their field would be even more enlightening than those of students still in their programs. However, little is known about how undergraduate student responses compare with those of alumni. Does the passage of time change the capacity of people to reflect on their learning experiences during college? Volume Ten | Winter 2015 5 RESEARCH & PRACTICE IN ASSESSMENT Literature Review With these things in mind, perhaps the viewpoints of alumni who are already in the field or struggling to enter their field would be even more enlightening than those of students still in their programs. In addition to the pure content knowledge gained in a student’s chosen major, administrators, faculty, and staff at institutions of higher education claim to prepare their students with a multitude of skills, ranging from effective communication practices to analytical and creative thinking skills (Tait & Godfrey, 1999). Although not all skills learned in higher education settings may transfer directly to the workplace (Stasz, 2001), those at institutions must make every effort to prepare students to be suitable employees. A major function of higher education is to help students develop skills that will lead them to success in the workplace (Evers, Rush, & Berdrow, 1998; Stasz, 2001). While some acquired skills are considered discipline-specific, many “transferable skills” that will lead to workplace success, such as problem solving and effective communication, are applicable to a broad range of fields (Bradshaw, 1985; Stasz, 1997). There is a need for generic skills across multiple types of jobs, and students possessing them appear more marketable to potential employers. The Association of American Colleges and Universities has recently addressed many of these skills, including critical and creative thinking, inquiry and analysis, and written and oral communication, as essential learning outcomes for higher education, hoping to encourage deliberate progress in their development. If curriculum and programming at institutions are lacking in these areas, the employability of their graduates will decrease (Evers et al., 1998). Alumni surveys can provide direct information on career attainment, as alumni can report back to the institution not only their current job(s) and income, but how useful the skills they learned at their institution are to their current occupation and how their educational experiences may have shaped the development of these skills and competencies. Because of the need to develop such a range of different skills, many higher education institutions have begun to scrutinize whether or not they are effectively teaching these skills in their curriculum, and alumni surveys can provide this type of information. As there is increasing pressure for colleges and universities to shorten the time in which it takes students to earn their degrees, some aspects of the curriculum must be cut. Multiple perspectives on the importance of a variety of skills can help departments prioritize their required course content. If arts programs are to address these criticisms concerning skill development, collecting information from current students as well as alumni is an instrumental aspect of curricular modification. 6 Volume Ten | Winter 2015 Although alumni can provide an abundance of important information, logistical issues are involved in the surveying of alumni. While student populations are considered a more captive audience with the confidence of accurate email addresses, alumni populations are less defined. Alumni surveys also often have lower response rates compared with student surveys (Smith & Bers, 1987), for a variety of reasons including outdated contact information, suspicion of money solicitation, and decreased institutional loyalty after graduation. Indeed, response rates across a variety of groups have been falling over the past decade (Atrostic, Bates, Burt, & Silberstein, 2001; Baruch, 1999; Porter, 2004). One must also be aware of the increasing demands of technology when it comes to survey research. Individuals are often encumbered with endless requests to complete online surveys, and while their internet access is virtually unlimited and enables flexibility in the location of completing surveys, taking surveys on smartphones and tablets can be additionally burdensome (Buskirk & Andrus, 2012; Lambert & Miller, 2015; Mavletova, 2013). These new issues further add to the complexity of surveying alumni. Nevertheless, it is imperative that administrators at higher education institutions acquire knowledge from their alumni. Arts programs are one disciplinary area that has been under fire for a lack of preparation in skills needed for the “real world” of work, and it is often difficult to align some of the arts curriculum with rigid accountability standards that may not take into account the unique skills and experiences of arts students (Johnson, 2002). One study found that practical business and management-related skills were greatly underemphasized within arts curricula (Bauer, Viola, & Strauss, 2011), and artists themselves recognize the need for “learning on the fly” and the power of networking and similar smart career mindsets (Smilde, 2008). Conversely, there is also research to suggest that students in the arts are especially adept at certain types of skills, including incorporating verbal studio feedback into revisions of their work (Edstrom, 2008) and critical thinking and interpersonal understanding (Badcock, Pattison, & Harris, 2010). If arts programs are to address these RESEARCH & PRACTICE IN ASSESSMENT criticisms concerning skill development, collecting information from current students as well as alumni is an instrumental aspect of curricular modification. Furthermore, arts programs in particular have recently been under scrutiny for the career outcomes of their graduates. Data indicate that those majoring in the arts have some of the lowest income levels, especially among recent college graduates (Carnevale, Cheah, & Strohl, 2012), and arts majors are widely considered in the popular press to be “worthless” in terms of income and employment (Cantor, 2012). Institutions can combat this accusation with alumni data. In addition to simply reporting income and employment status, it may be helpful to use alumni data in expanding the definition of what a “successful” graduate looks like. Research suggests that other aspects of one’s career, such as opportunities to be creative or contribute to the greater good, can provide just as much, if not more, of a rewarding experience as can the traditional measures of income and prestige (Lambert & Miller, 2013). This may be particularly pertinent in fields such as the arts or education, which are not generally associated with higher career earnings. Thus, especially when looking at the arts, alumni views of their educational experiences might shed some light on the true value of their time at their institutions. The current study compares information from an arts alumni survey and a survey of graduating seniors to explore how the views on the experiences of the two groups may differ and strengthen one another. Thus, especially when looking at the arts, alumni views of their educational experiences might shed some light on the true value of their time at their institutions. Research Questions Given the need for student and alumni surveys in higher education assessment, the purpose of this study is to explore the relationship between student and alumni views. The following general research questions guided this study: 1. Are there differences in how students and alumni perceive aspects of their institutional experiences and the skills and competencies that they acquire at their institutions? 2. What are the implications of interpreting alumni reports as unbiased assessments of strengths and weakness of a program? Conversely, do alumni evaluate their institutions with “rose-colored glasses” and cast things in a positive light, or do they evaluate their education more harshly once they gain a more practical knowledge of the working world? 3. Finally, if differences between students and alumni do exist, whose report should be given precedence in making curricular or programming assessments and changes? Should institutions give more weight to student reports that have the accuracy of closeness in time to the experience, or those reports of alumni that have the advantage of pragmatic perspective and hindsight? Methodology To address these questions, this study used data from the Strategic National Arts Alumni Project (SNAAP) and the National Survey of Student Engagement (NSSE). SNAAP is an online annual survey of arts graduates from a broad spectrum of institutions, including independent colleges of art and design, music conservatories, and arts schools, departments, or programs at comprehensive colleges and universities. The arts are defined broadly to include a range of fields such as music, theatre, dance, design, architecture, creative writing, film, media arts, illustration, and fine arts. SNAAP surveys alumni on a wide range of content, including formal education and degrees, institutional experiences, postgraduate resources for artists, past and current career information, avocational arts engagement, income and debt, and demographic information. The 2011 SNAAP administration included over 36,000 total respondents at 66 participating institutions. Participants were sent an invitation email including a link to the survey with a unique identification number. Participants could log in to their unique link multiple times, so they were not constrained to respond to all survey questions during a single sitting. However, the unique link tracking system ensured that participants could only submit their completed survey once. The median completion time was 22 minutes. Volume Ten | Winter 2015 7 RESEARCH & PRACTICE IN ASSESSMENT NSSE is an annual online survey of first-year and senior students that gives a snapshot of college student experiences inside and outside of the classroom. The items on NSSE gather information on the extent to which students engage in and are exposed to educational experiences that represent good practices related to desirable college outcomes. The 2012 NSSE administration included over 285,000 respondents at 546 institutions. The median completion time for the core NSSE survey was 13 minutes. Each year, experimental item sets are appended to the end of the core NSSE survey. As part of the 2012 NSSE administration, a set of experimental items asked first-year and senior students at selected institutions about skills and experiences that matched questions on the SNAAP questionnaire. Sample When thinking back to their institutional experience as a whole, it may be that alumni are viewing it through rose-colored glasses. For the purposes of this study, only data from those institutions that participated in both the 2011 SNAAP administration and the additional item set on the 2012 NSSE administration were used. SNAAP is administered in the fall, while NSSE has a spring administration. Therefore, these two data sources were collected at the closest points in time to one another, compared to other years of survey data from either project. The sample consisted of more than 222 seniors and 593 recent undergraduate alumni (graduating between 2001 and 2010) at six different four-year institutions. The seniors were selected based on reporting an arts major in one of the corresponding SNAAP arts programs of participation. The alumni cohorts of 2001 to 2010 were chosen because their experiences were closer to those of the graduating seniors, and no major curricular changes had occurred in those years at these six participating institutions. As with most survey research, females responded at a higher rate than their male counterparts. Nearly two-thirds of both the graduating senior and alumni respondents were female (72% and 61% respectively). In contrast, the race of respondents was similar to the population of these six institutions (73% white for NSSE and 70% for SNAAP), with the only exception being that Asian respondents were slightly over-represented for SNAAP respondents (5%). About one-third of the respondents were first-generation students (37% and 30%) and nearly all respondents were U.S. citizens (98% for both surveys). The response rates for the six institutions ranged from 14% to 25% for SNAAP and 27% to 51% for NSSE, with an average institutional response rate of 19% for SNAAP and 34% for NSSE. Measures It may also be informative to borrow some concepts from cognitive psychology in a further discussion of how students and alumni rely on memory searches to respond to survey items. 8 Volume Ten | Winter 2015 The measures that are the focus of this study are taken from one individual item and two additional item sets. The first question asked students and alumni to give an overall rating of their institutional experience on a 4-point Likert scale ranging from Poor to Excellent. This question is on the core survey for both NSSE and SNAAP. In contrast, the next two sets were developed for SNAAP and are on the SNAAP core survey, but were added to NSSE as additional questions appearing at the end of the core NSSE survey. The second set of questions asked participants to rate their satisfaction with nine aspects of their time at the institution, including academic advising, opportunities for degree-related internships or work, instructors, sense of belonging and attachment, and opportunities to network with alumni and others. The set was on a 4-point Likert scale from Very dissatisfied to Very satisfied with an additional Not relevant option. For the purposes of this study, the Not relevant responses were removed from the data to create ordinal variables. Finally, the third question set asked about 16 different skills and competencies developed at their institution. Participants were asked, “How much did [your institution] help you acquire or develop each of the following skills and abilities?” and provided responses using a 4-point Likert scale with the end points of Not at all to Very much. The skills and competencies used included critical thinking, broad knowledge and education, creative thinking, research skills, persuasive speaking, project management skills, technological skills, artistic technique, financial and business management skills, leadership skills, networking and relationship building, and teaching skills. All skills and aspects of time at institution included in the question sets are listed in Table 1. The demographic variables of gender, race, citizenship status, and parent education were included on both survey instruments as well. RESEARCH & PRACTICE IN ASSESSMENT Table 1 Comparison of Graduating Seniors and Alumni on Institutional Experiences and Development of Skills Adjusted Meansa Student Overall Experience 3.27 Aspects of Time at Institution Opportunities to present, perform, or exhibit your work 3.30 Opportunities to take classes outside of your 3.12 major/discipline Instructors in classrooms, labs, and studios 3.37 Academic advising 2.99 Advising about career or further education 2.80 Opportunities for degree-related internships or work 2.68 Opportunities to network with alumni and others 2.71 Sense of belonging and attachment 3.09 Freedom and encouragement to take risks 3.17 Skills and Abilities Critical thinking and analysis of arguments and 3.41 information Broad knowledge and education 3.30 Listening and revising 3.49 Creative thinking and problem solving 3.59 Research skills 3.30 Clear writing 3.21 Persuasive speaking 2.96 Project management skills 3.21 Technological skills 3.23 Artistic Technique 3.71 Financial and business management skills 2.24 Entrepreneurial skills 2.23 Interpersonal relations and working collaboratively 3.18 Leadership skills 3.05 Networking and relationship building 3.07 Teaching skills 2.86 a Adjusted for gender, race, U.S. citizenship status, and first-generation status. * p < .05, ** p < .01, *** p < .001 Alumni 3.39 * Effect Size (d) .17 * *** ** -.16 -.35 -.27 ** *** * * -.23 -.30 -.21 -.21 *** ** -.38 -.27 * *** -.21 -.28 Sig. 3.29 3.25 3.37 2.84 2.44 2.41 2.61 3.19 3.16 3.34 3.30 3.44 3.53 3.11 2.96 2.78 3.02 3.12 3.63 1.92 1.99 3.17 2.88 2.83 2.73 Data Analysis Analysis of covariance (ANCOVA) was conducted to determine whether differences of reported satisfaction and skill development exist between graduating seniors and alumni. Prior to the estimation of the models, exploratory analyses were conducted testing the assumptions underlying the application of ANCOVA and all were met (Glantz & Slinker, 2001). Using SNAAP data from the previous fall also guaranteed that no NSSE respondents would be eligible for participation in SNAAP after their graduation, which would violate the independent samples assumption of the statistical analyses. The adjusted means are reported for each of the groups, along with the statistical significance of the difference between the two groups. The statistical software used (Statistical Package for the Social Sciences v20.0) automatically implements a corrected formula to account for unequal sample sizes. Next, effect sizes (standardized mean differences using Cohen’s d for ANCOVAs, calculated by dividing the adjusted mean difference by the square root of the mean square error) were calculated to determine the magnitude of the graduating senior and alumni differences. The effect size with controls represents how much of the raw difference is left unexplained after adjusting the means for student and alumni characteristics. Control variables included gender, race, U.S. citizenship status, and first-generation status, as previous research (Pascarella & Terenzini, 2005) suggests that there are differences in student engagement and educational experiences for students based on these characteristics. Because these variables are categorical, they were dummy-coded prior to inclusion in the analyses. Taken together, the general pattern suggested in these results is that alumni provide more positive evaluations of their institutions overall, yet more critical judgments when certain specific aspects are concerned. Results Comparison of the ratings of their overall institutional experience suggests that alumni give higher general appraisals than their graduating senior counterparts when evaluating their educational experience as a whole (p < .05, Cohen’s d = .17). Using the adjusted means, Volume Ten | Winter 2015 9 RESEARCH & PRACTICE IN ASSESSMENT While it is hard to determine which group has a more accurate report of the experience, important institutional information can be gained through surveying both students and alumni. Students may be better able to provide information about affective components of their experience, while alumni may be better judges of specific things needed in the workplace. significant differences were found for three of the nine specific aspects of time at institution (academic advising, career advising, and opportunities for internships). In contrast with the overall institutional experience evaluation, these results suggest that alumni give lower specific appraisals for particular aspects (Cohen’s d = -.16, d = -.35, d = -.27, respectively). Adjusted means comparisons for the amount of institutional contribution to acquired skills and competencies show a similar pattern, with alumni giving statistically significant, lower appraisals for 8 of the 16 skills (Cohen’s d ranging from -.21 to -.38). The skills with significantly lower ratings were clear writing, persuasive speaking, networking and relationship building, leadership skills, research skills, project management, financial and business skills, and entrepreneurial skills. All ANCOVA results are shown in Table 1. Discussion When thinking back to their institutional experience as a whole, it may be that alumni are viewing it through rose-colored glasses. The arts alumni included in this study tended to rate their institutions slightly more favorably than the senior students graduating with arts majors when making universal assessments. However, when considering more nuanced aspects of their educational experiences, alumni perceptions may have a more lackluster pallor. In terms of their satisfaction with aspects of their time at the institution, post-graduation experiences in the workplace may better enable alumni to reflect on certain aspects and realize where improvements could help them in their current careers. In particular, alumni were less satisfied than graduating seniors in the areas of academic advising, career advising, and opportunities for internships or degree-related work. Applying the old adage of “hindsight is 20/20,” it may be the case that as students, respondents do not realize that they need better advising or an internship until they enter the workforce and then gain a more realistic perception. This highlights the importance of surveying both students and alumni as part of an institutional assessment plan. While student surveys may be easier, in terms of a readily available population, they may not always provide the most insightful or reflective information. In addition to this more complex understanding of satisfaction with certain aspects of their time, alumni may also learn they needed to have better developed skills only once they have gained work experience. Alumni were less satisfied than graduating seniors with their institution’s contribution to their development of clear writing, persuasive speaking, networking and relationship building, leadership skills, research skills, project management, financial and business skills, and entrepreneurial skills. These results could be interpreted to mean that upon leaving the institution and entering the workforce, alumni perceptions shift in terms of some communicative and procedural skills. Writing, speaking, networking, and leadership are important aspects of communication that may be experienced differently in an applied setting, such as the workplace, in comparison to a classroom situation. Likewise, some task-based procedural skills like research, project management, finance, and entrepreneurship may also be more completely comprehended and valued once an individual transitions from student to employee. When current senior students answer that their institution has contributed “very much” to the development of a certain skill, it may be that they are referencing their development since their first year at the institution and think that they have made great strides. There is also the possibility that once alumni enter the workforce, they are referencing their skill levels in comparison with colleagues who are quite advanced in these skills resulting from years, or perhaps even decades, of actual use. 10 Volume Ten | Winter 2015 It may also be informative to borrow some concepts from cognitive psychology in a further discussion of how students and alumni rely on memory searches to respond to survey items. In responding to an item about overall satisfaction with their institution, people may use a heuristic recall strategy, which quickly scans through all associated memories, seeking the most relevant cases (Reisberg, 2012). This strategy is substantially different from an algorithmic one, which systematically evaluates all possible steps of a procedure (Davis & Palladino, 2012). When responding to the items concerning satisfaction with aspects of time and their acquisition of skills, a longer list appeared containing all of the items in the set, grouped under a common stem. For these items, respondents could work through the list a single item at a time, focusing on recall for each one before moving on to the next. This type of format may lend itself to an algorithmic approach, as opposed to the more heuristic strategy that allows an efficient recall of a more general topic area. Although the heuristic approach is RESEARCH & PRACTICE IN ASSESSMENT more efficient, it also risks error; thus, the memories available for recall may differ between students and alumni, partially explaining the different direction of patterns for alumni and students for the different types of survey items. Taken together, the general pattern suggested in these results is that alumni provide more positive evaluations of their institutions overall, yet more critical judgments when certain specific aspects are concerned. However, it should also be noted that in terms of the magnitude of the differences between the alumni and student responses, the effect sizes were all in the moderate to small range (Cohen, 1992). Although this is common for social science and educational research (Gonyea & Sarraf, 2009; Hayek, Gonyea, & Zhao, 2001), it is still a consideration in the interpretation of the results. The statistical significance of the comparisons is certainly important, but the practical significance of the comparisons, most of which were small to moderate is an essential component for a complete understanding of the results as well. When institutions with limited resources are considering which potential curricular and programming changes they should prioritize, those aspects with the larger effects might be the more practical areas on which to focus. Limitations Although there are strengths of this study, some limitations should be noted. Given the data collection procedures and response rates, the sample may not be representative of all arts alumni and students, and caution should be made when making generalizations. It may also be the case that respondents to student surveys are different than respondents to alumni surveys, but there is evidence to suggest that despite their lower response rates, respondents to alumni surveys are just as representative as student surveys (Lambert & Miller, 2014). Furthermore, this study relied on self-reported data, which may not always be completely objective. However, most studies looking at self-reports in higher education suggest that selfreports and actual measures of constructs such as abilities are positively related (Anaya, 1999; Converse & Presser, 1989; Hayek, Carini, O’Day, & Kuh, 2002; Laing, Sawyer, & Noble, 1987; Pace, 1985; Pike, 1995) and that social desirability bias is not a substantive concern for reports of basic cognitive and academic behaviors (Miller, 2012). It should also be noted that this study design was cross-sectional rather than longitudinal, and although the students and alumni were matched for major and institution, there were still different individuals responding from each group. Additionally, the quantitative nature of the data may have missed some of the nuance and tone of student and alumni perceptions of their institutions and skill development. Thus, if administrators and faculty want the complete picture of what can help create the optimal institutional experiences for students and also prepare them for the workforce, gathering information from both students and alumni may be the best assessment practice in this situation. Conclusion While it is hard to determine which group has a more accurate report of the experience, important institutional information can be gained through surveying both students and alumni. Students may be better able to provide information about affective components of their experience, while alumni may be better judges of specific things needed in the workplace. Being closer in time to the experience may have the advantage in terms of memory accuracy, but temporal distance may have the advantage of reflective insight. Thus, if administrators and faculty want the complete picture of what can help create the optimal institutional experiences for students and also prepare them for the workforce, gathering information from both students and alumni may be the best assessment practice in this situation. Future research should not only expand the topics on which student and alumni comparisons can be made, but also incorporate a longitudinal design that matches data at the respondent level. Moreover, it may be useful to incorporate matched assessment data that are not self-reported. For instance, employer feedback on the skill development of alumni or summative rubrics from faculty in required major courses may supplement the findings from alumni and student surveys. Furthermore, qualitative approaches such as focus groups and one-on-one interviews could provide an additional source of information for assessment purposes. The SNAAP survey instrument actually includes several different open-ended questions for alumni to elaborate on various topics, and institutional users often report that these quotes are very powerful in conveying the survey findings to numerous audiences. For example, when asked about how the institution could have better prepared them for their career, one alumnus in this study included a specific curricular suggestion, noting, “One Volume Ten | Winter 2015 11 RESEARCH & PRACTICE IN ASSESSMENT thing that I really enjoyed at [my institution] was the push to pursue your own ideas, but the design program could also incorporate projects that focus on the designer/client relationship.” This type of qualitative information can further enhance the value and application of the quantitative data when making program updates. Alumni surveys may be especially important as part of assessment cycles. The responses of alumni may be used to make curricular changes, which then impact current students, who can be assessed as students and then later as alumni to determine whether or not the changes were effective. This process can also be interpreted as a means of institutional transparency, as alumni already have their degree so they have a different focus and less at stake, and institutions are willing to share their feedback, both positive and negative, in order to make upgrades. Accessing the perspectives of both students and alumni are important sources of data for improvement in higher education; therefore, surveys of both populations should be administered for the best information possible. 12 Volume Ten | Winter 2015 RESEARCH & PRACTICE IN ASSESSMENT References Anaya, G. (1999). College impact on student learning: Comparing the use of self-reported gains, standardized test scores, and college grades. Research in Higher Education, 40, 499-526. Atrostic, B. K., Bates, N., Burt, G., & Silberstein, A. (2001). Nonresponse in U.S. government household surveys: Consistent measure, recent trends, and new insights. Journal of Official Statistics, 17(2), 209-226. Badcock, P. B. T., Pattison, P. E., & Harris, K. L. (2010). Developing generic skills through university study: A study of arts, science and engineering in Australia. Higher Education, 60, 441-458. doi: 10.1007/s10734-010-9308-8 Baruch, Y. (1999). Response rates in academic studies – A comparative analysis. Human Relations, 52, 421-434. Bauer, C., Viola, K., & Strauss, C. (2011). Management skills for artists: ‘Learning by doing’? International Journal of Cultural Policy, 17(5), 626-644. Borden, V. M. H., & Kernel, B. (2013). Measuring quality in higher education: An inventory of instruments, tools, and resources. Retrieved from http://apps.airweb.org/surveys/Default.aspx Bradshaw, D. (1985). Transferable intellectual and personal skills. Oxford Review of Education, 11(2), 201-216. Buskirk, T. D., & Andrus, C. (2012). Smart survey for smart phones: Exploring various approaches for conducting online mobile surveys via smartphones. Survey Practice, 5(1). Cabrera, A. F., Weerts, D. J., & Zulick, B. J. (2005). Making an impact with alumni surveys. New Directions for Institutional Research, 2005, 5-17. doi: 10.1002/ir.144 Cantor, M. (2012, April 23). The 13 most worthless majors. Newser. Retrieved from www.newser.com. Carnevale, A. P., Cheah, B., & Strohl, J. (2012). College majors, unemployment, and earnings: Not all college degrees are created equal. Washington, DC: Center of Education and the Workforce, Georgetown University. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159. Converse, J. M., & Presser, S. (1989). Survey questions: Handcrafting the standardized questionnaire. Newbury Park, CA: Sage. Davis, S. F., & Palladino, J. J. (2012). Psychology (7th ed.). Upper Saddle River, NJ: Prentice Hall. Edstrom, A. M. (2008). Art students making use of studio conversations. Art, Design & Communication in Higher Education, 7(1), 31-44. doi: 10.1386/adche.7.1.31/1 Evers, F. T., Rush, J. C., & Berdrow, I. (1998). The bases of competence: Skills for lifelong learning and employability. San Francisco: Jossey-Bass. Glantz, S. A., & Slinker, B. K. (2001). Primer of applied regression and analysis of variance (2nd ed.). New York: McGraw-Hill. Gonyea, R., & Sarraf, S. (2009, June). Contextualizing NSSE effect sizes: Empirical analysis and interpretation of benchmark comparisons. Paper presented at the Annual Forum of the Association for Institutional Research, Atlanta, GA. Hayek, J. C., Carini, R. M., O’Day, P. T., & Kuh, G. D. (2002). Triumph or tragedy: Comparing student engagement levels of members of Greek-letter organizations and other students. Journal of College Student Development, 43(5), 643-663. Hayek, J., Gonyea, R., & Zhao, C. (2001, November). Reporting and interpreting effect sizes in higher education research. Paper presented at the Annual Conference for the Association for the Study of Higher Education, Richmond, VA. Johnson, C. (2002). The position of the fine art student in the context of public accountability of higher education, with specific reference to the Quality Assurance Agency experience. Teaching in Higher Education, 7(2), 215224. doi: 10.1080/13562510220124277 Kuh, G. D., & Ewell, P. T. (2010). The state of learning outcomes assessment in the United States. Higher Education Management and Policy, 22(1), 1-20. Kuh, G. D., & Ikenberry, S. O. (2009). More than you think, less than we need: Learning outcomes assessment in American higher education. Urbana, IL: University of Illinois and Indiana University, National Institute of Learning Outcomes Assessment. Kuh, G. D., Jankowski, N., Ikenberry, S. O., & Kinzie, J. (2014). Knowing what students know and can do: The current state of student learning outcomes assessment in U.S. colleges and universities. Urbana, IL: University of Illinois, National Institute of Learning Outcomes Assessment. Volume Ten | Winter 2015 13 RESEARCH & PRACTICE IN ASSESSMENT Lambert, A. D., & Miller, A. L. (2013, April). Assessing alumni success: Income is NOT the only outcome! Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, California. Lambert, A. D., & Miller, A. L. (2014). Lower response rates on alumni surveys might not mean lower response representativeness. Educational Research Quarterly, 37(3), 38-51. Lambert, A. D., & Miller, A. L. (2015). Living with smartphones: Does completion device affect survey responses? Research in Higher Education, 56(2), 166-177. Laing, J., Swayer, R., & Noble, J. (1987). Accuracy of self-reported activities and accomplishments of college-bound seniors. Journal of College Student Development, 29(4), 362-368. Mavletova, A. (2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31(6), 725-743. doi: 10.1177/0894439313485201. Miller, A. L. (2012). Investigating social desirability bias in student self-report surveys. Educational Research Quarterly, 36(1), 30-47. Pace, C. R. (1985). The credibility of student self-reports. Los Angeles: The Center for the Study of Evaluation, Graduate School of Education, University of California at Los Angeles. Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students (vol. 2). San Francisco: Jossey-Bass. Pike, G. R. (1995). The relationship between self-reports of college experiences and achievement test scores. Research in Higher Education, 36(1), 1-22. Porter, S. R. (2004). Raising response rates: What works? New Directions for Institutional Research, 121, 5-21. Reisberg, D. (2012). Cognition: Exploring the science of the mind (5th ed.). New York: W. W. Norton & Company. Smilde, R. (2008). Lifelong learners in music; research into musicians’ biographical learning. International Journal of Community Music, 1(2), 243-252. Smith, K., & Bers, T. (1987). Improving alumni survey response rates: An experiment and cost-benefit analysis. Research in Higher Education, 27(3), 218-225. doi: 10.1007/BF00991999. Stasz, C. (1997). Do employers need the skills they want? Evidence from technical work. Journal of Education and Work, 10(3), 205-223. Stasz, C. (2001). Assessing skills for work: Two perspectives. Oxford Economic Papers, 3, 385-405. Tait, H., & Godfrey, H. (1999). Defining and assessing competence in generic skills. Quality in Higher Education, 5(3), 245-253. 14 Volume Ten | Winter 2015