Growing a New Culture of Assessment: Planting ePortfolios in the Metro Academies Program

International Journal of ePortfolio http://www.theijep.com 2011, Volume 1, Number 1, 71-83 ISSN 2157-622X Growing a New Culture of Assessment: Planting ePortfolios in the Metro Academies Program Alycia Shada, Kevin Kelly, Ruth Cox, and Savita Malik San Francisco State University and City College of San Francisco This paper provides a look at the development of a new culture of assessment in higher education with the use of electronic portfolios (ePortfolios). It uses the metaphor of horticulture to describe how an inter-institutional program, Metro Academies of Health, has gone through the first two parts of the ePorticulture cycle—preparing for the use of ePortfolios and planting the first ePortfolio "seeds" within cohorts of students at both an urban community college and 4-year state university. Metro serves as a case study for potentially rich, albeit challenging, ePortfolio integration within a program that serves primarily low-income, first-generation college students. Given the chronically poor outcomes of many of today’s college students, ePortfolios operate as a high-impact practice that provides students and educators with a tool for assessment to improve academic success. Metro aims for a successful and strategic ePortfolio implementation by beginning with a foundation of research on best practices and gives a series of recommendations that apply to new or growing ePortfolio programs. For centuries, educators have been experimenting with the science and art of promoting, collecting, and assessing student work—just as horticulturalists have explored improvements in the cultivation of plants. While horticultural practices have evolved into an extremely complex science, so too has our potential to use new tools and technologies to nurture and harvest a wider range of student work. Dependence on standardized assessment strategies as the primary means of harvesting student knowledge often does not adequately prepare students for the ever-changing future. Wardlaw (2006) made the case that expectations for learning have changed in response to a new global context, requiring students to gain skills in communication, teamwork, problem solving, analysis, reflection, performance improvement, innovation, and lifelong learning, among other things. However, curriculum design has changed only marginally since the start of the modern academy in the Renaissance period. Emerging socio-technology trends must play a wider role in influencing changes in curriculum design going forward. Darling-Hammond (2009) stated that on-demand and curriculum-embedded assessments should be used together to “measure the full range of knowledge and skills represented in standards” (p. 29). We believe that learners must be guided toward clear, concise academic learning outcomes and, like DarlingHammond, that good practice in comprehensive assessment will require a wider variety of assessment strategies over time. Cultivating a common cultural approach to curriculum and assessment has proven to be a significant, ever-present challenge. Yet the Association of American Colleges and Universities (AAC & U, 2009) believes that "to achieve a high-quality education for all students, valid assessment data are needed to guide planning, teaching, and improvement." They also advocate for well-planned electronic portfolios that can "provide opportunities to collect data from multiple assessments across a broad range of learning outcomes while guiding student learning and building self-assessment capabilities and eportfolios" and "assessment of work in them can inform programs and institutions on progress in achieving expected goals" (AAC&U, 2009). In 1993, early research on ePortfolios from the Coalition of Essential Schools and the Annenberg Institute for School Reform identified five core factors to consider when exploring the successful planning and implementation of electronic portfolios: vision, assessment, technology, logistics, and culture (Niguidula, 1997). While the ePortfolio movement has evolved and grown dramatically, consideration of all of these basic factors still makes sense. We have learned a lot about what it takes to nurture and harvest a good "crop" of portfolios in our experience of working on ePortfolio development within a large public university. While there are many factors that may determine the success or failure of comprehensive assessment, we believe that the most essential element that needs to be planted is that of shifting, re-defining, or adapting the existing culture of assessment. The Complexity and Culture of Assessment The attitudes and practices underlying how disciplines expect students to demonstrate their learning varies radically—from high-stakes testing to observation/demonstration to comprehensive portfolios. Advancing Change in Educational Assessment at San Francisco State University New digital technologies like electronic portfolios have opened the way for profound changes in Shada, Kelly, Cox, and Malik educational assessment. Since 2005, San Francisco State University (SF State) has been developing resident expertise and organizational capacity to support and advance the development, use, and sustainability of electronic portfolio tools. Academic Technology, in conjunction with participating colleges and departments, offers on-going consultation, support, and training for both students and faculty on the creation of ePortfolios at both the undergraduate and graduate levels. ePortfolios are now used as a full or partial comprehensive, formative/ summative assessment strategy within 22 of 75 departments. Over the past six years working with a variety of departments, we’ve discovered that elements of the “ground-work” phase—e.g., preparation, faculty buyin, shared planning, and cultural change—are often the most challenging yet important aspects of launching a successful ePortfolio project “planting” or implementation. This case study examines a unique opportunity to collaborate on the structure, design, reflection strategies, and practical applications for an emerging project— the Metro Academies. The Metro Academies is a reformed approach to the first two years of college that may be completed in both community colleges and four-year universities. Metro Academies uses an ensemble of high-impact educational interventions spotlighted by AAC&U. The project goals are the retention of community college and university lower division students; successful transfer for community college students; and accelerated mastery of rigorous knowledge and competencies in key foundation areas— writing, quantitative thinking, public speaking, and critical thinking. Demonstration sites are currently operational at City College of San Francisco (CCSF) and SF State, the first time Academic Technology has worked with a partnership of this kind. Despite SF State’s broad experience with ePortfolios, Metro represents a new challenge. Not only it is a small undergraduate program for first- and second-year college students, many of whom are lowincome, first-generation college, but the program also spans across two institutions and aims to develop a deeply developmental ePortfolio in already content-rich courses. This unique program offers great challenges, but also great opportunities for a rich integration of ePortfolios. The ePorticulture Cycle The redesign of comprehensive evaluation methods occur across several aspects of the educational process, with the most significant taking place within the culture of assessment. To that end, Kelly and Cox (2011) coined the term "ePorticulture": The act or custom of learning, developing intellectually and professionally, and transmitting New Culture of Assessment 72 knowledge through the creation, review, and assessment of authentic, reflective, and integrative student work that is shared over time via electronic portfolios. Etymology: e (electronic) + portfolio (a selection of a student's work compiled over a period of time and used for assessing performance or progress) + culture (the integrated pattern of human knowledge, belief, and behavior that depends upon the capacity for learning and transmitting knowledge to succeeding generations). Just as the cycle of plant growth in horticulture has four components—1) preparing the soil, 2) planting seeds or transplanting plants, 3) growing or maintaining the plants, and 4) harvesting—so too does the ePorticulture cycle. Applying this metaphor to ePortfolio implementation in the Metro Academies, the program is “preparing the soil” by building faculty buyin, garnering institutional support, and encouraging students to begin to think about and articulate their academic and professional identities. To “plant the seeds,” the program is adopting the ePortfolio technology and developing processes. These processes include creating assignments that both align with class and program objectives, and provide opportunities for students to reflect on how their work relates to their goals of transfer and degree completion. “Growing and maintaining the plants” is analogous to navigating the ongoing logistics involving user motivation, training, and general technological and pedagogical support. Lastly, the “harvest” occurs when ePortfolios are created and shared. Producing a “crop” of ePortfolios that stakeholders can see helps build further support for additional investment and “planting.” This article will describe how an inter-institutional program, Metro Academies of Health, has gone through the first two parts of the ePorticulture cycle—preparing for the use of ePortfolios and planting the first ePortfolio "seeds." In one or two years, the authors plan to write a follow-up article to describe how the Metro program has grown and maintained ePortfolios, and harvested student work as participants transfer to the four-year institution or achieve their degree goals. Preparing for ePortfolios in Metro Academies: Emerging Socio-Technology Trends and HighImpact Practices To help the Metro Academies plan and “prepare the soil” to grow and maintain ePortfolios, Metro drew on earlier experiences at SF State, and researcher Alycia Shada conducted a comprehensive review of five case studies in the wider literature about programlevel ePortfolio implementation efforts. Shada followed Shada, Kelly, Cox, and Malik New Culture of Assessment 73 Figure 1 Sequence and Pairing of Metro Courses Source: Metro Academies the review with interviews of ten faculty members— representing five programs across the SF State campus (2011). This article incorporates a look at why ePortfolios are an important component to the Metro Academies, and overall recommendations for implementation. Why ePortfolios in the Metro Academies? Wilmarth (2010) claimed, “The case can be made that, at the dawn of the 21st century, converging technologies and emerging social trends lay the groundwork for entirely new societal landscapes.” These new landscapes can be found in the very meaning of the work we do and the lives we lead, and ultimately in the what, where, why, and how we learn. In the preparing and planting phases of ePorticulture, a program can use ePortfolios to address a current and emerging need—i.e., for students to have an environment in which they can collect, select, reflect upon, build, and publish a digital archive of their academic work to selected audiences. ePortfolios represent a potential key to open closed doors between disciplines, making transparent the expectations, values and goals that educators expect of students. Through the growing and harvesting phases of ePorticulture, ePortfolios also represent an opportunity for academia to help students to bridge their learning with the creation of a professional persona and a demonstration of work-force readiness. Metro’s vision is “to increase equity in college completion through engaging, supportive, rigorous, and socially relevant education” (Metro Academies, 2011). Metro aims to improve graduation rates for low-income, firstgeneration college students as well as improve the quality of their college academic experience. Metro accomplishes this by creating small learning communities of students who take paired courses together; generally a health education course partnered with a general education course that is infused with health-related content (see Figure 1). With a faculty that is committed to pedagogy and building a community of learners, Metro has become an ideal planting ground for a culture of ePortfolios. Students today are adept at representing themselves informally on the web through social networking, yet have rarely considered creating a more formal, academic identity through a published web-portfolio. We have observed how ePortfolios can serve as a tool to foster reflective learning, helping students build academic identity, make connections across coursework and various aspects of their lives, and allow for formative assessments by faculty and advisors. The Metro project represents an opportunity to actively apply some of our earlier experience and learning. The Context and Responsibility to Underrepresented Students The Metro program was developed in response to chronically poor outcomes of today’s college students—in terms of both low and inequitable college completion and the lack of development of academic skills. California was once considered a leader in providing access and excellence in higher education, but it has now fallen to have some of the worst college outcomes in the country. A recent report by the Public Policy Institute of California (PPIC) showed that community college transfer rates are low and “only about half of [California State University] students earn a bachelor’s degree within six years” (Johnson & Sengupta, 2009). Strategically working to improve students’ academic outcomes is more important than ever. Overall, underrepresented students (particularly lowincome students and students of color) have had very low rates of college completion and are a growing population (Offstein, Moore, & Shulock, 2010). According to a recent study of California community college students, only 31% of students “completed a certificate or degree, or transferred to a university within six years of enrolling” (Shulock & Moore, 2010). The study also found that underrepresented minority students (who are often low-income, first- Shada, Kelly, Cox, and Malik New Culture of Assessment 74 generation college-going) made up only 30% of the students who successfully completed college, whereas they made up 43% of the “incoming degree seekers.” Furthermore, Latino students were “only half as likely as white students to transfer (14% to 30%)” and “black students were also less likely than white students to transfer (20% to 29%)” (Shulock & Moore, 2010). One unique aspect of the Metro program is the crossinstitutional partnership between the community college and the CSU. This collaboration could lead to new ways of thinking about using ePorfolios as a transfer tool and has implications for new ways to integrate between 2-year and 4-year colleges. Metro aims to show how effective interventions, such as the use of electronic portfolio, when cultivated with intention, can help improve college completion for these student populations. Studies related to improving overall transfer rates and the students' transfer experience itself recommend a variety of strategies, several of which Metro Academies has instituted or has begun to institute. Key strategies include, but are not limited to, the following: The Philosophy Behind Metro and Inclusion of ePortfolios as a High-Impact Practice • Metro’s program model is centered on several of the Association of American Colleges and Universities’ (AAC&U) high-impact practices. “High-impact practices” are educational practices that have proven to be extremely effective in creating positive results for “students from widely varying backgrounds” (Kuh, 2008, p. 1). These practices have shown to be particularly effective for historically underserved students and those who enter higher education with lower test scores than their peers. These practices include strategies such as learning communities, writing-intensive courses, collaborative assignments and projects, and first-year seminars. The most recent addition to the list was the use of ePortfolios (Rhodes, 2011). The program's emphasis on accelerated learning addresses the fact that up to 75% of community college students and more than half of public university students arrive on campus with test scores indicating that they are not fully prepared for college work (Shulock, 2010). As a broader aim, Metro Academies seeks to develop leadership and employment capacity among people in low-income urban communities, displaced workers, and working adults. Participation in this initiative is geared towards those interested in a career in public health, but can also lay the groundwork for movement into a number of fields. With their general education requirements complete, students move on a fast track to majors such as Health Education, Sociology, Child and Adolescent Development, Urban Studies, Political Science, Recreation and Tourism, and Psychology. Metro Academies is designed to help students transfer to the California State University (CSU) system. • • • • Creation of inter-institutional programs to facilitate transfer: Inter-institutional partnership programs like Metro create a seamless experience for students (Balzer, 2006). The Metro Academies program has become a model for other interinstitutional projects. The Metro curriculum— including health-infused general education courses paired with lower-division health education courses—is designed to prepare students for transfer, as well as for entrance into a variety of majors such as Health Education, Urban Studies, or Social Work. Involve transfer students sooner as members of the four-year campus community: After conducting transfer student interviews, Flaga (2002) recommended that four-year campuses "address those students' needs to adapt to a more decentralized support environment than community colleges generally provide" (Kelly, 2009). Metro provides students exposure to various aspects of the four-year campus, ranging from virtual environments like ePortfolios to physical environments through orientations, program meetings on both campuses, and introductory visits to SF State during the semester prior to transfer. Addressing social integration needs of transfer students: Gumm (2006) identified social integration as an important variable for predicting both a) students' decisions to remain in school (99.1% correct) and b) commitment to academic goals and the institution itself (99.7% correct for predicting persisters). By using a cohort model, Metro provides social integration opportunities from the beginning, as cohort members will have each other as a support network after transfer, as well as a network of faculty who help facilitate their integration into their junior year. Use ePortfolios to facilitate the transfer process: Kelly (2009) recommended that discipline-specific programs should provide ePortfolios for students to showcase that they had met specific requirements (e.g., general education, program prerequisites). Students could also demonstrate skills or experiences related to their intended field of study after transfer. Metro Academies students begin using ePortfolios in their first semester of the twoyear program (see next section for more details). Increased utilization of and communication between advisors at all institutions: Researchers and transfer students themselves outlined the Shada, Kelly, Cox, and Malik importance of advisors and the need for increased communication between advisors from two-year and four-year institutions (Kisker, 2007; Flaga, 2002). Metro faculty members from both institutions meet regularly to discuss curriculum, the use of ePortfolios, increasing student success, and more. Potential Benefits of ePortfolios for Metro Participants Despite the grim statistics, we have hope for improving the outcomes of our students. Several recent studies have shown that students who use ePortfolios tend to have higher retention rates, higher GPAs, higher course pass rates, and significantly higher levels of engagement (Yancey, 2009; Clark & Eynon, 2009; Kirkpatrick, Renner, Kanae, & Goya, 2009). After a study conducted at LaGuardia Community College (LGCC) in New York, Clark & Eynon (2009, p. 21) found that Data gathered using the Community College Survey of Student Engagement show that students in e-portfolio-intensive courses at LaGuardia are more likely to show high degrees of engagement with critical thinking, collaboration, and writing. Analysis of course pass rates and semester-tosemester retention also show higher rates of success for students in e-portfolio-intensive courses, compared to students in similar courses that do not use e-portfolios. Challenges in Implementing ePortfolios ePortfolios can provide many services and function as a multi-faceted tool. Research shows many profound benefits for students, instructors, and higher education in general. However, little research has been able to definitively say what exactly it is about ePortfolios that make them “work” nor has it been able to isolate certain components to producing certain benefits (Yancey, 2009); however, it seems that comprehensive, wellintegrated ePortfolio systems serve important purposes as both a process and a product. Although ePortfolios are deeply integrated into many institutions’ curriculum and culture, in many cases their implementation falls flat. Levels of integration vary and can range from being fully vetted throughout an institution and supported by a statewide initiative (Clark & Eynon, 2009) to sometimes only showcasing a couple of assignments in a few classes (Cambridge, Cambridge, & Yancey, 2009). ePortfolios represent a variety of complex objectives, various stakeholders, and a range of ways in which users’ processes and skills must change in order to use the system effectively. New Culture of Assessment 75 In “The ‘Sticky’ ePortfolio System,” Ali Jafari (2004) claimed that ePortfolios “will become a fully implemented, successful tool…[and] will play a significant role in higher education. However… developing and implementing a successful ePortfolio project—one that is ‘sticky,’ one that works and is adopted by users—will first involve many challenges” (p. 38). Bret Eynon, leading scholar and driver of ePortfolios at LGCC, said that ePortfolio systems often “briefly bloom and fade” and that some of the challenges to ePortfolios’ sustainability are their “sophisticated learning design,” that they often “break traditional boundaries of curriculum and pedagogy,” and that they are a “disruptive pedagogy”—meaning their success implies and often requires “broad institutional collaboration and change” (Eynon, 2011). Translating Eynon’s thoughts to our ePorticulture metaphor, institutions, programs, and individual instructors must do more to prepare the ground pedagogically and support students as they grow and maintain competencies-based evidence. Only then will the blooms last, pollinate, and become fruit for advisors or prospective employers. As noted earlier, the ePorticulture preparation phase is both critical and difficult. Chen and Light (2010) pointed out in Electronic Portfolios and Student Success, “the value of e-portfolios lies not in the specific tool itself, but in the process and in the ways in which the concept and the related activities and practices are introduced to students” (p. 27). This suggests the importance of the ways in which an ePortfolio system is integrated into the curriculum and pedagogy. Simply adopting the tool is likely not enough to affect real educational change. Additionally, Kathleen Yancey warned, “the inability to get students engaged or excited about their e-portfolios will result in a flawed implementation” (Yancey, 2009). Therefore, as programs “prepare the ground,” they should include planning time to determine how they will help students find meaning through reflective writing, and help faculty use ePortfolios for assessment and advising. While Metro provides an ideal planting ground for ePortfolios, it also holds many challenges. Institutional resources are scarce, the needs and resources of faculty vary by course and institution, and students often enter the program requiring remediation and have vast disparities in technical skills. Furthermore, the program does not have a strong culture of technology and substantial changes will need to be made by instructors, students, and program administrators to support the implementation of ePortfolios. The challenges Metro faces however are not unique—successful and sustainable implementations are difficult. As part of the critical preparation phase, Metro leaders and Academic Technology team members have begun to work with a Shada, Kelly, Cox, and Malik small group of Metro faculty from both institutions. Together, they will simplify the technology transition for faculty and students, and improve the pedagogical connections through the alignment of key assignments and the development of reflective writing prompts. Planning for a Successful Planting of ePortfolios Metro's Current Status with ePortfolios Throughout the 2010-11 academic year, instructors of Metro’s core courses—lower division courses in the Department of Health Education—made ePortfolio accounts available to their students. SF State currently supports only one ePortfolio software platform— eFolio. Because eFolio has worked well for the university’s various programs and departments, Metro will continue to only offer this one platform. The students and instructors had approximately one ePortfolio workshop with Academic Technology and most have uploaded a couple of academic artifacts to their ePortfolio. In general, however, this first pass at issuing ePortfolio sites was not integrated into the curriculum and the support and goals at the program level were unclear. With the support of a FIPSE Connect to Learning mini-grant, the Metro Academies faculty began a series of meetings in the 2011-12 academic year that address the integration of ePortfolios into their curricular design to support integrative learning and reflection. These developments provide a fresh start for the project. We see the use of ePortfolios in Metro as a way to develop meaningful prompts and to track and evaluate student progress in challenging general education subjects such as English and math. By “planting and maintaining” their ePortfolios, community college students in the Metro Academies cohorts will document their developing academic skills (academic artifacts), professional and life experience, interests, and co-curricular skills. In helping students grow ePortfolios and prepare for harvesting by different stakeholders, advisors and faculty will also use the portfolios in formative advising and for career development. This guidance will be especially important for those who need a successful early harvest—those students transferring from CCSF to SF State (or other CSU campuses). Metro leadership and Academic Technology staff introduced the new ePortfolio project to the all-faculty meeting at the beginning of the 2011 spring semester. Following this meeting, eleven faculty members completed an anonymous open-ended survey, geared at determining faculty values and attitudes about using the ePorfolio in their own classrooms. The survey was administered in follow-up faculty meetings, after participants had an opportunity to reflect on the New Culture of Assessment 76 introduction to the ePortfolio tool and project. The survey planted the following questions: 1. 2. 3. What are some things that excited you about using the ePortfolio tool in the classroom? What are some things that cause anxiety in using the ePortfolio tool in the classroom? What specific support can you anticipate needing around ePortfolios? Qualitative responses were transcribed onto one document, indexed and coded for salient themes. In general, instructors indicated excitement over the possibilities of student learning and reflection, as well as the ability to showcase work. Instructors indicated anxiety around issues such as dealing with the technology (learning it as well as having adequate access to it), the overall time commitment, and having adequate support to deal with students’ varying learning curves. They anticipated needing support around the integration of ePortfolios into the curriculum and readily available tech support (e.g., quick responses and drop-in hours; Shada, 2011). With this information, Metro is developing a strategic implementation plan that can lead to a successful and sustainable integration of ePortfolios into the curriculum and overall program. Because implementing ePortfolios into the program and curriculum can be a substantial undertaking, it is particularly important to think through the inputs (planting), activities (growing), expected outputs and outcomes, as well as the intended overall impact (harvesting). This exercise can help surface any underlying assumptions of the stakeholders and help clarify objectives and expectations. The logic model can also be revisited and revised during and after implementation and is intended to serve as a guide for discussion among Metro’s leadership and faculty rather than a comprehensive model. ePortfolio Lessons Learned and Applied to Metro Over the years, when working with a variety of departments, we have noted that the most successful programs have been those that have an identified and required beginning and completion course tied to ePortfolio use. The sequential structure of the Metro program will allow Academic Technology to “plant” or issue ePortfolio accounts to all students through “gateway” courses on both campuses, promote full-faculty buy-in on requiring the timely uploading of “signature” assignments each term, and require finishing the portfolios in a capstone course. Shada, Kelly, Cox, and Malik Recommendations for Metro Based on Shada’s research, we make the following recommendations at the institutional, program, and course levels for preparing the ground and planting seeds within Metro’s ePortfolio implementation. Regarding best practices, Metro faculty can learn from one another as well as from other instructors who have pioneered ePortfolio programs at SF State. Shada’s research resulted in a collection of best practices throughout the institution (see Appendix A for details). Institutional Level Strategically discuss critical issues with key stakeholders. Collectively make decisions with key stakeholders, particularly faculty and leadership team and continuously seek their involvement in on-going decision-making processes. Understand their needs, interests, and concerns. Understand their language and how ePortfolios can help them. Topics to discuss include: • • • • • • Definition/s, objective/s, and goals of ePortfolios; clarification of process and roles Assignments to go into the ePortfolios (which assignments and how many artifacts for each competency) How to adapt the VALUE rubric appropriately for the program’s needs Identification of external stakeholders, or perceived external stakeholders and plan for communicating with them (e.g., talk to leaders in impacted majors at SF State, talk to SF State advising office—would they use ePortfolios? What would they like to see in them?) Feedback on success and/or concerns of implementation and overall project Perceived benefits of ePortfolios Provide resources. Create documents to serve as information and resource guides for instructors and students. Content should include important contact information, log-in and troubleshooting information, and where to go for different issues, as well as a brief overview of the purpose and structure of the ePortfolios. In addition, compile documents with sample assignments, assignment instructions, writing prompts, and grading rubrics. Allow time. Allow time for instructor and student work and provide resources. Instructors will need time to revise their syllabi and potentially make pedagogical shifts. Students and instructors will both need time to learn the technology. Students will need time to reflect. The program will need time to create and refine the data collection process for evaluation of the ePortfolio New Culture of Assessment 77 program. Hosting workshops and meetings may be effective ways to give stakeholders (both students and faculty) time to do some of this work. Provide opportunities for stakeholders to reflect and communicate. Provide support to instructors and students. Provide support staff and identify one “go-to” person for additional support. Consider providing support staff via faculty peers and student assistants—this may be more cost effective and will help enhance the ePortfolio culture as well as help empower individual stakeholders. Provide support in multiple ways, including group workshops, mentoring in the classroom, instructional materials, and one-on-one help. Trainings should be ongoing and also made available to new hires. Provide stipends if/when possible. Be flexible, but strategic. Begin with instructors who have an interest and allow initial implementation to be uneven. Plan meetings strategically—make sure that the timing works for faculty schedules and needs and ensure that the meetings are “timely, well-taught, and designed for appropriate stages of concern and levels of use” (Brzycki & Dudt, 2005). Reiterate that the project will maintain flexibility and revisit program matrices, and keep a focus on long-term goals. Allow for a flexible implementation, but provide some structure and accountability for the project participants. Program Level Implement incrementally. Initially, implement more fully in the gateway and capstone courses, but also begin to plan to make it a developmental ePortfolio and determine what that means for the “in-between” courses and/or the program. Consider if the ePortfolio will be reinforced outside of Metro’s current courses (e.g., in workshops, orientation, end-of-program celebration, advising sessions, etc.). Provide resources to help instructors make pedagogical shifts. Provide sample prompts and assignments for teaching reflection, scaffolding reflection, and writing reflective prompts. Encourage “best practices” among instructors for teaching reflection. Develop a plan for program assessment. Develop a timeline with leadership staff for assessing overall achievement of program learning outcomes and determining how curriculum and/or pedagogy may adapt in response to this data. Be mindful of possible conflicts in goals related to student learning and goals related to program assessment. Integrate into advising. During every advising session, have the advisor open up the student’s ePortfolio. Provide tailored support to some students and faculty. Decide how to support students who are less Shada, Kelly, Cox, and Malik comfortable with technology. Perhaps they can schedule one-on-one sessions with Academic Technology, or with the program’s ePortfolio “go-to” person. Provide clear and quick technological support, particularly to CCSF students and instructors. Identify and continuously address CCSF-specific barriers to ePortfolio development. Create a culture of making connections, setting goals, and envisioning a future self. Incorporate the concept behind ePortfolios into the culture of the program. Discuss “making connections,” “looking forward/envisioning a future self,” and “goal setting and revising” throughout the program. Understand the external audiences. Communicate with potential external audiences to determine external validity of the ePortfolios (e.g., determine if perceived benefits are true). Provide documentation of the basics. Provide documentation for instructors, students, and leadership. Documentation will help communicate the resources and support that it is available and provide consistency in communication of goals and objectives of the ePortfolio project. Plan long term. Clarify goals regarding having a developmental ePortfolio and how that may affect program capacity; develop a strategic plan to achieve this. Consider ways for the program to alleviate the time commitment required of individual instructors (e.g., create a peer mentor program, hire student assistants). Provide a formal way for students to showcase their ePortfolios. Course Level Make room for new curriculum. At the course level, anticipate challenges with finding “extra” time in already content-rich courses. Curriculum may need to be taken out of the courses, particularly in the gateway or capstone courses. Allow some autonomy in course-level integration. Allow instructors the autonomy to decide if they want to incorporate the ePortfolio throughout the entire semester or isolate it as its own activity. Encourage best practices. Facilitate and encourage “best practices” conversations among the faculty. Use a common rubric. Collectively adapt and continue to adapt the VALUE (or another commonly agreed-upon) rubric to evaluate each student’s overall ePortfolio. Determine at what point/s the overall ePortfolio will be graded. Use peer review. Incorporate peer review processes into the assessment. Begin with an autobiography and goals statement. Have students begin the ePortfolio process by writing some form of intellectual/academic New Culture of Assessment 78 biography and goals statement. Encourage them to “reflect on their education and think about [their] dreams” (SF State instructor) and think about their skills, strengths and weaknesses. Have them revisit these throughout the program. Determine flexibility in proof of competencies. Decide whether or not students may include non-Metro coursework as proof of competencies. Decide how to handle allowing artifacts to represent a variety of mediums (e.g., written documents, slideshows, video presentations, lab reports, spreadsheets, art, music). Integrate ePortfolios into course theme. Encourage instructors to integrate the theme of the ePortfolios into what they are already doing. Avoid making the ePortfolio an “add on.” Focus on process, not product. Remember that the process of creating an ePortfolio is often when students experience the most benefit. Emphasize and make time for the process and understand that the final product does not necessarily need to be “perfect.” Conclusion With the active support of Metro Academies faculty and administration, we have been presented with the opportunity to cultivate a common cultural approach to curriculum and assessment. The Association of American Colleges and Universities (2009) outlined that "to achieve a high-quality education for all students, valid assessment data are needed to guide planning, teaching, and improvement" and that "good practice in assessment requires multiple assessments, over time." They also advocate for wellplanned electronic portfolios that can "provide opportunities to collect data from multiple assessments across a broad range of learning outcomes while guiding student learning and building self-assessment capabilities and eportfolios" and "assessment of work in them can inform programs and institutions on progress in achieving expected goals" (AAC & U, 2009). As the analogy of ePorticulture continues to play out within the Metro Academies, the preparation is underway for a new integration of ePortfolios across two institutions. The hope is that planting the portfolios soon makes way for deep reflection and growth of the student experience throughout their four years in higher education. We will continue to document our collective efforts as we complete the first two ePorticulture phases and begin the next two—how we grow and maintain the program’s efforts, how the individual students grow and maintain their ePortfolios, and how all the stakeholders review and harvest their work in different contexts. We hope to identify more guidelines that other programs may find useful as they seek to grow their own cultures of assessment. Shada, Kelly, Cox, and Malik References The Association of American Colleges and Universities. (2009). VALUE: Valid assessment of learning in undergraduate education. Retrieved from http://www.aacu.org/value/ Balzer, J. (2006). Community college and university degree partnership programs: A study of the student experience. Unpublished doctoral dissertation, Oregon State University, Corvallis, Oregon. Cambridge, D., Cambridge, B., & Yancey, K. (2009). Electronic portfolios 2.0: Emergent research on implementation and impact. Sterling, VA: Stylus. Chen, H., & Light, T. P. (2010). Electronic portfolios and student success: Effectiveness, efficiency, and learning. Washington, DC: Association of American Colleges and Universities. Clark, J. E., & Eynon, B. (2009). E-portfolios at 2.0Surveying the field. Peer Review, 11(1), 18-23. Darling-Hammond, L. (2009). Lessons from abroad: International standards and assessments. A webinar presented for Edutopia and the Stanford Center for Opportunity Policy in Education in collaboration with the Council for Chief State School Officers. Eynon, B. (2011). Making connections: ePortfolio, collaboration & change. Presented at the Association of American Colleges and Universities e-Portfolio Forum 2011, San Francisco, CA. Flaga, C. T. (2002). The process of transition for community college transfer students. Unpublished doctoral dissertation, Michigan State University, East Lansing, Michigan. (ERIC Document Reproduction Service No. ED482192). Gumm, J. E. (2006). Transfer transitions: First semester experiences of transfer students at selected Texas Christian universities. Unpublished doctoral dissertation, Baylor University, Waco, Texas. Retrieved from http://hdl.handle.net/2104/4193 Jafari, A. (2004). The “Sticky” ePortfolio system: Tackling challenges & identifying attributes. EDUCAUSE Review, 39(4), 42. Johnson, H., & Sengupta, R. (2009). Closing the gap: Meeting California’s need for college graduates. Sacramento, CA: Public Policy Institute of California. Kelly, K. (2009). Student perceptions of the higher education transfer process from two-year to four-year institutions: A qualitative study viewed through the lenses of student departure, social network, and complexity theories. Unpublished doctoral dissertation, University of San Francisco, San Francisco, California. Retrieved from UMI Open Access: http://gradworks.umi.com/33/70/3370305.html Kelly, K., & Cox, R. (2011). ePorticulture: Growing a new culture of assessment. In D. Cambridge (Ed.), New Culture of Assessment 79 Global diffusion of e-portfolio: Solutions for collaborative education (in press). Hershey, PA: IGI Global. Kirkpatrick, J., Renner, T., Kanae, L., & Goya, K. (2009). A values-driven eportfolio journey: Na wa’a. Electronic portfolios 2.0: Emergent research on implementation and impact (pp. 97-102). Sterling, VA: Stylus. Kisker, C. B. (2007). Creating and sustaining community college-university transfer partnerships. Community College Review, 34(4), 282-301. Kuh, G. (2008). High-impact educational practices: What they are, who has access to them, and why they matter. Washington, DC: Association of American Colleges and Universities. Metro Academies. (2011). Metro Academies Initiative flier. Retrieved from www.metroacademies.org Offstein, J., Moore, C., & Shulock, N. (2010). Advancing by degrees: A framework for increasing college completion. California State University, Sacramento. Institute for Higher Education Leadership & Policy and The Education Trust. Niguidula, D. (1997). Picturing performance with digital portfolios. Educational Leadership, 55(3), 26-29. Rhodes, T. (2011). Show me the learning: E-portfolios and VALUE. Presented at the Association of American Colleges and Universities Annual Meeting 2011, San Francisco, CA. Shada, A. (2011). Electronic portfolio implementation in the Metro Academies program at San Francisco State University and City College of San Francisco. Unpublished master's thesis. Mills College, Oakland, California. Shulock, N. (2010). Beyond the rhetoric: Improving college readiness through coherent state policy. California State University, Sacramento. Institute for Higher Education and Leadership Policy and the Southern Regional Education Board. Shulock, N., & Moore, C. (2010). Divided we fail: Improving completion and closing racial gaps in California’s community colleges. California State University, Sacramento. Institute for Higher Education and Leadership Policy. Wardlaw, C. (2006). Mathematics in Hong Kong/China. Improving on being first in PISA. Paper presented at the 50th Annual Meeting of the Australian Mathematical Society, Sydney, Australia. Wilmarth, S. (2010). Five socio-technology trends that change everything in learning and teaching. In H. Hayes Jacobs (Ed.), Curriculum 21: Essential education for a changing world. Alexandria, VA: ASCD. Shada, Kelly, Cox, and Malik Yancey, K. B. (2009). Electronic portfolios a decade into the twenty-first century: What we know, what we need to know. Peer Review, 11(1), 28-32. ____________________________ ALYCIA SHADA is a Lecturer and Program Specialist in the Department of Health Education at San Francisco State University. She recently earned her Master’s in Public Policy at Mills College, with a focus on higher education policy and implementation processes for electronic portfolios. She currently works for the Metro Academies at San Francisco State University; a program focused on improving pedagogical practices and institutional support for low-income, firstgeneration college students. KEVIN KELLY is the Online Teaching and Learning Manager for Academic Technology (AT) at SF State. He received his doctorate in Organization and Leadership, focusing his dissertation research on students' perceptions of the higher education transfer process. He lectures about instructional design, learning improvement, learning with technology and distance education at SF State and Santa Clara University. He was a member of the CSU team participating in the National Coalition on E-Portfolio Research. RUTH COX coordinates the ePortfolio Initiative at SF State, works campus-wide with faculty on comprehensive assessment strategies using student New Culture of Assessment 80 electronic portfolios, and teaches Masters of Public Health (MPH) students in the Health Education Department. Ruth has designed and implemented curriculum and educational programming in academic and corporate settings since 1989. She has taught a range of courses using distance education strategies at SF State and Santa Clara University. As a visiting scientist at Cisco Systems, she advised the World Wide Education Group on new collaborative learning software and online teaching solutions for K-12 and higher education applications. SAVITA MALIK is Curriculum Director and Instructor for Metro Academies, a collaboration between the health education and early childhood departments at both SF State and CCSF. She has taught a variety of courses in the health education department and developed a personal ePortfolio as part of her master's program in public health. She is currently a doctoral candidate, examining the impact of faculty community on teaching practice. She has been involved in faculty development work since 2002. Acknowledgements This article was a featured presentation at the 2011 ePortfolio World Summit organized by the Association for Authentic, Experiential and Evidenced-Based Learning (AAEEBL). Shada, Kelly, Cox, and Malik New Culture of Assessment 81 Appendix A: Summary of Best Practices from Faculty Interviews For the faculty interview component of her research, Shada asked the following series of questions in approximately 30-minute semi-structured interviews, in an attempt to learn the details about how each instructor was using ePortfolios in their curriculum and program structure. 1. At a program or department level, how are ePortfolios used or integrated into the curriculum (e.g., gateway and capstone courses, required number of assignments, etc.)? 2. At a course level, how are ePortfolios integrated into the curriculum? Do you know of any specific reflective prompts, assignments, or activities that are particularly effective? Do you have any exercises or activities related to reflective writing? Would you be willing to share these with Metro Academy faculty? 3. Are ePortfolio assignments integrated across courses within your department or program? If so, how is this done? 4. If you were leading a faculty development effort (to integrate ePortfolios into the curriculum), what would you do? What challenges might you expect and how would you recommend overcoming them? 5. How are ePortfolios evaluated in your department (or course) (e.g., peer review, faculty formative/summative review, rubrics, etc.)? 6. Are there additional ways you would like to use ePortfolios in the future? 7. Any other comments or advice for programs trying to deeply integrate ePortfolios into their curriculum? 8. Why did you decide to begin using ePortfolios? 9. Do you find that using ePortfolios in your curriculum is more time consuming than not? If so, what specifically takes time? 10. What is the overall objective of your ePortfolios? 11. Do you think that students are using their ePortfolios after graduation or for other reasons? In addition, if Shada had any information (provided by Academic Technology) about specific work that instructor was doing, she asked them about that work. The findings are included below. Shada, Kelly, Cox, and Malik New Culture of Assessment 82 Appendix A Summary of Best Practices: Findings from Faculty Interviews Best Practice Description Discussion of online security One ePortfolio assignment includes a discussion of online security/safety for building an ePortfolio. Topics include what information is appropriate and safe to post on an ePortfolio and how to write your email address to avoid receiving spam mail. Discussion of equity One ePortfolio assignment has students look at the equity of various ePortfolio platforms. Through a social justice lens, students discuss accessibility in terms of financial barriers and universal accessibility design. Reflection One instructor has students write in-depth reflections for four areas of learning. The reflections are generally three to five pages in length and accompany three to five academic artifacts. This program scaffolds reflection over semesters and the instructor has found that more open prompts tend to be more valuable when asking students how they think their learning will transfer. Some of his guiding questions include, “What are the core understandings of each domain? What understandings are shared throughout all of their courses? Then, what are the disagreements? What are the strands of knowledge that differ in the different classes that they’ve taken?” He then asks them to “place themselves in that conversation” and then “situate themselves in those disagreements” and to think about how this will influence their future work. These reflections then help the instructors of the program determine how well the program’s curriculum is meeting the intended learning objectives. One drawback, the instructor noted, was that there are many courses that their students take that the program does not have influence over. Continuous goal setting and planning One program facilitates ‘Portfolio Workshops’ throughout the program, to give students an opportunity to rethink their goals and how they are going to reach them. These workshops are not held in a computer lab and do not cover the technical aspects of the ePortfolio. Instead, these workshops help students think about what they want the content of their ePortfolio to look like and how to make decisions throughout their program that help lead them to their professional and academic goals. These workshops are held by two faculty members and they try to hold them about once a semester. They begin with asking students what their goals are and then writing everyone’s goals up on the board. They then discuss what things the students can do to achieve those goals (e.g., what classes to take) and what have the students already done (e.g., what classes have they taken, what activities have they done). The students then outline what things they would like to be able to do and what they would like to improve upon. The students walk away from the workshop with a list of a couple concrete things they plan to do the following semester. They are encouraged to come to a later workshop to refocus, particularly if their goals have changed. Student feedback of the workshops has been extremely positive. Students say that “the workshops have helped them put things in perspective and know why they’re taking certain classes and not just doing assignments for the sake of doing assignments” (quoted from an instructor). Shada, Kelly, Cox, and Malik Best Practice Peer review Survey of best practices within a program Feedback from external audiences Process for tying artifacts to competencies Documented resources Facilitating initial faculty meetings Creating consistency New Culture of Assessment 83 Description Two programs demonstrated ways to incorporate peer feedback. One required that students present on their ePortfolio toward the end of the semester and receive informal but guided peer feedback from the class on how to improve their ePortfolio before the end of the semester. The presenter is then also able to provide information and advice (to the students who are not as far along in the process) regarding how much time each section took, what was particularly difficult, etc. Another program assigns small groups of students to a faculty advisor, who then facilitate a peer review process before students submit a draft to their advisor. Peers generally work in teams of two or three. In one instance, an instructor had been advocating for the program to transition to ePortfolios from a traditional paper-based portfolio and although fellow faculty members seemed interested in the idea, the idea was not moving forward. He decided to survey the faculty to learn what assignments were going to the portfolios, what kinds of reflections were being used, and how they hoped the program could do better. Presenting this information was what ultimately got the faculty excited and enthused to move forward. The instructor stated, “that was when I felt we had buy in, was when I wasn’t the one pushing it. When the idea I wanted was coming organically from the faculty. But that required not just providing resources to the faculty, but getting them to reflect and letting them see what their peers were doing and suggesting. At least in our small program, that was a very powerful thing.” One program that focuses on trying to make the ePortfolio become a tool to help their students move on to the professional world, met with two employers in the field to receive feedback on the content of their students’ ePortfolios. One program that uses a competency-based ePortfolio provides students with lists of the possible artifacts that might fit with each competency. Depending on the particular competency, the artifacts may be predetermined, or the student may have the autonomy to decide what piece of academic work fits best there. Some competencies may have one predetermined artifact and one artifact that is open to the student’s choosing. One program–with the help of Academic Technology—developed an in-depth handbook that serves as a guide for both faculty and students on how to use ePortfolios. The handbook includes information such as the ePortfolio content requirements, information on the process, assignment checklists, a guide to using the software, evaluation and grading guides, and a sample peer evaluation form. One program started their ePortfolios by having a faculty retreat and collectively discussing things such as what to name each section of the ePortfolio template, what assignments to include, how much of the students’ grade should be attributed to the ePortfolio and what the core assignments related to the ePortfolio should be (a culminating assignment, a presentation, etc.). Several instructors noted the importance of creating consistency among the faculty, particularly in terms of overall goals and objectives. One program had the faculty collectively design a rubric to use, and although it can be slightly adapted, it has been helpful for students to have that consistency throughout the program. Another instructor also noted that if faculty members are not all on the same page with objectives, the group can run into a lot of problems down the road.