Undertaking a systematic literature review

Undertaking a systematic literature review Dr David Armstrong School of Education, Flinders University David.armstrong@flinders.edu.au (Presentation originally given at Flinders Educational Futures Research Institute [FEFRI], School of Education, Flinders University on Friday 9th October 2015 as part of their research methods series ‘FEFRI matters’). Aims • Discussion of the ‘how’ (practice) and ‘why’ (rationale) inherent in conducting a systematic literature review • To allow consideration of how a systematic lit review often differs from the more usual narrative type of literature review. • The presenter - David Armstrong – will refer to a published, systematic literature review in this discussion: Armstrong, D. (2014). • Opportunity for questions 2 An account of one case • What follows might be characterised as a case study of one systematic lit review with strengths and weaknesses inherent in the case study methodology (illustrative rather than necessarily generalizable) • Case study with a peer-reviewed outcome (a fact) Armstrong, D. (2014). Educator perceptions of children who present with social, emotional and behavioural difficulties: a literature review with implications for recent educational policy in England and internationally. International Journal of Inclusive Education, 18(7), 731-745. 3 An account of a personal case To scale Image source: http: all-free-download.com 4 Why produce a literature review? A common purpose of literature reviews (systematic and other) is to survey and frame key literature in an emerging or poorly/incompletely surveyed field of enquiry. Rozas, L. W., & Klein, W. C. (2010, p. 389) comment: ‘Without a great deal of technical methodological sophistication, those well done reviews provided a more comprehensive intellectual grounding by highlighting connections that went beyond what was known in any individual study to produce a more fully developed understanding of an issue and its larger context.’ Implication: literature reviews of all types often have (or are perceived to have) a practical utility for practitioners, policy-makers and researchers in highlighting these hitherto unknown or unclear connections and patterns. 5 Traditional or systematic? In their book, Doing Your Literature Review: Traditional and Systematic Techniques, Jesson et al (2011, p.15) describe the appeal of undertaking a systematic literature review: ‘The appeal of this style of review’ they suggest, ‘lies in its claim to be a more neutral technical process, which is rational and standardised thereby demonstrating objectivity and a transparent process to the reader…’ Jesson et al (2011, p.15) pragmatically add: ‘These features sit easily in a scientific framework but less so in a more open, interpretative paradigm common in the social sciences…so you need to select the review approach which is most appropriate for your research’ 6 The area (or field) of review Field: Educator perceptions of children with social, emotional and behavioural difficulties (SEBD) Armstrong and Hallett (2012). Study of what 150 educators from all over England and wales wrote about their experience of and perception about children with SEBD, ‘documentary realities’ (Atkinson and Coffey, 2004) The systematic review discussed (Armstrong, 2013) fits with those characterised by Rozas and Klein (2010). • The writer/researcher called on prior experience (reading) of ‘well done’ literature reviews and about what made some of them systematic. 7 Observation: systematic literature reviews published in high-quality, peer reviewed journals are often carried out by researchers who are already well-versed in a field, they are usually an academic practice carried out by mature researchers (Rozas and Klein, 2010). See AER Website: http://www.aera.net/ Image source: http://www.aera.net/Publications/Journals/tabid/10232/D efault.aspx 8 The importance of policy Controversial UK white paper, The Importance of Teaching (2010) claims about behaviour in schools and international research on behaviour • UK MP Michael Gove (Education Minister in 2010) Image source: https://commons.wikimedia.org/wiki/File:Michael_Gove_ cropped.jpg 9 Aims of the literature review • Evaluate policy claims about behaviour in light of published literature (review with a purpose) • Review field at the same time: specifically how educators perceive children or adolescents affected by more severe and persistent difficulties with behaviour (students defined as having Social, emotional and behavioural difficulties) • Be systematic – important given that research in this area traverses psychology and education, calling on educational psychology and developmental psychology • Be thorough 10 Testing times for writing a review Source: Central panel of ‘The temptation of St Anthony’ by Hieronymus Bosch c1450-1516 courtesy of www. Hieronymus Bosch 11 Understanding systematic reviews • Not absolute • Higher level of precision in the conduct of the review, including the definition and application of search categories and terms (inclusion and exclusion of published studies) • Systematic scrutiny of the technical features of literature in a field Boeker, M. Vach, W. and Motschall, E. (2013) Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC Medical Research Methodology, 131 (13) 2-12. 12 Features of systematic reviews • Focus on the methodology and other technical aspects of literature (‘attributes’ in the software NVivo) • population size (small study involving 8 students or 100 schools?) • population type & demographics (teachers and students or just teachers?) • significance (0.05 or 0.01) and power of study • patterns in these technical attributes and what they disclose about the state of research in a field • Presenting this analysis in an accessible way for the non-specialist reader (Armstrong, 2013, p. 736) 13 Armstrong (2014) Databases searched: Scopus, Eric, Psychlit, Psychinfo and also SAGE online journal Database (meta database) Plus journals hand-searched: British Journal of Special Education Educational and Child Psychology Journal of Research in Special Educational Needs British Educational Research Journal Educational Psychology in Practice Emotional and Behavioural Difficulties European Journal of Special Needs Education International Journal of Inclusive education Journal of Special Education Psychology in the Schools Support for Learning 14 Armstrong (2013) Search terms (AND OR) • SEBD AND teacher perception • Teacher perception AND SEBD • EBD AND teacher perception • Teacher perception AND EBD • Support from an experienced academic librarian was helpful The search terms: ‘SEBD AND teacher perception’ plus ‘EBD AND teacher perception’ were both used and in order to address any international differences in the use of EBD or SEBD (see ‘Results and discussion’). It is acknowledged that this review did not include books, book chapters, unpublished, theses, online reports, online ‘blogs’ or the many other ways in which comment and information about issues can be made public in the twenty-first century. This is further discussed in the later ‘Limitations’ section of this paper. (Armstrong, 2013, p. 734) 15 Features of systematic reviews Systematic review tend to dig in to the detail of methodology in the literature they review: Methodologically, the majority of papers identified were qualitative in their orientation and analysed data from a small to medium size population (see Table 1). This ranged from a very small sample: (n=8), Goodman and Burton (2010); through to populations falling within the moderate sampling category: n=391, Poulou and Norwich (2002). Data collection methods ranged widely in studies and included… (Armstrong, 2013 p. 174). 16 Features of systematic reviews Systematic review tend to evaluate the field in light of their attention to the methodological quality and maturity of literature surveyed In contrast, the under-representation of papers with a rigorous research design might also be viewed as disclosing a lack of systematic approach by studies on this topic….For example, it can be suggested that teachers perceptions of SEBD as a category and of children with SEBD were often not sufficiently critically explored in papers reviewed because the chosen research design and research tools used were often too imprecise to analyse data at this level of detail. Only Soles et al. (2008) used several psychometric, standardised measures to independently assess the extent of a child’s difficulties....’ (Armstrong, 2013, p. 176) 17 Reflection & Discussion • Q. what is one key practical implication arising from the last 6 slides for any researcher undertaking a systematic literature review: (clue) 18 Phases of submission Another story…2012-2013 • Emotional and Behavioural Difficulties (missing policy in Wales; policy analysis needs to be developed – helpful) • British Educational Research Journal (BERJ) (not right for our readership and only publish limited lit reviews) and then accepted by • International Journal of Inclusive Education (praise) 19 Implication…? In choosing to write a literature review (systematic or not) you have a restricted number of journals - not all publish literature reviews – some only occasionally…choose carefully Choose carefully but chance still plays a role You are maximising the chance that your submission will be wellreceived if: • • • • • • It is well-written with a strong level of critical analysis Follows a journal’s house style & is submitted per their advice Is accompanied by a letter of introduction Specifies and clearly frames why this is an important field to review For a systematic review, your submission should consider the qualities and attributes set out in this presentation as well those you can apply from further reading about systematic literature reviews It will interest and inform the intended readership 20 A happy end for the researcher 21 References Armstrong, D., and F. Hallett. (2012). Private Knowledge, Public Face: Conceptions of Children with SEBD by Teachers in the UK – A Case Study. Educational and Child Psychology, 29 (4), 77–87. Armstrong, D. (2014). Educator perceptions of children who present with social, emotional and behavioural difficulties: a literature review with implications for recent educational policy in England and internationally. International Journal of Inclusive Education, 18(7), 731-745. DFE (Department for Education). (2010) The Importance of Teaching – The Schools White Paper. London: TSO. Jesson, J., Matheson, L., & Lacey, F. M. (2011). Doing your literature review: traditional and Systematic Techniques. New York: Sage. Rozas, L. W., & Klein, W. C. (2010). The value and purpose of the traditional qualitative literature review. Journal of Evidence-Based Social Work, 7(5), 387-399. 22 Thank you for listening • Questions? 23