Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                

Preliminary reliability of an observer rating scale for assessing medication adherence on psychiatric wards

Journal of Evaluation in Clinical Practice, 2009
...Read more
Preliminary reliability of an observer rating scale for assessing medication adherence on psychiatric wards Mitchell K. Byrne MAppPsych, 1 Frank P. Deane PhD, 2 Aimee Willis BPsych, 3 Barbara Hawkins PhD 4 and Rebecca Quinn PhD 5 1 Lecturer, Clinical Psychology, 2 Director, Illawarra Institute for Mental Health, 3 Research Officer, Illawarra Institute for Mental Health, Illawarra Institute for Mental Health and School of Psychology, University of Wollongong, Wollongong, New South Wales, Australia 4 Director of Research, 5 Director, Center for Adult Services, Western State Hospital, 9601 Steilacoom Boulevard SW, Lakewood, WA, USA Keywords inpatient, measure, medication adherence, psychiatric, rating Correspondence Mitchell K. Byrne School of Psychology University of Wollongong Wollongong NSW 2522 Australia E-mail: mbyrne@uow.edu.au Accepted for publication: 8 February 2008 doi:10.1111/j.1365-2753.2008.00989.x Abstract Objectives Medication non-adherence is a major contributor to poor outcomes following discharge from psychiatric hospitals. It is therefore imperative that staff are able to accu- rately assess the extent to which patients have engaged with treatment. This study presents data on a new observer rating instrument of patient medication adherence. Methods Staff participating in a medication adherence training programme (‘Medication Alliance’) were given a brief overview of the Observer Rating of Medication Taking (ORMT) scale. Participants then watched six video vignettes of patient adherence behav- iour and provided a rating on their scale for each vignette. Participant ratings were then compared with ‘expert’ ratings. Results Percentage agreement between ‘experts’ and participants ranged from a low of 68% through to 98% agreement. Only one vignette was rated significantly differently [Mdiff = 0.33, t(49) = 2.08, P = 0.007], with 32% of people falling outside the expert rating range (all of those ratings being higher). This difference was attributed to between group differences, [F(2,47) = 3.49, P < 0.05] and post hoc assessment suggested that the differ- ences between expert and trainee ratings for the vignette may be explained by trainee characteristics, as distinct from inherent characteristics of the rating scale. Conclusion The ORMT can help mental health professionals identify particular non- adherent behaviours thus facilitating identification and treatment of likely non-adherence before discharge. The scale appears to be accessible to a variety of professions with a range of experience and requires minimum training in order to be used reliably. Introduction Individuals taking prescribed medication have been found to dem- onstrate significant levels of non-adherence [1], with this being especially common in psychiatric patients [2]. Patients may fail to take their medication; they may discontinue their treatment early; or they may continue to take their medication, but deviate from the prescription [3]. These problems are significant throughout the health sector and have led to a need for accurate and practical tools to screen adherence [4]. There are a wide range of methods which have been used to assess medication adherence, including self-reports, prescription monitoring, pill counts, biological markers and electronic moni- toring [5]. However, most methods are open to manipulation, distortion, and misinterpretation [5] and no single method has been found to have successful application to all forms of research in the area [3]. Furthermore, even when the measurement of adherence matches the research question, cut off points for adherence versus non-adherence are often chosen arbitrarily [3]. Judging whether or not any particular method is valid in mea- suring medication adherence is further complicated by the inherent limitations of each method. For example, testing urine and blood drug levels cannot quantify the manner or pattern of compliance and do not give information about extended intervals [6]. Further- more, individual variation in metabolism can often distort bio- logical markers or levels [3]. Self-report is arguably the simplest method to measure compliance and some authors have found self-report to be quite accurate, with 75% of patients identifying themselves as compliant or non compliant consistently with their pill count [7]. While some research has identified self-report as having high sensitivity for repeat non-adherence [4], self-report has more generally been found to be susceptible to unreliable responding, particularly towards patients over estimating com- pliance [3]. The skill of the interviewer and the structure of Journal of Evaluation in Clinical Practice ISSN 1356-1294 © 2009 The Authors. Journal compilation © 2009 Blackwell Publishing Ltd, Journal of Evaluation in Clinical Practice 15 (2009) 246–251 246
questions asked of the individual can influence the accuracy and validity of this method [3]. Pill counts calculate the number of dosage units taken by the patient, but this has also been seen as untrustworthy [6]. There is no guarantee that patients have actually consumed the medication, and it also does not indicate the pattern of adherence [3]. Elec- tronic monitoring records the time and date on each occasion a bottle of medication is opened. However, the opening of a bottle does not necessarily mean that the medication has actually been taken and as such, this method is likely to overestimate adherence [5]. The most significant issue with the range of methods available to measure medication adherence is that they seem to lack validity in that there is inconsistency between measures. The significant discrepancies between measures suggest the need for further research to help improve tools and strategies to assess medication adherence. More importantly, tools need to be open to reliability assessments based on observable behaviour. One option for achieving such an outcome is through the use of observer rating scales to measure adherence behaviour. There is one known example of an observer rating scale for assessing medication adherence. As part of a study by Kemp, Hayward, Applewhaite, Everitt and David [8] to assess the effects of Compliance Therapy in promoting medication adherence in psychiatric patients, a 7-point observer rating scale was used. This scale scored ‘complete refusal’ as ‘1’ and ‘active participation’ as ‘7’. Compliance behaviour was described allowing patient behav- iour to be categorized. Initial ratings of compliance were made by the patients’ primary nurses, then 3 months later a composite measure was obtained using the same scale, but from a variety of sources (including relatives, psychiatrists and general practitio- ners). Evaluations at 6 months were completed by an independent rater. Concurrent validity of the scale was demonstrated through strong correlations between observer assessments and self- report measures. Unfortunately, these ‘observations’ were global summary ratings and the criteria were not well specified, nor was reliability of the measure or reliability between raters reported. The current study describes the development of an observational scale of medication adherence behaviour in inpatient settings and provides a preliminary assessment of its interrater reliability. Previous studies and measures have been found to focus on medication adherence in outpatients. However, a frequent reason for relapse and re-hospitalization is non-adherence with medica- tion [9]. As a result a common goal of inpatient treatment is re-establishment of medication taking and improved adherence behaviour of patients. It is important to acknowledge that psy- chiatric hospitals need to not only focus on the stabilization of patients, but also self-management of their disorder so that upon their discharge they are capable of managing their illness in the community. Unfortunately, there are no behaviourally defined observer rating scales to help systematically assess adherence in inpatient settings. As part of scale development for the current study, informal consultations were made with senior clinical staff and observations were made on the ward of medication dispensing in inpatients settings (Washington State, USA and also New South Wales State, Australia). The most common form of medication dispensing was for a ‘medication call’ at specified times of the day (typically morning, afternoon and evening) when patients would come to a nurses station to receive their medications. The level of prompting at this time varied somewhat but generally involved a ‘call’ for patients in the day rooms to come to collect their medications. Typically, individuals would then go to the nurses’ station (and line up if necessary). Those who did not attend or who were in their rooms would usually be prompted by a nurse or other ward staff. By far this was the most common form of medication dispensing and so we chose to use this as the pivotal event for observational ratings of medication adherence behaviour. Although there were other methods of dispensing, these were relatively infrequent. For example, where staffing allowed, nurses would go to individual patient’s rooms or a medication cart might go from room to room. However, often there were insufficient staff and there were concerns about security of medications and safety with this method. In addition, some wards allow patients to ‘self- medicate’, which involves them holding their own medications and dispensing at the required times. Although such strategies may be ideal for helping patients increase their personal responsibility for self-management, in practice, self-medication on wards was rare, again owing to concerns about security of medications (e.g. theft, sale) and because most wards did not have the facility for secure (locked) storage lockers for individual patients. This study aimed to provide a preliminary assessment of the reliability and levels of agreement of the medication adherence observer rating scale among a cross-section of inpatient ward staff. Methods and materials Participants Data were derived from participants in a medication adherence training programme (‘Medication Alliance’ [10]). In all, 60 par- ticipants completed the programme but data on the medication adherence observer rating scale were only obtained from 50 of these participants. Given that the medication adherence observer rating scale data were anonymous, demographic information was derived from the full sample of 60 participants who completed other programme evaluation activities and provided background descriptive data. As a result, this is can only be considered broadly representative of the 50 participants who provided medication adherence observer rating scale data for this study. Participants were mental health staff from the Western State Psychiatric Hospital in Lakewood, Washington, USA. In all, 58% were female and 42% male, with a mean age of 47.5 years (SD = 10.6 years). Most of the participants were nurses [37%], although a large number were Mental Health Technicians (MHTs – 29%). MHTs do not have formal tertiary qualifications in health services but rather are trained ‘on the job’ to support clinical personnel. The remainder of the participants were institutional counsellors [8%], social workers [7%], rehabilitation specialists [7%], psychiatrists [5%], psychologists [3%] and occupational therapists [3%]. Most had worked for a significant period in their profession (M = 14.9 years, SD = 10.3 years), with the bulk of this experience at Western State Hospital (M = 9.3 years, SD = 7.2 years). The highest educational achievement was under- graduate degree [45%], followed by senior high school [31%] and finally postgraduate qualifications [24%]. Three quarters of the participants [77%] stated that they had never received training in M.K. Byrne et al. Observer rating scale of medication adherence © 2009 The Authors. Journal compilation © 2009 Blackwell Publishing Ltd 247
Journal of Evaluation in Clinical Practice ISSN 1356-1294 Preliminary reliability of an observer rating scale for assessing medication adherence on psychiatric wards Mitchell K. Byrne MAppPsych,1 Frank P. Deane PhD,2 Aimee Willis BPsych,3 Barbara Hawkins PhD4 and Rebecca Quinn PhD5 1 Lecturer, Clinical Psychology, 2Director, Illawarra Institute for Mental Health, 3Research Officer, Illawarra Institute for Mental Health, Illawarra Institute for Mental Health and School of Psychology, University of Wollongong, Wollongong, New South Wales, Australia 4 Director of Research, 5Director, Center for Adult Services, Western State Hospital, 9601 Steilacoom Boulevard SW, Lakewood, WA, USA Keywords inpatient, measure, medication adherence, psychiatric, rating Correspondence Mitchell K. Byrne School of Psychology University of Wollongong Wollongong NSW 2522 Australia E-mail: mbyrne@uow.edu.au Accepted for publication: 8 February 2008 doi:10.1111/j.1365-2753.2008.00989.x Abstract Objectives Medication non-adherence is a major contributor to poor outcomes following discharge from psychiatric hospitals. It is therefore imperative that staff are able to accurately assess the extent to which patients have engaged with treatment. This study presents data on a new observer rating instrument of patient medication adherence. Methods Staff participating in a medication adherence training programme (‘Medication Alliance’) were given a brief overview of the Observer Rating of Medication Taking (ORMT) scale. Participants then watched six video vignettes of patient adherence behaviour and provided a rating on their scale for each vignette. Participant ratings were then compared with ‘expert’ ratings. Results Percentage agreement between ‘experts’ and participants ranged from a low of 68% through to 98% agreement. Only one vignette was rated significantly differently [Mdiff = 0.33, t(49) = 2.08, P = 0.007], with 32% of people falling outside the expert rating range (all of those ratings being higher). This difference was attributed to between group differences, [F(2,47) = 3.49, P < 0.05] and post hoc assessment suggested that the differences between expert and trainee ratings for the vignette may be explained by trainee characteristics, as distinct from inherent characteristics of the rating scale. Conclusion The ORMT can help mental health professionals identify particular nonadherent behaviours thus facilitating identification and treatment of likely non-adherence before discharge. The scale appears to be accessible to a variety of professions with a range of experience and requires minimum training in order to be used reliably. Introduction Individuals taking prescribed medication have been found to demonstrate significant levels of non-adherence [1], with this being especially common in psychiatric patients [2]. Patients may fail to take their medication; they may discontinue their treatment early; or they may continue to take their medication, but deviate from the prescription [3]. These problems are significant throughout the health sector and have led to a need for accurate and practical tools to screen adherence [4]. There are a wide range of methods which have been used to assess medication adherence, including self-reports, prescription monitoring, pill counts, biological markers and electronic monitoring [5]. However, most methods are open to manipulation, distortion, and misinterpretation [5] and no single method has been found to have successful application to all forms of research in the area [3]. Furthermore, even when the measurement of adherence 246 matches the research question, cut off points for adherence versus non-adherence are often chosen arbitrarily [3]. Judging whether or not any particular method is valid in measuring medication adherence is further complicated by the inherent limitations of each method. For example, testing urine and blood drug levels cannot quantify the manner or pattern of compliance and do not give information about extended intervals [6]. Furthermore, individual variation in metabolism can often distort biological markers or levels [3]. Self-report is arguably the simplest method to measure compliance and some authors have found self-report to be quite accurate, with 75% of patients identifying themselves as compliant or non compliant consistently with their pill count [7]. While some research has identified self-report as having high sensitivity for repeat non-adherence [4], self-report has more generally been found to be susceptible to unreliable responding, particularly towards patients over estimating compliance [3]. The skill of the interviewer and the structure of © 2009 The Authors. Journal compilation © 2009 Blackwell Publishing Ltd, Journal of Evaluation in Clinical Practice 15 (2009) 246–251 M.K. Byrne et al. questions asked of the individual can influence the accuracy and validity of this method [3]. Pill counts calculate the number of dosage units taken by the patient, but this has also been seen as untrustworthy [6]. There is no guarantee that patients have actually consumed the medication, and it also does not indicate the pattern of adherence [3]. Electronic monitoring records the time and date on each occasion a bottle of medication is opened. However, the opening of a bottle does not necessarily mean that the medication has actually been taken and as such, this method is likely to overestimate adherence [5]. The most significant issue with the range of methods available to measure medication adherence is that they seem to lack validity in that there is inconsistency between measures. The significant discrepancies between measures suggest the need for further research to help improve tools and strategies to assess medication adherence. More importantly, tools need to be open to reliability assessments based on observable behaviour. One option for achieving such an outcome is through the use of observer rating scales to measure adherence behaviour. There is one known example of an observer rating scale for assessing medication adherence. As part of a study by Kemp, Hayward, Applewhaite, Everitt and David [8] to assess the effects of Compliance Therapy in promoting medication adherence in psychiatric patients, a 7-point observer rating scale was used. This scale scored ‘complete refusal’ as ‘1’ and ‘active participation’ as ‘7’. Compliance behaviour was described allowing patient behaviour to be categorized. Initial ratings of compliance were made by the patients’ primary nurses, then 3 months later a composite measure was obtained using the same scale, but from a variety of sources (including relatives, psychiatrists and general practitioners). Evaluations at 6 months were completed by an independent rater. Concurrent validity of the scale was demonstrated through strong correlations between observer assessments and selfreport measures. Unfortunately, these ‘observations’ were global summary ratings and the criteria were not well specified, nor was reliability of the measure or reliability between raters reported. The current study describes the development of an observational scale of medication adherence behaviour in inpatient settings and provides a preliminary assessment of its interrater reliability. Previous studies and measures have been found to focus on medication adherence in outpatients. However, a frequent reason for relapse and re-hospitalization is non-adherence with medication [9]. As a result a common goal of inpatient treatment is re-establishment of medication taking and improved adherence behaviour of patients. It is important to acknowledge that psychiatric hospitals need to not only focus on the stabilization of patients, but also self-management of their disorder so that upon their discharge they are capable of managing their illness in the community. Unfortunately, there are no behaviourally defined observer rating scales to help systematically assess adherence in inpatient settings. As part of scale development for the current study, informal consultations were made with senior clinical staff and observations were made on the ward of medication dispensing in inpatients settings (Washington State, USA and also New South Wales State, Australia). The most common form of medication dispensing was for a ‘medication call’ at specified times of the day (typically morning, afternoon and evening) when patients would come to a © 2009 The Authors. Journal compilation © 2009 Blackwell Publishing Ltd Observer rating scale of medication adherence nurses station to receive their medications. The level of prompting at this time varied somewhat but generally involved a ‘call’ for patients in the day rooms to come to collect their medications. Typically, individuals would then go to the nurses’ station (and line up if necessary). Those who did not attend or who were in their rooms would usually be prompted by a nurse or other ward staff. By far this was the most common form of medication dispensing and so we chose to use this as the pivotal event for observational ratings of medication adherence behaviour. Although there were other methods of dispensing, these were relatively infrequent. For example, where staffing allowed, nurses would go to individual patient’s rooms or a medication cart might go from room to room. However, often there were insufficient staff and there were concerns about security of medications and safety with this method. In addition, some wards allow patients to ‘selfmedicate’, which involves them holding their own medications and dispensing at the required times. Although such strategies may be ideal for helping patients increase their personal responsibility for self-management, in practice, self-medication on wards was rare, again owing to concerns about security of medications (e.g. theft, sale) and because most wards did not have the facility for secure (locked) storage lockers for individual patients. This study aimed to provide a preliminary assessment of the reliability and levels of agreement of the medication adherence observer rating scale among a cross-section of inpatient ward staff. Methods and materials Participants Data were derived from participants in a medication adherence training programme (‘Medication Alliance’ [10]). In all, 60 participants completed the programme but data on the medication adherence observer rating scale were only obtained from 50 of these participants. Given that the medication adherence observer rating scale data were anonymous, demographic information was derived from the full sample of 60 participants who completed other programme evaluation activities and provided background descriptive data. As a result, this is can only be considered broadly representative of the 50 participants who provided medication adherence observer rating scale data for this study. Participants were mental health staff from the Western State Psychiatric Hospital in Lakewood, Washington, USA. In all, 58% were female and 42% male, with a mean age of 47.5 years (SD = 10.6 years). Most of the participants were nurses [37%], although a large number were Mental Health Technicians (MHTs – 29%). MHTs do not have formal tertiary qualifications in health services but rather are trained ‘on the job’ to support clinical personnel. The remainder of the participants were institutional counsellors [8%], social workers [7%], rehabilitation specialists [7%], psychiatrists [5%], psychologists [3%] and occupational therapists [3%]. Most had worked for a significant period in their profession (M = 14.9 years, SD = 10.3 years), with the bulk of this experience at Western State Hospital (M = 9.3 years, SD = 7.2 years). The highest educational achievement was undergraduate degree [45%], followed by senior high school [31%] and finally postgraduate qualifications [24%]. Three quarters of the participants [77%] stated that they had never received training in 247 Observer rating scale of medication adherence relation to medication adherence prior to their participation in Medication Alliance training. Measures The Observer Rating of Medication Taking (ORMT) (see Appendix) ranges from 1 (‘fully non-adherent’) through to 7 (‘fully adherent’). A copy of the ORMT is appended to this article. Each of the 7 points has behavioural descriptors. These are supplemented with descriptors for rating between each of the 7 points (e.g. ‘between 3 and 4’). There is a broad orienting guide with ratings from 7 to 4 that generally indicate adherence with varying degrees of prompting, and from 4 to 1 which indicate nonadherence with varying degrees of resistance. On the ward, each patient’s medication taking behaviour would be rated once per day for one week using the ORMT. If the patient has multiple doses across each day, then one rating is taken sequentially for each medication period, rotating between periods across the week. An average score for the individual patient across the week can then obtained to monitor progress over time. Materials The study used six brief video vignettes that provided examples of different responses to medication taking on a ward as stimulus materials for participants to rate adherence levels. Transcripts for the vignettes are provided below along with the length of each video. The vignettes were designed to assess the range of ratings possible on the ORMT, but also included vignettes that were likely to be close to each other in terms of adherence ratings. This would allow a test of the range, but also the capacity to assess relatively subtle differences in behaviour. The videos were rated independently by the first two authors, who have extensive experience in medication adherence training (e.g. ref. [10]). In addition, a third expert who was an experienced clinical psychologist also provided an independent rating. All three independent expert ratings (ER) had the scores in the same rank order and all except one (Vignette B) were within one rating point of each other. The average of these ratings, rounded to the nearest 0.5 was taken as the ‘expert’ ratings for each of the vignettes. The final ER for each vignette were: A = 2.0, B = 4.5, C = 3.0, D = 6.0, E = 6.5 and F = 7.0. Vignette A (32 seconds) Two staff approach a patient sitting in the day room reading a paper. Staff: ‘John, John listen mate we’ve been talking to you for about, what the last twenty minutes or something like that about taking your medication, ok? Listen if you are not going to take it voluntarily we are going to have to give you an injection.’ Patient tosses paper to the floor as he rapidly jumps up out of seat with a broad stance, legs apart, hands on hips and very (overly close) to the smaller of the two staff and glares at her. The staff takes a step backward and states calmly but firmly . . . Staff: ‘Now John what you need to do now is just sit down and calm down.’ Patient: Turns rapidly. Roughly knocks a chair out of the way knocking it over and says, ‘I’m going to my room.’ (patient exits to his room) 248 M.K. Byrne et al. Vignette B (20 seconds) Patient is listening to the radio in the day room with headphones on. Staff: ‘Mary, Mary, I’ve called you a couple of times to come and get your medications. Did you hear me the first time?’ Patient: (looks up distracted, removes headset and states in neutral tone) ‘Yes’ Staff: (in friendly manner) ‘Alright well why don’t we go through now and get it?’ Patient: (slowly rises, places headset onto the table and walks towards the nurses station to get her meds without saying anything). Vignette C (22 seconds) Patient sitting in the day room next to another patient. Patient 1: ‘How ya going?’ Patient 2: ‘Oh, not too bad. How are you?’ Patient 1: ‘Good, good.’ Nursing staff approaches patient. As the nurse approaches the patient the target patient says to the other patient. Patient: (cynical and hostile directed to the other patient) ‘Here comes the damned pusher man again, they’re worse than drug dealers. Staff: (firmly) ‘Come on John it’s time to take your medication. Come through with me.’ Patient: (stands rapidly and says in hostile resentful manner) ‘So if I don’t take my medication you won’t let me out for a damned cigarette!’ Staff: ‘Come on man come through, this way.’ Vignette D (11 seconds) Nurse approaches a patient on the ward who is reading a magazine. Staff: ‘Hey Paul, you need to get your medication now OK?’ Patient stops reading magazine immediately and gets up from the couch, placing the magazine onto the table and goes to get the medication. Vignette E (48 seconds) Clock on the nursing station wall shows 18.30 hours. Patient approaches the nursing station unprompted. Patient: ‘Is it medication time yet?’ Staff: ‘Ah, no. Dorothy not, not quite yet. Ahem you’ve got about another half hour to go. Yeah, so how about you come back in about half an hour – yeah?’ Patient: ‘Oh. OK. Hey, are we going to have another one of those Recovery groups again?’ Staff: ‘Ahem we might be doing that. I’ll talk to you about that later, OK Dorothy?’ Patient: ‘OK.’ Staff: ‘see you in half an hour.’ Patient: ‘OK.’ Fades out, then fades in to show Clock at 19.00 hours. Nurse: ‘Hi Dorothy. By the way do you know what you’re taking?’ Patient: ‘Yep.’ Nurse: ‘What meds you’re taking?’ © 2009 The Authors. Journal compilation © 2009 Blackwell Publishing Ltd Observer rating scale of medication adherence M.K. Byrne et al. Table 1 A comparison of expert and participant ratings of medication adherence in the vignettes Trainee ratings Comparisons of trainee against expert rating (ER) Vignette ER M SD % within 0.5 of ER % below ER % above ER A B C D E F 2.0 4.5 3.0 6.0 6.5 7.0 1.92 4.83* 3.20 5.99 6.68 6.96 0.48 0.82 0.75 0.62 0.57 0.16 80 68 72 90 96 98 28 36 22 16 8 8 12 44 34 28 60 0 *P < 0.05, 2-tailed, Trainee mean ratings were significantly different from the Expert rating. Patient: ‘Yes, two white ones.’ Nurse: ‘You’re right. I wonder, do you know what they are for?’ Patient: ‘Yep.’ (picks up cup with medication, turns and walks away from the nursing station) Vignette F (15 seconds) Patient approaches the nursing station, taps on the door frame and asks staff unprompted . . . Patient: ‘Is it medication time yet? Staff: (who is reading some reports) ‘Oh Hi Dot, yes it is. Hang on one second.’ (get medications from cupboard). ‘So you know your medication recently changed?’ Patient: ‘Yes.’ Procedure As part of a training session on medication adherence (‘Medication Alliance’), participants were trained in the use of the ORMT. All participants were informed that participation was voluntary and that their responses would be anonymous. They were told not to put their names or any other identifying information on their forms. Data were derived from three different training groups. Participants were first given a brief overview of the ORMT measure. This involved the trainer reading the descriptions for the anchors of the 7, 1 and 4 ratings (in that order). Participants then reviewed each of the other points to become familiar with the scale. This whole process took approximately 10 minutes. They then watched the six video vignettes and provided a rating on their scale for each. They also had access to the scripts to refer to if needed. After each video, participants were given about 2 minutes to make their rating. After all the ratings were completed, each of the vignettes was provided with an agreed ‘class’ ranking on a white board based on a brief class discussion and a show of hands to indicate the general consensus score. The ER of each vignette was then revealed and discussion about what differentiated each of the ratings was provided. The whole session took approximately 30 minutes for each group. All ratings were returned anonymously to the researchers at the end of the session. Results The mean ratings from the trainees along with the ER can also be found in Table 1. First, the percentage of trainees who provided ratings within 0.5 either side of the ER were considered to have a © 2009 The Authors. Journal compilation © 2009 Blackwell Publishing Ltd satisfactory level of agreement. Percentages ranged from a low of 68% agreement (Vignette B) through to 98% (Vignette F). The percentage who overestimated and underestimated based on the expert value is also provided for descriptive purposes, with these values not including those who matched the ER. For example, for Vignette A, 28% rated below the ER and 12% above therefore 60% matched the ER. Finally, one sample t-tests were conducted using the ER as the criterion value. Using a 2-tailed test, there was a significant difference between ER and trainee mean ratings for Vignette B (Mdiff = 0.33), t(49) = 2.08, P = 0.007), with 32% of people falling outside the ER range (all of those ratings being higher). The proportion of participants whose ratings ranked the vignettes in the correct order, from least adherent to most adherent (i.e. A, C, B, D, E, F) was also investigated. It was found that 48% of participants ranked the Vignettes in the correct order. Owing to some of the vignettes being quite close in their ER, quite a few equal rankings resulted, especially for Vignettes E and F where 50% of rankings were equal. As a result, the proportion of those who correctly ranked the vignettes was calculated treating vignettes E and F as the same rank. After making this adjustment, 76% of participants were classified as ranking the vignettes in the correct order. Finally, in order to explore the possibility that differences in group characteristics or training might be responsible for some of the variability in ratings, a one-way anova was conducted for the three training groups as the Independent variable and each of the vignette ratings as dependent variables. There were no significant differences between groups except for ratings on Vignette B, F(2,47) = 3.49, P < 0.05. Post hoc Least Squares Difference tests indicated that for this vignette, the mean rating by Group 1 (M = 4.44, SD = 0.56, n = 18) was significantly lower than both Group 2 (M = 5.04, SD = 0.94, n = 14) and Group 3 (M = 5.06, SD = 0.85, n = 18) (both P < 0.05). This suggests that the differences between expert and trainee ratings for Vignette B in part be explained by trainee or training characteristics as distinct from inherent characteristics of the rating scale. It is notable that the mean for Group 1 was closer to that of the ER. Discussion These results are promising in terms of using this measure to help mental health professionals identify particular non-adherent behaviours. It appears that in the majority of cases the trainees were relatively accurate in their ratings, with the averages being quite consistent with the ER and a high percentage of trainees 249 Observer rating scale of medication adherence rating within 0.5 either side of the ER. Participants demonstrated most difficulty when rating vignette B. Analysis of between group differences among trainees suggested that the relatively low reliability for vignette B may in part be explained by either trainee characteristics (occupation) or variation in training experience. Unfortunately, we could not match trainee descriptive characteristics with vignette ratings to test this further. Vignettes B and C both had relatively low percentages of trainees rating the vignette correctly compared with the ER. It may be that these vignettes are more difficult to rate in terms of identifying the behaviours represented. It is also possible that the scale does not sufficiently allow differentiation in these ranges. This suggests the need for closer examination of the training process. However, if this does not improve rating accuracy then the scale may require further refinement. Further to this, participant ratings at the extreme ends of the scale were highly accurate. This adds additional validity to the premise that future developments should focus on refining ratings around the more central parts of the scale. The proportion of correct rankings was relatively low in part owing to the number of equal ratings for Vignettes E and F. However, when vignettes E and F were combined into one, the proportion of participants who ranked the vignette correctly, rose to 76%, indicating that the equal ratings of these two vignettes was responsible for a significant number of incorrect rank orders in the first instance. Although in both vignettes the patient was proactive in seeking their medication, in one (E) she initially arrived at the wrong time. Equally high ratings of adherence were often provided owing to the lack of attention to this issue. Further to this, in the hospital setting where training occurred, such proactive adherence tended to be viewed as highly desirable and not problematic in relation to the relatively frequent problems with non-adherence. Future studies might also explore the impact of occupation and years of experience in the mental health sector, in order to determine whether these are contributing variables to accurate medication adherence assessment. This would shed light on the differential training needs of mental health staff in order to ensure that the scale is used reliably. None the less, the scale does appear to be accessible to a range of professions with a range of experience, at least in an analogue environment. However, the utility of the scale lies in its use and reliability in real life settings. Actual observations of medication taking by mental health professionals need to be rated and this needs to be compared with rates of improvement and health gain. 250 M.K. Byrne et al. There remains some way to go before a reliable and informative process for assessing inpatient adherence behaviour is achieved. The present study has provided a beginning to that process and in time should contribute to the development of clinical management plans for inpatients in mental health facilities. More specifically, it will allow more systematic assessment of the progress and skills required to increase a person’s ability to more independently manage their own medication. References 1. Orr, A., Orr, D., Willis, S., Holmes, M. & Britton, P. (2007) Patient perception of factors influencing adherence to medication following kidney transplant. Psychology, Health and Medicine, 12 (4), 509–517. 2. Stephenson, B. J., Rowe, B. H., Haynes, R. B., Macharia, W. M. & Leon, G. (1993) Is this patient taking the treatment as prescribed? Journal of the American Medical Association, 269, 2779–2781. 3. Farmer, K. C. (1999) Methods for measuring and monitoring medication regimen adherence in clinical trials and clinical practice. Clinical Therapeutics, 21 (6), 1074–1090. 4. Svarstad, B. L., Chewning, B. A., Sleath, B. L. & Claesson, C. (1999) The brief medication questionnaire: a tool for screening patient adherence and barriers to adherence. Patient Education and Counseling, 37, 113–124. 5. Byerly, M. J., Thompson, A., Carmody, T., Bugno, R., Erwin, T., Kashner, M. & Rush, A. J. (2007) Validity of electronically monitored medication adherence and conventional adherence measures in schizophrenia. Psychiatric Services, 58 (6), 844–847. 6. Feinstein, A. R. (1990) On white-coat effects and the electronic monitoring of compliance. Archives of International Medicine, 150, 1377– 1378. 7. Haynes, R. B., Taylor, D. W., Sackett, D. L., Gibson, E. S., Bernholz, C. D. & Mukherjee, J. (1980) Can simple clinical measurements detect patient noncompliance? Hypertension, 2, 757–764. 8. Kemp, R., Hayward, P., Applewhaite, G., Everitt, B. & David, A. (1996) Compliance therapy in psychotic patients: Randomised controlled trial. British Medical Journal, 312 (7027), 345–349. 9. Ascher-Svanum, H., Faries, D. E., Zhu, B., Ernst, F. R., Swartz, M. S. & Swanson, J. W. (2006) Medication adherence and long-term functional outcome in the treatment of schizophrenia in usual care. Journal of Clinical Psychiatry, 67 (3), 453–460. 10. Byrne, M. K., Deane, F. P., Lambert, G. & Coombs, T. (2004) Enhancing medication adherence: clinician outcomes from the Medication Alliance training program. Australian and New Zealand Journal of Psychiatry, 38, 246–253. © 2009 The Authors. Journal compilation © 2009 Blackwell Publishing Ltd M.K. Byrne et al. Observer rating scale of medication adherence Appendix Observer rating of medication-taking behaviour on the ward Patient’s Name:___ Rater’s name:___ Date:___ Time:___ Please rate the patient’s medication adherence behaviour by rating the least adherent behaviour observed. Place a cross on the line that best reflects their behaviour. Note: scores from 7 to 4 relate to prompting. Scores less than 4 relate to resistance. 7 = The patient always presents for their medication at the designated place and time. Never requires prompting. Between 6 & 7 = The patient mostly presents for their medication at the designated place & time. Responds immediately to minimal & indirect prompting (e.g. ‘Mary, what should you be doing now?’). Fully adherent to medication 7 A little prompting 6 A lot of prompting 5 Passively adheres 4 6 = The patient occasionally fails to present for their medication. Responds immediately to minimal but direct prompting (e.g. ‘Mary, it’s time to take your medication.’). Between 5 & 6 = The patient often fails to attend for their medication but responds to direct prompting. 5 = The patient often requires a lot of prompting and encouragement to engage in medication taking. They only rarely self-initiate and often need help to access the medication. Between 4 & 5 = The patient almost always needs reminding & direct prompting to get their medication. They only rarely take responsibility for taking medication without prompting. They may need support & prompts such as being accompanied to the clinic room, pouring a cup of water & being handed the medication. Adherent with prompting No prompting 4 = The patient passively adheres. They do not self-initiate & do not access medication without direction. They require constant reminders but do not resist taking medication. Between 3 & 4 = The patient occasionally delays or procrastinates when required to attend for medication. They exhibit mild signs of resistance such as complaining to staff about medication, or nonverbal signs such as screwing up their face when swallowing tablets. They occasionally require some persuading to take their medications. Between 2 & 3 = The patient is verbally hostile towards staff about medication taking. They are openly resistant but comply reluctantly upon direction. Require incremental amounts of warning from staff in order to comply. If not openly hostile & resistant, they may be covertly resistant and may be observed trying to hide medications (e.g. ‘cheeking’ meds and pretending to swallow). A little resistance 3 A lot of resistance 2 Requires restraint 1 2 = The patient may engage in physical intimidation towards staff or aggression directed at property as a result of resistance around medication taking. Between 2 & 1 = The patient is physically aggressive or resistant towards staff when refusing to take medication. They may require presence of extra staff for safety purposes & as a back up in case the patient requires physical restraint. 1 = Physical restraint is required in order to administer medications. © 2009 The Authors. Journal compilation © 2009 Blackwell Publishing Ltd Non-adherent with resistance 3 = The patient complains about medication at various times across the day (not limited to medication taking times). They engage other patients in discussion about medications, or may make comments that accuse staff of coercion around medication taking. Almost always require persuading to take medications. Fully non-adherent to medication 251
Keep reading this paper — and 50 million others — with a free Academia account
Used by leading Academics
Eric S . Kim (UBC)
University of British Columbia
Thomas L Webb
The University of Sheffield
Sascha Gruss
Universität Ulm
Christian Dunker
Universidade de São Paulo