Measuring the unmeasurable

MEASURING THE UNMEASURABLE? © Julia Horváth (International Pető Institute) horvath@peto.hu © Ildiko Kozma (International Pető Institute) kozma@t-online.hu © Aniko Salga (International Pető Institute) horvath@peto.hu It has become evident that in order to observe, assess and record the condition of motor disabled children at individual and group level, which then set the direction for development, it is necessary to provide well documented common criteria which can then be used in different contexts. During conductive education the child is motivated to achieve higher levels of development and the purposefulness of education and plan-making are determinants of the development level of the dysfunctioning and their achievements and also form the basis for future advancements. As the education process is dynamic and constantly changing, so the evaluation of results cannot be limited to single, occasional actions but is rather a continuous process. This process can be analysed using a set of scales which have been designed to measure seven different areas of development based on dynamic (operative) observation. Keywords: conductive education, Pető method Conductive Education (Pető method) is a worldknown educational system which based on the residual capacity of the damaged brain. One of the basic elements of these system is the observation The effectiveness of Conductive Education (and every education ) depends upon the results, the progress of the children.The authors are working on the measurement scale –based on observation- measuring the progress of the children in the human functions. The working group responsible for carrying out diagnostic assessment was set up almost 20 years ago and has since followed the theses laid down by professor Pető and preserved and developed the tradition. Simultaneously with the spread of conductive education the need for making systematic measurements also became apparent and so the adaptation of the system to environments outside Hungary and its measurement emerged together. In the context of conductive education several researchers, e.g. MacKay (1993), have commented on the fact that it is almost impossible to assess the effectiveness of an education process from 1 just one aspect. Various attempts have been made to evaluate motor disabled children's performance in different environments and under different conditions, referring to the effectiveness of conductive education at different levels. Axioms Conductive education is a learning process. (Elements: motivation intention - planning - execution - adaptation - product.)Measurings and tests measure partial elements. (Standards of those elements are not always set, not even in medicine. In education, empirical research into what to expect at a given age is certainly scarce.) Medical and instrumental methods for measuring partial activities of CP children are already available or under elaboration and empirical studies are published in scientific reviews. There is no standardised educational measurement instruments for individuals with CP.(Cerebral Palsy) We have a measurement instrument of process diagnostics which is now being tested and its efficiency measured. We have to make use of the available variety of instruments and define competence areas. The primary goal of educational research is to employ scientific results in practice and to offer educators measurable and valid standards. Methods to be recommended should be comprehensible and available and, in addition to the problems of condition or function, show the direction of development. Evaluation in education In the 1970's the question of how to measure the efficiency of education and instruction came to the fore. Tyler's model was introduced into educational practice in Hungary in the 1970's and was complemented with the effects of the system concept. Evaluation was seen as a process where values are allocated directly to some phenomenon or performance and indirectly to the goal and the process generating the phenomenon. When ascertaining the value we look for relations between the goal, the process and the achieved result. Necessary corrections can be pursued on the basis of the feedback concerning the excess and the nature of the relation between expectation and reality. We may correct the goal and/or optimise the process or rate the outcome. Pedagogical evaluation: a methodologically varied method, adjusted to the function and goal of evaluation, employing feedback and covering all pedagogical categories and phenomena (goals, contents, processes, environment, conditions, results etc.). Types of evaluation. Scriven recommended that we should distinguish between formative and summative evaluation. Later on a third, the diagnostic function was added. The objective of diagnostic evaluation is to explore the situation. Before deciding, intervening and developing in educational matters it is advisable to gather information on pupils' circumstances at the beginning of the given phase of education/instruction, whether they meet the requirements and which are the areas where they fall behind their peers or excel. The objective of formative evaluation is to control, develop and shape during the process. It is meant to reinforce success in learning and to explore errors and difficulties rather than classify. Thus the necessary corrections can be made in terms of the goals, the content and the process of education and instruction. 2 The objective of summative evaluation is classification. At the end of a particular phase of education/instruction its objectives are summary and final classification. Pupils are rated on the basis of their performance. This action also has a screening function. The primary condition of its efficiency is that it has to provide objective, valid and reliable data. Judgement is a transition between qualitative and quantitative evaluation. In this case, personal characteristics, behaviours and performances are classified as acceptable or unacceptable. Judgement is made whether the behaviour, performance or some other quality of the person in question differs from a measure which has to a certain extent been set. Once we try to put the personal characteristics, behaviours and performances to be evaluated on a scale and express the levels of the assessed areas with the degrees of the scale, we are approaching quantitative interpretation and thus carrying out estimation. Evaluation by measuring is much more accurate. It means that the scale set on the measurement instrument is compared to the quality to be examined. Pedagogical evaluation can hardly be 'accurate', for an objective measuring of the pupils' emotional attitudes, personal characteristics, behaviours, various performances etc. is no easy task. In the past decades, however, scientific research has created more and more accurate measurement instruments and in education new possibilities have opened up for adapting measuring. Tests are typical instruments of pedagogical measurement. A wide variety of tests have been created in order to measure knowledge, intelligence, cognitive competence, different abilities, aspects or components of the personality etc. Methodological conditions of measuring Objectivity. The evaluation must be objective and unbiased. Validity represents a complex problem of measuring. The greatest challenge is to ascertain whether the evaluation method chosen actually measures what it was meant to measure. Reliability means that the repeated measuring pf a particular quality will provide the same result. To prove this is generally not too difficult in natural sciences, with pedagogical measurements it is much more difficult. In terms of validity and reliability different levels are accepted. In cases of judgement or estimation, the evaluation may be carried out by a teacher. As regards classification and selection, the index-numbers must possess the characteristics of standard evaluation. Appraising the success of education, especially performances linked with motor action and assessing progress as compared to the initial condition on record is particularly difficult in the case of CP children or adults. While physical measurements provide more and more accurate data due to the advances in technology, pedagogical measurements entail greater risk that mistakes may occur. Pedagogical measurements will never be fully objective. The reliability of our findings depends on three factors: the instrument we work with, the person carrying out the assessment and the circumstances. Thus while preparing instruments to record pedagogical phenomena we must do our best to create accurate questions, unambiguous categories, clear and comprehensible instructions and evaluation criteria. Even with carefully constructed measurement instruments, the knowledge, prejudiced or unprejudiced attitude, evaluation skills and current condition of the assessor 3 will certainly have an impact on the reliability of our findings. The circumstances of the assessment may also contribute to the reliability of the data. The assessor's obligation is twofold: to increase reliability to the maximum level and to remain impartial. A really correct researcher is ready to indicate the deficiencies and weaknesses of the measurement instrument he/she is using and draws the conclusions accordingly. We also share this ambition. Observation The aim of diagnostic in conductive education is to detect the situation. In a broader sense: to get comprehensively acquainted with the individual whose development is impeded (methods employed: psychical examination, observation, psychological tests). In the narrower sense: conductive pedagogical activity (which is a diagnostic procedure in the course of which interrelations are lookoed for between the goal, the process and the achieved result. Corrections are made subject to feedback concerning the extent and nature of relations between expectations and reality (the goal and/or the process are optimised and the result is rated. A fundamental method of education, it contributes to realising pedagogical principles by exploring pedagogical facts. Observation:  helps us determine the characteristics of the education process;  provides a picture of the mechanism of the process;  helps us follow the development and changes of phenomena;  reveals the effects of education methods;  provides experience and data. By adopting experiences and interpreting data we may explore the principles of effect mechanisms in education. Observation in the conductive education process Conductive observation is an integral part of our education practice. Along with facilitation methods, observation is a core element of CE, an essential part of development work. Ongoing observation helps us recognise results, describe performances, explore the possibilities of further progress and try new procedures. It also enables us to detect errors, make amendments, control and evaluate our pedagogical work. Our findings serve as a basis for determining the task system of the educational programme and for elaborating specific facilitations for the individuals. The observation process is launched when we first meet the dysfunctioning individual. To ensure objectivity, the real education process is observed. Observation is accomplished directly and personally. Conductive observation contributes to systematic planning, the selection of varied methods and the realisation of differentiation. Conductive observation is comprehensive. The individual's personality and the development process are observed globally. Manifestations of the personality are registered without highlighting performances. Psychic and motor phenomena are to be assessed in the system of interrelations. Interest, attention and memory have a great impact on the execution of movements. The quality of the solution is also affected by the individual's attitude towards the task, the activity to be performed. Observation must be systematic as well. The role of observation is equally important in task and 4 spontaneous situations. Observing the individual in a task situation provides the opportunity to see their learning skills, appropriate answers and reactions, task consciousness, attention, motivation basis. Observation in a spontaneous situation provides information on the dysfunctioning person's adaptation skills. Conductive observation is operative, progressive and comparative After determining the problems and setting the aims, we define tasks, elaborate solutions and simultaneously select the phenomena to be observed. Dysmotoric solutions may reappear and need correction. With the help of operative observation we may limit the occurrence of such solutions and change the methods of assistance. Operative observation also contributes to the daily adaptation and internalisation of already attained knowledge and desirable capacities. Progressivity means that our observation primarily focuses on the process and pace of development. When examining the dysfunctioning individuals' system of activities and variations of practising, the emphasis is on changes and progress. The progressive observation of the programme contributes to the adaptation of different task solutions in more advanced and more complex situations. Observation is comparative when focusing on the group as a whole. During the group sessions dysfunctioning persons carry out the same tasks, pursue the same activities. Their individual solutions, however, are differentiated. Comparative observation enables us to compose dynamic groups that promote progress. The positive impact of the group prevails and individuals are incorporated in group work. Comparative observation contributes to the organisation of the education process, the implementation of differentiation and thus guarantees specific progress for the individuals. Recording observation Conductive observation is performed with a purpose, thus our findings need recording. Later on the records may be collected and elaborated for our research. Recording our experiences enables us to detect uniformity and specificities related to different groups and educational situations and reveal the connections between cause and effect. Subject to the criteria, our findings may be recorded in full or partial (selective) protocols. A full protocol must comprise all data, observed phenomena, verbal and non-verbal manifestations. When preparing a partial or selective protocol, we must define the precise subject, the content of observation and the concrete phenomenon to be observed and, subject to the above, record all data, the activity and manifestations of the dysfunctioning individuals or the activity of the conductors. We may use a system of categories for writing down our observations. First we have to define the criteria containing the essential characteristics related to the phenomenon and the categories in order to classify the units. Diagnostic assessment methods that are often used for the appraisal of performances of people with disabilities, including those with motor disabilities. The applied methods of movement analysis can be classified according to the procedure pursued. 5 Motodiagnostics Motoscopy (descriptive) Slight motor coordination disorders can often be detected by motoscopic examination.The assessor observes and desribes the movements and posture of the individual. If the description has enough detail and at the same time focuses on the essential facts, it provides a realistic picture of the child's general and motor behaviour. Reflecting personal views, this description is not suitable for scientific evaluation. To make it more objective, observation may be carried out in a set situation or motor categories may be defined in advance. If well constructed questionnaires, symptom and performance lists and evaluation sheets are applied, motoscopy may meet the requirements of test correctness, i.e. objectivity, reliability and validity and thus be suitable for diagnostic purposes. In indirect motoscopy examinations the assessor gathers information by asking a third party goal oriented questions about the person to be assessed e.g. about the independent execution of everyday motor functions or the amount of help needed for performing those functions. This procedure is called function status as well. Margaret Hochleiter carried out typical motoscopy examinations. Her series of observation criteria was applied for detecting minor motor disorders of cerebral origin. The method can be used with any age group without limitation. Considering that it is not an objective procedure, it is not suitable for control purposes. Motometry i.e. measuring procedures The examination is carried out through measuring - time, error, range of movement, accuracy. The findings allow comparison and are suitable for standardisation. The disadvantage of the method is that only few motor manifestations can be measured correctly. The final result is built up from rather heterogenous partial performances. The Motor coordination test for children is a pure motometry method. Motometry and motoscopy are often combined e.g. in the Oseretzky scale. Motography It is mainly used for assessing movement in sports. Movement is registered with a path-time diagram which may be subject to detailed analysis later on. Motography allows qualitative analysis. It refelects the process of a movement or a chain of movements. Film recording, light tracking, mechanical, pneumatical and electric procedures are used for registration. Video analysis combined with electromyography and dynamical examination is one of the most important global evaluation method. It is adopted for assessing individuals whose motor control shows functional disorder. Along with the widely used traditional kinematic parameters (step length, step width, walking speed etc. the position, speed, acceleration of discretionary body points as well as articular angles, angular speed and angular acceleration can be determined. Bacsó P. and her team used a specifically extended Dempster model for comparing the motion of 3-4-year-old able-bodied children with the motion of spastic children who were able to walk, had not had surgery and presented good communication skills. 6 GMFM = Gross Motor Function Measure It measures the chronological changes of main motor functions in CP cases. The procedure was designed in order to meet the delicate and functional measurement needs of physiotherapy. Test elements are set in 5 partial areas. The test is not appropriately subtle, unsuitable for detectig small changes:  Lying/rolling: 17 elements  Sitting: 20 elements  Crawling/kneeling: 14 elements  Standing: 13 elements  Walking, running, jumping: 24 elements  The elements are evaluated with 0 to 3 scores (quantitative element)  Photos, videos (qualitative element) 3-dimensional movement analysis in adult hemiplegia Others attempt to measure performance adopting a sensomotoric approach e.g. the Test for condition and motion assessment by K. Lakatos which has undergone several upgradings but has not been standardised yet. It is suitable for assessing 5-12-year-old children whose main diagnosis is immaturity for school, attention disorder, hyperactivity, motor coordination problems (mildly afflicted CP cases), loss of partial abilities or learning disorder rather than intellectual impairment. The test can be applied with individuals presenting one or a combination of the above mentioned symptoms. Kiphard's sensomotoric and psycosocial development chart. Optical perception, hand movements, gross movements, speech, auditory perception and social contacts are assessed up to the age of 4 years. It was tested in our Institute for recording the performances of children with mild CP. The age limit is a disadvantage. GMP diagnostics are used for examining the process of speech perception and speech comprehension in individuals. It may reveal the deficiency of partial ability that hinders successful learning. Only those children can undergo the test who are able to speak. In the assessment process speech perception, visual perception, short-term verbal memory, short-term visual memory, serial perception, vocabulary activation, text comprehension, hand dominance, the knowledge of spatial directions, segmentation, central synthesis, sentence comprehension, sound differentiation, transformation perception, drawing skills, the knowledge of directions on the plane and lateralisation are measured. The method is also suitable for diagnosing CP children. Whatever procedure is chosen for examining motion, we must always take into account that motion cannot be treated in isolation. According to the motion theory of Weizsäcker's school the individual's own movements and perceptions must be seen in interrelation with the environment and as a biological unity. Therefore motodiagnostics must leave the framework of mechanical motion diagnostics. Motor performance is shaped by variables that are interrelated e.g. perception, environment, personality characteristics, motivation, emotionality and should not be seen as independent from each other. Motor experiences have an important role, movement exercise has an increasing impact on performance. These are all interrelated with the quality of concentration and cognitive factors. Few tests are suitable for examining every aspect of the dysfunctioning person's performance. Even less measurement methods are available that have been standardised and can be carried out and evaluated by teachers. The greatest challenge is to examine children with multiple disabilities who 7 present considerable delay from the standard in the cognitive area and are unable to speak. The objective evaluation of their performance is a very complex issue. It has been widely observed that children with multiple disabilities who are familiar with and deeply attached to their conductors show a relapse in both their motor and cognitive performances, respond unfavourably to the assessment situation and need or would need many times as much time to appropriately perform as set for the test. Therefore in our Institute only test without time restrictions are applied. Tests employed in the Pető institute The Brunet-Lesine scale is used up to the age of 3 years for the early detection of disturbances in children's psychomotor development. It is a nondiagnostic test. We have adapted its set of instruments (Everything is bigger.). Before school age the Budapest-Binet test and the Bender-Gestalt test are employed. The Wechsler-Raven test is used for children with normal intellect to slight mental deficiencies. Attention assessment Assessment of the perception of spatial position Edtfeldt test DIFER is a test which has been developed in Hungary. It is a diagnostic development testing and criteria oriented enabling system that supports school entry. Series of evaluation criteria and series measuring partial performances and elements are used with the particular subjects of instruction: − Kindergarten performance tests − Series of criteria to examine drawing activity Aspects and approaches to selective attention, copying and memory can already be assessed at an early age. Conductive observation The need for a systematic, well documented, common approach to the observation of the characteristics of motor disabled children has become vital in recent years. This is essential so that there can be appropriate criteria, suitable for both individual and group assessment, to record the children's condition and guide the direction of development. During conductive education it is the intention to transform the dysfunctions, through the motivation of the child and appropriate plans of action, so that higher levels of development can be achieved which then form the basis for further advancements. As the nature of this educative process is dynamic and permanently varying the evaluation of results cannot be limited to single, occasional actions but rather it is a continuous process. It is during this process that the level of the development is determined and its trend and rhythm are examined and analysed. A further consideration is that development and cognitive activity are interdependent and they cannot be divided, for example, into active and passive phases; diagnostics and therapy are not separated. In this context, therefore, the purpose of assessment is to determine the level of directly observable skills which then form the basis for improvement. In conductive education the need has arisen for a particularly fine measuring system in order to control and describe the changes in 8 development which other approaches are not sensitive enough to monitor. This system can be referred to as conductive observation.In the first place conductive observation requires an adequate analytical framework in order to measure process variables; such observation is multifactorial and continuous and is not limited to a particular phase of development; furthermore, this assessment is an interactive one which, by using immediate feedback, ensures that conditions are optimal for the successful execution of the tasks at any given moment. The development of an operative scale It can be seen, therefore, that conductive observation embraces the whole person and includes all aspects of behaviour such as changing place and position, manipulation, self-care, speech, social behaviour and intelligence. However, the necessity to devise a systematic and comprehensive scale to include all these aspects of behaviour was paramount. The need for this was also underscored as a consequence of two other factors. Firstly, the attempt to computerise the measurement series was not successful as the outcome was too crude and it could not discriminate even the obvious steps of development; secondly, there was the problem of subjectivity in the description of improvements and it was the case that these were not always suitable for a more objective analysis and recording of development. Thus, an unambiguous, easy to manage observation schedule was needed which decreased the subjective element in the observation process and allowed a more systematic approach which could be represented numerically and also presented in graphical form to evaluate change and development. On the basis of operative observation and also observing the continuous changes in symptoms, a general measurement series was devised comprising an analysis of the various relevant human functions. The operative observation schedule which was eventually devised was able to encompass a broad range of responses, was sensitive to the dynamics of the situation and reflected activity more appropriately. The observation series is suitable for evaluating the following dimensions: 1. Position changing 2. Walking 3. Manipulation 4. Self-care 5. Communication 6. Intelligence 7. Social behaviour and contact. More global performances were broken down into smaller units of analysis which allowed a more systematic and quantitative measurements to be made. The context of the observation was also important and in the case of individual analysis changes were monitored and evaluated in the same way for each person; in group situations where individual behaviour is monitored within a more interactive situation again a more systematic approach can be achieved while at the same time occasionally modifying the programme to achieve optimal performance. The results of these observations can be recorded, computerised and represented in graphical form which provides a more dynamic feedback to aid development. It is appropriate at this point to examine the scales in more detail. The analysis of behaviour into the main scales and the subdivision of the areas 9 into sub items for measurement was based on careful observation and previous experience of working with children over many years. The scales themselves are not homogeneous and where existing measures can be employed which are valid and relevant to the situation they are applied accordingly. Position changing There are four scales in the domain of position changing. "Standing up from a stool and sitting down on a stool" is a seven point scale, while "standing up from the floor and sitting down on the floor is an eight point scale. "Sitting up from supine, sitting and lying down to a supine position on the plinth" is also an eight point scale, and this includes the maintenance of a sitting position. "Rolling 360 degrees on the floor commencing from supine" is a seven point scale and relates to rolling in both directions as well as keeping to a straight path for rolling. Walking Two scales in the domain of walking were constructed. The first was a five point scale called "without aids and bare footed” and the second comprised a ten point scale called "with aids" concerned with walking with personal assistance or with the aid of furniture or sticks. Manipulation Six scales were constructed in the domain of manipulation. There are three scales for the right limb and three identical scales for the left limb. "Differentiated movements of the fingers" is a six point scale and relates to how the fingers are moved, whether the thumb and fingers can be opposed, and whether an object can be manipulated with the hand. "Reaching” has seven points on the scale and is concerned with whether a child can only reach out in a variety of directions. “Holding an object" has an eight point scale and records only whether a child can grasp an object if it is placed in the hand, can grasp and hold an object but not release it, or can grasp, hold and release a variety of objects. Self care For the analysis of self care, five scales were devised. "Dressing” is divided into an eight points scale and is concerned with putting on and taking off clothes and footwear. "Shoe laces" has six points and relates to tying and untying laces, while "buttons" also has six points and is concerned with doing up, and undoing buttons. "Feeding" has eight points and covers activities such as participating in feeding, feeding with fingers, feeding with cutlery with and without making a mess. "Toiletting" is an eight point scale and is concerned with indicating the need, as well as the use of potty, toilet and toilet paper. Speech and Communication There are two scales in the domain of speech and communication. The eight point scale describing "speech" ranges from attempting to speak 10 pronouncing only parts of words to sentence construction and the clear pronunciation of words. "Communication" is an eight point scale concerned with verbal and non-verbal communication and whether communication is appropriate for the age of the child. Cognition There are four scales in the domain of cognition. “General knowledge" is a seven point scale concerned with the child's knowledge of its personal circumstances as well as the general environment. The “attention” scale has eight points which describes a range of skills from whether the child can attend to a task for brief moments only, to attending in small groups and attending to a task even when distractions are present. "Learning ability" has six points concerned with whether a child can learn something new only with much repetition, whether something can be learned quickly or whether something recently learned can be used with previously learned material. Social behaviour Four scales were constructed in the domain of social behaviour. "Formal group role" is a seven point scale covering such activities as how a child behaves in a group such as a class situation, whether the child works within a group, whether the group accepts the child and whether the child will take a leading role in the group. "Relationships with adults" has eight points and describes the extent to which the child establishes relationships with adults in general. "Relationships with children in a family situation" is an eight point scale and records whether a child establishes reciprocal social relationships with only a few children in a particular situation, with many children but only of a particular age, or with many children regardless of their age. "Initiating individual contact with other people" is divided into eight points and is concerned with how a child responds to a social advance and initiates a social advance in particular, whether a child is generally indifferent to social advances made by others and accepts contacts when approached by an adult or child or initiates and accepts social advances. For those children with spina bifida there is an additional scale which deals with voiding functions. This comprises eight scales containing details of urological assessments, urological changes, urinary passage infections, ways of voiding, quantity of voluntary voiding, duration of dry periods, frequency of soiling clothes with urine or stools (daily average), defecation indication of needs were observed and measured. The question can now be asked as to the most appropriate use of the scales in the planning and evaluation of development. At the beginning of the conductive education programme for a dysfunctioning child a comprehensive and detailed analysis can be undertaken concerning the level of skills and development; when combined with information obtained from other sources, such as specialist reports, it is possible to provide a comprehensive assessment of the problem and a determination of the appropriate goals to be set for future development. During the application of the programme the series of observations can be made to assist with the analysis of the factors affecting the level of achievement and also facilitate the exploration of how these might inter-relate with other factors. These scales are also useful to assess the advancements in the level of the tasks and to evaluate the effects of changes in the task series on individual improvement. In certain cases when development appears to falter the 11 reasons for this can be explored systematically and changes to the individual or group programme can be made accordingly. Also these scales provide an important longitudinal measure to assess the performance of an individual over a long period of time as well as providing for comparative information to assess similarities and differences between several children in the group as, for instance, when a programme is modified. The piloting of these scales took place initially with the groups at the Birmingham Institute during the collaborative programme and also with two groups at the Petõ Institute. At the present moment the scales are being applied to other groups at the Peto Institute and further work is proceeding with specialised groups, such as those children with spina bifida. As already indicated the advantages of using these scale is that, by securing a more objective analysis of development, they facilitate both longitudinal and cross sectional analysis and ensure better continuity and progression in the application of programmes. There is still further development work which needs to be undertaken in certain areas and, of course, these scales are complementary in that a complete picture of the individual's progress has to include other observations and reports as well. The successful application of the conductive education programme relies on a sound analysis of the individual's needs and this can be best be met through the application of systematic and objective criteria which characterises these rating scales. These scales will add to the range of instruments used to evaluate the development of children with cerebral palsy and will, therefore, contribute to the enhancement of professional practice in the future. Reference MACKAY, G., MCCOOL, S., CHESELDINE, S. and MCCARTNEY, E. (1993): Goal Attainment scaling: a technique for evaluating conductive education. British Journal of Special Education, 20., 4. 12