Variability of creativity judgments

Learning and Individual Differences 18 (2008) 367–371 Contents lists available at ScienceDirect Learning and Individual Differences j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / l i n d i f Variability of creativity judgments Xavier Caroff ⁎, Maud Besançon Université Paris Descartes, Laboratoire de Psychologie et Neurosciences Cognitives (CNRS FRE 2987), France A R T I C L E I N F O Article history: Received 27 March 2007 Received in revised form 22 March 2008 Accepted 1 April 2008 Keywords: Creativity Consensual assessment Judgement Originality Appropriateness A B S T R A C T The Consensual Assessment Technique (CAT), developed by Amabile [Amabile, T.M. (1982). Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43, 997–1013], is frequently used to evaluate the creativity of productions. Judgments obtained with CAT are usually reliable and valid. However, notable individual differences in judgment exist. This empirical study shows that creativity judgments for advertisements vary, depending on (1) the level of two underlying components of creativity — originality and appropriateness, (2) the creative ability of the judges, i.e. variations in their ability to be original, and finally, (3) instructions or training that they received about the topic of creativity assessment. Effects of advertisements' appropriateness and judges' ability to be original on individual differences in creativity judgments are discussed. © 2008 Elsevier Inc. All rights reserved. 1. Introduction Researchers have tended recently to adopt a consensual definition of creativity which emphasizes two criteria: creativity is the capacity to realize a production which is new and, at the same time, adapted to the context (e.g., Amabile, 1996; Lubart, 1994; Sternberg & Lubart, 1995). The novelty of a production is characterized by its original and unexpected nature. Adaptation has been conceived in terms of appropriateness (e.g., Runco & Charles, 1993; Runco, Illies, & Eisenman, 2005), usefulness, value (e.g., Ford, 1996), or resolution with respect to problem constraints (e.g., Besemer & Treffinger, 1981). In spite of its multidimensional nature, creativity has been mostly assessed as a unidimensional construct (Sullivan & Ford, 2005). Since two decades, Amabile's Consensual Assessment Technique (CAT; Amabile, 1982, 1996; Hennessey & Amabile, 1999) has been commonly used to evaluate creativity of productions in different domains (e.g., Hickey, 2001; Baer, Kaufman, & Gentile, 2004; Dollinger, Urban, & James, 2004; Lee, Lee, & Youn, 2005). For Amabile (1982), the procedure requires selecting appropriate judges, which have some experience with the domain of endeavor, and asking them to assess independently productions relative to one another. Judges should not be given any specific criteria for creativity assessment; on the contrary, they are asked to use their own subjective understanding. Despite the obvious success of CAT, further research is needed to ⁎ Corresponding author. Laboratoire de Psychologie et Neurosciences Cognitives, Institut de Psychologie — Université Paris Descartes, 71 avenue Edouard Vaillant, 92774 Boulogne-Billancourt Cedex, France. Fax: +33 1 55 20 59 85. E-mail address: xavier.caroff@univ-paris5.fr (X. Caroff). 1041-6080/$ – see front matter © 2008 Elsevier Inc. All rights reserved. doi:10.1016/j.lindif.2008.04.001 understand what characteristics of the productions, the judgment task, and of the judges themselves, could influence creativity assessment (Amabile, 1996). The main objective of the current research is to investigate the extent to which individual differences in creativity ratings of advertisements are due, first, to judges' differential reliance on two underlying dimensions of creativity — originality and appropriateness — and the integration of these in a global judgment of creativity; second, to characteristics of the judges — e.g. variations in their personal creative ability— and third, to different instructions that judges received about the topic of creativity. 1.1. Information integration in creativity judgments Although the definition of creativity emphasizes, as mentioned above, originality and appropriateness, few studies have varied these dimensions experimentally (e.g., Hammaker, Shafto, & Trabasso, 1975; Hood, 1973; Malgady & Barcher, 1979; Runco & Charles, 1993). To our knowledge, the experiment conducted by Runco and Charles (1993) is the only one to have investigated how both originality and appropriateness contribute to judgments of creativity. These authors created experimentally three sets of productions allegedly obtained in a divergent thinking test. Their results showed that creativity ratings increased either when the proportion of original and appropriate ideas simultaneously increased, or when the proportion of original but inappropriate ideas increased. But, on the contrary, creativity ratings tended to decrease when the proportion of appropriate but common ideas increased. The importance of both underlying dimensions in creativity assessment of real-world productions, such as advertisements, is not 368 X. Caroff, M. Besançon / Learning and Individual Differences 18 (2008) 367–371 well known. To our knowledge, very few studies have investigated this issue, and mostly by means of multivariate analysis of ratings. For example, using subscales of the Creative Product Semantic Scale (CPSS; Besemer, 1998; Besemer & O'Quin, 1986, 1999; O'Quin & Besemer, 1989), White, Shen, and Smith (2002) observed that advertising professionals and members of the general public could agree on advertisement ratings for originality and logic — one of the subscales of the resolution dimension which is conceptually very similar to appropriateness. In another study, in which advertisements were rated by means of semantic differential scales, Sullivan and Ford (2005) showed that originality and value (appropriateness) comprised a unified dimension closely associated with creativity ratings. However, for situations in which both dimensions are experimentally varied, the question remains if, in rating creativity of advertisements, judges could differentially rely on both characteristics. Following Runco and Charles's (1993) recommendation, the present research investigates the mechanism by which judges integrate information concerning originality and appropriateness in their ratings of creativity. More precisely, subjects were asked to rate the creativity of a set of advertisements purposely selected to represent various levels of originality and appropriateness previously scored by experts. We hypothesize that creativity ratings will increase as the originality level of productions increase (Hood, 1973; Runco & Charles, 1993). Although it is assumed that their appropriateness level will also modulate creativity ratings, it is hard to formulate any particular hypothesis concerning this specific effect. We can only speculate that to be judged creative, an advertisement should be sufficiently appropriate (by definition); but not too much, to avoid banality. Moreover, if subjects integrate information on originality and appropriateness to determine their judgments, an interaction between these two factors should be observed. 1.2. Individual differences in judgments related to the creative ability of the judges If the characteristics of productions constitute one source of judgment variability, personal characteristics of the judges themselves could lead to individual differences in creativity ratings. For advertisements, White et al. (2002) noticed that such ratings were significantly linked with certain demographic variables (such as age and gender), and with other descriptive variables (professional experience in advertising and reading newspapers). Moreover, raters' creative ability may lead to individual differences in creativity judgments and could interact with some characteristics of the productions to be judged. For example, Hood (1973) investigated the link between judges' originality, measured in an Unusual Uses Test, and their ratings of originality for productions obtained with an alternative form of the same test. His results showed that creativity ratings of judges who were low in originality were more affected by productions' originality levels compared to judges with moderate originality, or high originality who did not discriminate well the creativity of productions and for whom mean ratings were lower. These results suggest that highly original judges could have developed a very restrictive conception of creativity which leads them to consider that only extremely original productions are creative. The present research will try to replicate this interaction between judges' originality and productions' originality in creativity ratings. However, there is no theoretical reason to suppose that raters' originality will interact with advertisement appropriateness. 1.3. Individual differences in judgments related to instruction The CAT postulates that if the judges possess some expertise in the domain in which the productions are evaluated, they will sponta- neously recognize the creativity in the work (Amabile, 1982, 1996). Amabile (1982, p. 1002) even stated that “if appropriate judges independently agree that a given product is highly creative, then it can and must be accepted as such”. Numerous studies indicate that creativity assessment with the CAT is usually reliable under various experimental conditions (e.g., Amabile, 1996; Baer, 1994; Baer et al., 2004; Hennessey, 1994; Hennessey & Amabile, 1999). Nevertheless, judges' expertise level is an additional source of individual differences for judgments of creativity. First, it is notable that sometimes interjudge agreement can be inferior for expert than for non-expert persons (Dollinger et al., 2004; Hickey, 2001). This result could be interpreted as evidence that expert judges develop their own subjective understanding of creativity based on their experiences within the domain of production. Second, it is not surprising that inter-judge agreement could be rather low for non-expert judges. In an attempt to compensate this lack of agreement, Dollinger and Shaffran (2005) proposed to train non-experts judges to calibrate their ratings before asking them to evaluate creativity of productions. This training consisted simply in giving psychologist judges a set of drawings to illustrate prototypical levels of creativity according to expert (artist) judgments. With this small procedural modification, the authors found that interjudge reliability improved for psychologists, compared to results obtained in a previous study (Dollinger et al., 2004). These findings raise the more general question of the effect of instructions and training on judgments of creativity, even if the CAT procedure normally proscribes this approach (e.g., Amabile, 1982). This third source of variation for creativity judgment can be experimentally studied if we assumed that judges will be more inclined to take both originality and appropriateness into account in conditions that make explicit the relevant dimensions on which to base their creativity judgments. Thus, compared to a situation in which no conceptual framework for creativity was imposed, the respective importance of originality and appropriateness in determining creativity ratings should increase when judges are presented with a normative definition of creativity and explicit criteria for judgments, moreover when they are trained to judge both criteria before assessing creativity. If this hypothesis is correct, the three judgment situations will interact with both originality and appropriateness characteristics of advertisements in creativity ratings. 2. Method Hypothesis concerning the effects of productions' characteristics, judges creative ability and instruction will be tested by means of mixed factorial design, with three between subjects factors (gender, the two levels of raters' originality and the three experimental situations) and two repeated factors (the three levels of advertisement originality and the three levels of advertisement appropriateness). Gender was introduced in the design to check if this variable influences creativity judgment. 2.1. Subjects Participants were 95 volunteer advanced undergraduates or recent graduate students at the University of Paris – France — (Mean age = 27.7 years; SD = 4.9). Approximately one third of them deliberately decided to attend a course on creativity. 2.2. Material 2.2.1. Test of divergent thinking Participants' creative ability was assessed with a divergent thinking task from the French version of the Torrance Tests of Creative Thinking (TTCT; Torrance, 1976): the unusual uses of a cardboard box task. In this task, subjects were asked to find as many different uses as X. Caroff, M. Besançon / Learning and Individual Differences 18 (2008) 367–371 possible for one or several boxes. They had 10 min to complete this task. 2.2.2. Construction of a set of advertisements to be evaluated Automobile advertisements were chosen for this study because of participants' familiarity with this kind of advertisement due to its' prevalence in the regular press. To construct the experimental set of advertisements, we followed the methodological guidelines proposed by Anderson (1982a,b, 1991, 1996) to investigate information integration in judgment. This set of advertisements has been constructed from factorial design, in which the stimulus variables (originality and appropriateness) are under experimental control. For each of the 190 advertisements collected, advertising agency professionals (15 females and 9 males; aged from 25 to 50 years old) served as experts to rate their originality, appropriateness, and creativity using scales ranging from 0 (a lack of the dimension) to 7 (a high level for that dimension). All advertisements were rated by all judges. Then, originality, appropriateness, and creativity ratings were averaged over the judges. Selection of the experimental set of advertisements was based on mean ratings distributions obtained in the previous phase. Two kinds of advertisements were selected. For 9 advertisements, originality and appropriateness were judged as simultaneously present, and both characteristics varied respectively at a low, medium or high level. For the other advertisements, one dimension was evaluated as absent but the other one varied; i.e., 3 advertisements were considered inappropriate but with originality which was respectively low, medium and high, and 3 advertisements were considered common but with appropriateness which was respectively low, medium and high. For this set of 15 advertisements, interjudge agreement was acceptable; respectively, α = .92 for originality, α = .76 for appropriateness, and α = .94 for creativity. 2.3. Procedure The experiment began with the divergent thinking task. At the end of this task, each participant obtained an originality score corresponding to the statistical rarity of his or her different ideas. Based on the median score (Median = 14, Mean = 16.6 and SD = 11.4), two subgroups were created: “low originality participants” who obtained originality scores lower than the median, and “high originality participants” who obtained scores higher than the median. In the second phase, participants completed the experimental task, which consisted of estimating the creativity of the 15 selected advertisements, respectively in one of three experimental situations. In situation 1, participants were required to evaluate creativity without any particular instructions, which supposed that they would rely on their own implicit conception of creativity. In contrast, in situation 2, participants were given explicit criteria for judging creativity before they made judgments; and in situation 3, they were trained to evaluate creativity on its relevant dimensions just before the experimental task. Participants were randomly assigned to situation 1 or 2; only participants attending a regular course on creativity were tested in situation 3. Each of the three experimental groups was composed of 12 males and 12 females; one half had an originality score higher than the median, and the other half had an originality score lower than the median. Participants with scores that were very close to the median were not taken into account. The three groups did not differ significantly on mean originality level, or on mean age; respectively, F(2,69) = .80 NS and F(2,69) = 1.43, p N .05. In the three situations, participants were instructed that, after a warm up, the experimental task would consist of evaluating creativity for 15 different advertisements. During the warm up, advertisements were presented in a fixed randomized order, but participants were allowed to compare them. Then, the same advertisements were presented four times in the same order across participants. In each 369 judgment session, participants were asked to evaluate the creativity of each advertisement on a 7-point rating scale ranging from 1 (less creative) to 7 (very creative). The dependent variable was the mean judgment averaged, for each advertisement, over the four sessions. 3. Results Only the 9 advertisements corresponding to each combination of three levels of originality (low, medium and high) with three levels of appropriateness were taken into account. Participants' creativity judgments were analyzed by means of mixed factorial ANOVA, with three between subjects factors (gender, the two levels of raters' originality and the three experimental situations) and two repeated factors (the three levels of advertisement originality and the three levels of advertisement appropriateness). The results presented in Table 1 show that there was no overall difference between male and female judges (F(1,60) = .05 NS), and no significant interaction between gender and the others experimental factors. There was no significant creativity judgment difference between the two groups of raters differing on originality levels; F(1,60) = .04 NS. The only significant main effects concerned advertisements' characteristics. Mean creativity ratings were greater for the higher originality level (F(2,120) = 243.61, p b .001, partial η2 = .802), and the higher appropriateness level (F(2,120) = 3.80, p b .05, partial η2 = .060). The interaction between advertisements originality level and appropriateness level was significant; F(4,240) = 9.97, p b .001, partial η2 = .143. Fig. 1 suggests that participants integrated both relevant dimensions in a particular way. The crossing of the curves indicates that, when assessing creativity, participants were more influenced by advertisements' originality level when appropriateness was low than when it was moderate or high. But this interaction graph did not correspond to any algebraic integration model postulated by Anderson (1982a,b, 1996): it was neither composed of parallel curves nor Table 1 Results of the mixed factorial ANOVA Sources Effect Error df MS df MS 60 60 60 120 120 60 60 60 120 120 120 120 120 120 240 60 120 120 120 120 120 120 240 240 240 120 120 240 240 240 240 4.406 .053 .819 4.406 .037 .849 4.406 .408 .667 2.099 243.610 b.001 .996 3.796 .025 4.406 3.806 .056 4.406 1.303 .279 4.406 1.409 .252 2.099 .653 .523 2.099 3.378 .037 2.099 .747 .562 .996 .133 .876 .996 .194 .824 .996 2.652 .036 1.145 9.971 b.001 4.406 .762 .471 2.099 .072 .930 2.099 .254 .907 2.099 1.057 .381 .996 .999 .371 .996 .409 .802 .996 1.803 .133 1.145 .913 .457 1.145 .493 .741 1.145 1.012 .427 2.099 1.770 .139 .996 .971 .426 1.145 1.271 .282 1.145 .883 .531 1.145 1.022 .420 1.145 .680 .709 Gender (G) 1 Raters Originality (RO) 1 Situations (S) 2 Advert. Originality (AO) 2 Advert. Appropriateness (AA) 2 G × RO 1 G×S 2 RO × S 2 G × AO 2 RO × AO 2 S × AO 4 G × AA 2 RO × AA 2 S × AA 4 AO × AA 4 G × RO × S 2 G × RO × AO 2 G × S × AO 4 RO × S × AO 4 G × RO × AA 2 G × S × AA 4 RO × S × AA 4 G × AO × AA 4 RO × AO × AA 4 S × AO × AA 8 G × RO × S × AO 4 G × RO × S × AA 4 G × RO × AO × AA 4 G × S × AO × AA 8 RO × S × AO × AA 8 G × RO × S × AO × AA 8 .232 .162 1.797 511.337 3.781 16.772 5.742 6.208 1.370 7.091 1.569 .132 .193 2.641 11.420 3.356 .151 .533 2.219 .995 .408 1.795 1.046 .564 1.159 3.715 .967 1.455 1.012 1.170 .779 F Sig. Partial η2 .001 .001 .013 .802 .060 .060 .042 .045 .011 .053 .024 .002 .003 .081 .143 .025 .001 .008 .034 .016 .013 .057 .015 .008 .033 .056 .031 .021 .029 .033 .022 370 X. Caroff, M. Besançon / Learning and Individual Differences 18 (2008) 367–371 Fig. 1. Mean creativity judgments for advertisements of different levels of originality and different levels of appropriateness. showed a characteristic linear fan effect. For this reason, we could not investigate further the integration mechanism, by means of creativity ratings obtained for the 6 supplementary advertisements in which one characteristic was judged as absent and the other varied. As suggested by Hood (1973), the interaction between judges ability to be original and advertisement originality was significant; F(2,120) = 3.38, p b .05, partial η2 = .053. But surprisingly, the result was opposite from the one predicted: the effect of advertisement originality level on creativity ratings was greater for judges high in originality than for those low in originality (Fig. 2). The interaction between judges' originality and advertisement appropriateness was not significant; F(2,120) = .19 NS. We checked that for both groups of judges dichotomized by their originality level, participants were more influenced by advertisements originality level when appropriateness was low than when it was moderate or high; respectively, F(4, 120) = 5.54, p b .001, partial η2 = .156 for judges low in originality and F(4, 120) = 4.89, p b .01, partial η2 = .140 for judges high in originality. Contrary to our prediction, the effect of advertisements' originality levels on creativity judgments did not vary between the three situations of evaluation; F(4,120) = .75 NS. However, there was a significant interaction between advertisements' appropriateness levels and situations; F(4,120) = 2.65, p b .05, partial η2 = .081. This overall interaction (Fig. 3) was further explored by means of trend analysis. Results show that when participants were required to evaluate creativity without any particular instruction (situation 1), advertisement creativity ratings increased when appropriateness increased. This linear trend was significant; F(1,60) = 2.65, p b .05, partial η2 = .042. Fig. 2. Mean creativity judgments for advertisements of different levels of originality and judges ability to be original. Fig. 3. Mean creativity judgments for advertisements of different levels of appropriateness and for different judgment situations. But, for this same group, the quadratic term, which test the hypothesis of a curvilinear relationship, was not significant; F(1,60) = .39 NS. When subjects were given explicit criteria for judging creativity (situation 2), neither the linear trend nor the quadratic one were significant; respectively, F(1,60) = .02 NS and F(1,60) = 1.13, p N .05. When subjects were trained to evaluate creativity on its relevant dimensions (situation 3), both the linear and quadratic terms was significant; respectively, F(1,60) = 7.92, p b .01, partial η2 = .117 and F(1,60) = 4.93, p b .05, partial η2 = .076. 4. Discussion and conclusion The first objective of this experiment was to understand how originality and appropriateness are bound to creativity (e.g., Runco et al., 2005). In a multidimensional approach to creativity, originality is a necessary condition, but it is not sufficient for a production to be creative; an original production must be also adapted to the context (e.g., Amabile, 1996; Hood, 1973; Lubart, 1994; Runco & Charles, 1993; Sternberg & Lubart, 1995). Based on past research (e.g., Hood, 1973; Runco & Charles,1993), it is not surprising to notice, in the present study, that creativity ratings are more favorable when the level of originality of advertisements increases. However, results concerning the effect of the appropriateness dimension are somewhat more complex, but they could shed some new light on the integration of both underlying characteristics in creativity judgment. We noticed first that production's appropriateness could modulate the assessment of creativity based on the level of originality. For the whole sample, the effect of advertisements' originality on creativity ratings was more important when appropriateness was low than when it was moderate or high; in other words, appropriateness would decrease the impact of originality level on creativity ratings. This same interaction was also observed for both subgroups of participants dichotomized by their originality level. But results concerning the effect of the experimental situations allows further explanation of the creativity judgment mechanism. Experimentally varying the assessment design does not modify the effect of advertisements' originality level on creativity ratings. It is possible that judges spontaneously estimate the creativity of advertisements based on this dimension, and that this relationship can not be modified by any instruction or training. But the results are quite different for the second dimension. When participants were asked to judge creativity of advertisements without imposing a particular conceptual framework, creativity was linearly related to appropriateness. In comparison, when participants were informed how to judge it, creativity were not bound to appropriateness any more; and when subjects were trained to judge creativity based on its underlying dimensions, the most creative advertisements were those that were X. Caroff, M. Besançon / Learning and Individual Differences 18 (2008) 367–371 neither extremely high nor low on appropriateness. This last result suggests that a certain appropriateness threshold could be involved in creativity judgment; when this threshold is exceeded, productions would be considered as uncreative independently of their originality level. This interpretation should be tested in a further study. Such a study could also address the question of whether the present results generalize across domains or are domain specific. It is possible that advertising, and other artistic sectors as well, emphasize originality more than other domains such as real-world business problem solving, in which appropriateness may be more influential than originality when assessing creativity. A second objective of the research was to test if creative ability of the judges could modulate creativity ratings. The present findings show that the effect of advertisements' originality on creativity ratings was stronger for judges high in originality than for those low in originality. These results do not corroborate the hypothesis, based on Hood's (1973) results, that highly original judges could have developed a very restrictive conception of creativity, one which leads them to consider that only extremely original productions are creative. They nevertheless confirm that highly original judges discriminate much more advertisements' creativity based on their originality level. To further explore the link between judges' creative ability and their creativity judgments, it could be interesting also to assess rater's ability to produce appropriate products and to study its interaction respectively with products' appropriateness and originality in creativity ratings. In conclusion, this research emphasizes the importance of studying the mechanisms of information integration in creativity judgments and, in particular, the effect of the level of appropriateness on creativity rating. It also demonstrates that some variability characterizes creativity judgments, which justifies studying more systematically individual differences between raters (Hennessey, 1994). References Amabile, T. M. (1982). Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43, 997−1013. Amabile, T. M. (1996). Creativity in context. Boulder, Colorado: Westview Press. Anderson, N. H. (1982). Foundation of information integration theory. New York: Academic press. Anderson, N. H. (1982). Methods of information integration theory. New York: Academic press. Anderson, N. H. (1991). Contributions to information integration theory (3 volumes). Mahwah, New Jersey: Lawrence Erlbaum Associates. Anderson, N. H. (1996). A functional theory of cognition. Mahwah, New Jersey: Lawrence Erlbaum Associates. Baer, J. (1994). Performance assessments of creativity: Do they have long-term stability? Rooper Review, 17, 7−10. 371 Baer, J., Kaufman, J., & Gentile, C. (2004). Extension of the consensual assessment technique to nonparallel creative products. Creativity research journal, 16, 113−117. Besemer, S. P. (1998). Creative product analysis matrix: testing the model structure and a comparison among products—three novel chairs. Creativity Research Journal, 11, 333−346. Besemer, S. P., & O'Quin, K. (1986). Analyzing creative products: Refinement and test of a judging instrument. Journal of Creative Behavior, 20, 115−126. Besemer, S. P., & O'Quin, K. (1999). Confirming the three-factor creative product analysis matrix model in an American sample. Creativity Research Journal, 12, 287−296. Besemer, S. P., & Treffinger, D. J. (1981). Analysis of creative products: Review and synthesis. Journal of Creative Behavior, 15, 158−178. Dollinger, S. J., & Shaffran, M. (2005). Note on consensual assessment technique in creativity research. Perceptual and Motor Skills, 100, 592−598. Dollinger, S. J., Urban, K., & James, T. A. (2004). Creativity and openness: Further validation of two creative product measures. Creativity Research Journal, 16, 35−47. Ford, C. M. (1996). A Theory of Individual Creative Action in Multiple Social Domains. Academy of Management Review, 21, 1112−1142. Hammaker, M. K., Shafto, M., & Trabasso, T. (1975). Judging creativity : A method for assessing how and by what criteria it is done. Journal of Educational Psychology, 67, 478−483. Hennessey, B. A. (1994). The Consensual Assessment Technique: An examination of the relationship between ratings of process and product creativity. Creativity Research Journal, 7, 193−208. Hennessey, B. A., & Amabile, T. M. (1999). Consensual assessment. In M. A. Runco, & S. R. Pritzker (Eds.), Encyclopedia of creativity (pp. 347−359). San Diego: California: Academic press. Hickey, M. (2001). An application of Amabile's consensual assessment technique for rating the creativity of children's musical compositions. Journal of Research in Music Education, 49, 234−244. Hood, R. W. (1973). Rater originality and the interpersonal assessment of levels of originality. Sociometry, 36, 80−88. Lee, S., Lee, J., & Youn, C. -Y. (2005). A variation of CAT for measuring creativity in business products. The Korean Journal of Thinking & Problem Solving, 15, 143−153. Lubart, T. I. (1994). Creativity. In R. J. Sternberg (Ed.), Thinking and Problem Solving (pp. 289−332). New York: Academic Press. Malgady, R. G., & Barcher, P. R. (1979). Some information-processing models of creative writing. Journal of Educational Psychology, 71(5), 717−725. O'Quin, K., & Besemer, S. P. (1989). The development, reliability, and validity of the revised Creative Product Semantic Scale. Creativity Research Journal, 2, 267−278. Runco, M. A., & Charles, R. E. (1993). Judgments of originality and appropriateness as predictors of creativity. Personality and individual differences, 15, 537−546. Runco, M. A., Illies, J. J., & Eisenman, R. (2005). Creativity, originality, and appropriateness: What do explicit instructions tell us about their relationships. Journal of Creative Behavior, 39, 137−148. Sternberg, R. J., & Lubart, T. I. (1995). Defying the crowd: Cultivating creativity in a culture of conformity. New York: Free Press. Sullivan, D. M., & Ford, C. M. (2005). The relationship between novelty and value in the assessment of organizational creativity. Korean Journal of Thinking and Problem Solving, 15, 117−131. Torrance, E. P. (1976). Tests de pensée créative. Paris: Editions du centre de Psychologie Appliquée. White, A., Shen, F., & Smith, B. L. (2002). Judging advertising creativity using the creative product semantic scale. Journal of Creative Behavior, 36, 241−253.