An Integrated Approach to Automatic Synonym Detection in Turkish Corpus Tuğba Yıldız1 , Savaş Yıldırım1 , and Banu Diri2 1 Department of Computer Engineering, Istanbul Bilgi University, Eski Silahtarağa Elektrik Santrali, Kazım Karabekir Cad. No: 2/13, 34060 Eyüp, Istanbul, Turkey {tdalyan,savasy}@bilgi.edu.tr 2 Department of Computer Engineering, Yildiz Technical University, Davutpasa, 34349 Istanbul, Turkey banu@ce.yildiz.edu.tr Abstract. In this study, we designed a model to determine synonymy. Our main assumption is that synonym pairs show similar semantic and dependency relation by the definition. They share same meronym/holonym and hypernym/hyponym relations. Contrary to synonymy, hypernymy and meronymy relations can probably be acquired by applying lexico-syntactic patterns to a big corpus. Such acquisition might be utilized and ease detection of synonymy. Likewise, we utilized some particular dependency relations such as object/subject of a verb, etc. Machine learning algorithms were applied on all these acquired features. The first aim is to find out which dependency and semantic features are the most informative and contribute most to the model. Performance of each feature is individually evaluated with cross validation. The model that combines all features shows promising results and successfully detects synonymy relation. The main contribution of the study is to integrate both semantic and dependency relation within distributional aspect. Second contribution is considered as being first major attempt for Turkish synonym identification based on corpus-driven approach. Keywords: Synonym, relations. 1 near-synonym, pattern-based, dependency Introduction As one of the most well-known semantic relations, synonymy has been subject to numerous studies. By the definition, synonyms are words with identical or similar meanings. The discovery of synonym relations may help to address various Natural Language Processing (NLP) applications, such as information retrieval and question answering [1–3], automatic thesaurus construction [4,5], automatic text summarization [6], language generation [7], English lexical substitution task [8], lexical entailment acquisition [9]. A. Przepiórkowski and M. Ogrodniczuk (Eds.): PolTAL 2014, LNAI 8686, pp. 116–127, 2014. c Springer International Publishing Switzerland 2014  An Integrated Approach to Automatic Synonym Detection 117 Various methods have been proposed for automatic synonym acquisition. Recent studies were generally based on distributional similarity and pattern-based approach. General idea behind distributional similarity is to capture the semantically related words. Distributional similarity of words sharing a large number of contexts could be informative [10]. Pattern-based approach is the most precise acquisition methodology earlier applied by Hearst [11] and relies on lexicosyntactic patterns (LSPs). On the other hand, these methodologies themselves can be ambiguous and insufficient. Distributional similarity approach can cover other semantically related words and might not distinguish between synonyms and other relations. For example, list of top-10 distributionally similar words for orange is: yellow, lemon, peach, pink, lime, purple, tomato, onion, mango, lavender [12]. In addition, the pattern-based approach tends to capture hyponymy and meronymy relations as well, whereas it is apparently incompatible for synonyms detection. Thus, pattern-based approach or external features such as grammatical relations can be integrated into distributional similarity approach for identifying synonyms by narrowing distributional context. Although some studies have showed that classical distributional methods always have a higher recall than pattern-based techniques in this area [13], integrating two or more approaches were reported that system performance was improved [9, 13–15]. In this study, overall objective is to determine synonym nouns in a Turkish Corpus by relying on distributional similarity that is based on syntactic features (obtained by dependency relations) and semantic features obtained by syntactic patterns and LSPs respectively. The features of the proposed model consist of co-occurrence statistics, four semantic relations and ten syntactic dependency relations where a pair of words are represented with fifteen different features and a target class (SYN/NONSYN). One of the main contributions of the study is that the system first obtains acquirable semantic relations such as hypernymy, meronymy from corpus by LSPs to extract subtle relations such as synonymy. The second contribution of the study is considered to be the first major attempt for Turkish synonym identification based on corpus-driven approach. 2 Related Works A variety of methods have been proposed to automatically or semi-automatically detect synonyms from text source, dictionaries, wikipedia, search engines. Among them, the most popular methods are based on distributional hypothesis [10] which states that semantically similar words share similar contexts. The process of this approach was as follows: co-occurrence, syntactic information, grammatical relations of the words surrounding the target word are extracted as a first step. Afterwards target word is represented as a vector with these contextual features. At the second step, the semantic similarity of two terms is evaluated by applying a similarity measure between their vectors. The words can be ranked by their both semantic and syntactic similarity. Finally, top candidates are selected as the most similar words from ranked list. 118 T. Yıldız, S. Yıldırım, and B. Diri There have been various studies [4,16,17] which used distributional similarity to the automatic extraction of semantically related words from large corpora. Distributional approaches have been applied into monolingual [4,18,19], monolingual parallel [20, 21], bilingual corpora [20, 22], multilingual parallel corpora [23] and monolingual dictionary [24,25], bilingual dictionaries [12]. Some of the studies [26–29] were relied on multiple-choice synonym questions such as SAT analogy questions, TOEFL synonym questions, ESL synonym-antonym questions. These studies fell into different types with respect to weighting scheme, similarity measurement, grammatical relations, etc. However most of these studies are not individually sufficient for synonyms. Because this approach also covers near-synonyms and does not distinguish between synonyms and other relations, hence, recent studies used different strategies: integrating two independent approaches such as distributional similarity and pattern-based approach, utilizing external features or ensemble method with combining the results to obtain more accuracy. Mirkin [9] integrated pattern-based and distributional similarity methods to acquire lexical entailment. Firstly, they extracted candidate entailment pairs for the input term by these methods. Another study [31] emphasized that selection of useful contextual information was important for the performance of synonym acquisition. Therefore, they extracted three kinds of word relationships from corpora: dependency, sentence co-occurrence, and proximity. They utilized vector space model(VSM), tf-idf weighting scheme and cosine similarity. Dependency and proximity performed relatively well by themselves. The combination performance of all contextual information gave the best result. Other study of Hagiwara (2008) [14] proposed a synonym extraction method by using supervised learning based on distributional and/or pattern-based features. They constructed five synonym classifiers: Distributional Similarity (DSIM), Distributional Features (DFEAT), Pattern-based Features (PAT), Distributional Similarity and Pattern-based Features (DSIMPAT) and Distributional and Pattern-based Features (DFEAT-PAT). Other study [15] used three vector-based models to detect semantically related nouns in Dutch. They analyzed the impact of three linguistic properties of the nouns. They compared results from a dependency-based model with context feature with 1st and 2nd order bag-of-words model. They examined the effect of the nouns’ frequency, semantic specificity and semantic class. In one of the recent studies, [30], graded relevance ranking problem was applied to discover and rank the quality of the target term’s potential synonyms. The model used supervised learning method; linear regression with three contextual features and one string similarity feature. The method was compared to two different methods [14, 27]. As a result, proposed methods outperformed the existing ones. In Turkish, recent studies on synonym relations are based on dictionary definition TDK1 and Wiktionary2 [32, 34, 35]. Within this framework, the main 1 2 Türk Dil Kurumu (The Turkish Language Association) Vikisözlük: Özgür Sözlük An Integrated Approach to Automatic Synonym Detection 119 contribution of our work is its corpus-driven characteristics and it relies on both dependency and semantic relations. 3 3.1 Methodology Data The methodology employed here is to identify the synonym pairs from a large Turkish corpus of 500M tokens. A Turkish morphological parser, which is based on a two-level morphology [33], was used. A good way to evaluate system performance is to compare the results to a gold standard. First, as gold standard, human judgments about the similarity of pairs of word are used. We manually and randomly selected 200 synonym pairs and 200 non-synonym pairs to build a training data set. Secondly, non-synonym pairs are especially selected from associated (relevant) pairs such as tree-leaf, student-school, computer-game, etc. Otherwise, selection of irrelevant pairs for negative examples can lead to false induction. The model is considered accurate if it can distinguish correct synonym pairs from relevant or strongly associated ones. 3.2 Similarity Measurement and Representation Synonym pairs were gathered on the basis of co-occurrence statistics, semantic and grammatical relations. In order to compute the similarity between concepts and eliminate incorrect candidates, we used the cosine similarity measurement based on the word space model which is a representational Vector Space. In this study, words space was derived from a specialized context obtained by dependency patterns. Vector representation of words gives strong distributional indication for synonymy detection. Similarity measurement between two vectors sometimes needs term weighting. Weighting scheme for context vectors might be normalization, pmi, dice, jaccard or raw frequency. The scheme can vary depending on the problem, therefore, it must be tested on the domain. Since we do not observe any significant improvements between the weighting formula, raw frequency is used for context vectors. 3.3 Features Our methodology relies on the assumption that synonym pairs mostly show similar dependency and semantic characteristics in corpus. They share the same meronym/holonym relations, same particular list of governing verbs, adjective modification profile and so on, by definition. Even though it is no-use applying LSPs to extract synonymy, acquisition of other semantic relations such as meronymy could be easily done by simple string matching utilization and morphological analysis. By means of the acquisitions, the proposed model can determine if 120 T. Yıldız, S. Yıldırım, and B. Diri a given word pair is synonym or not. All attributes are based on relation measurements between pairs. For each synonym pair, 15 different features are extracted from different models: co-occurrence, semantic relations based on LSPs and grammatical relations based on syntactic patterns and head-modifier relation. Feature 1: Co-occurrence. The first feature is gathered statistics about the co-occurrence of word pairs with a broad context (window size is equal to 8 from left and right) from corpora. Contrary to hypernymy and meronymy relation, it is seems impossible to directly extract synonym pairs by applying LSPs to a big corpus. Synonym pairs are not likely to co-occur together in same context and specific patterns at the same time. Therefore, first-order distributional similarity does not work for synonyms. At least, second order representation is needed. Simple co-occurrence measure might not be used for synonymy but non-synonymy. Their co-occurrence could be lower than relevant pairs. We experimentally selected dice metric to measure co-occurring feature. It is computed by roughly dividing the number of co-occurrences by summation of marginal frequencies of words. Features 2/3: Meronym/Holonym. Detection of meronymy/holonymy is used to detect synonymy relation. After applying LSPs, some elimination assumption and measurement metrics such as chi or pmi to acquire meronym/holonym relation, we obtain a big matrix in which rows depict whole candidates, columns depict part candidates and cells represent the possibility of that corresponding whole and part are in meronymy relation. To measure the similarity of meronymy profile of two given words, cosine function is applied on two rows indexed by two given words. Applying cosine function on two columns gives the similarity of holonym profile. For the relation, three different clusters of LSPs are analyzed in Turkish corpus; General (GP), Dictionary-based (TDK-P) and Bootstrapped patterns (BP) [37, 38]. First cluster is based on widely used general patterns. These patterns are collected from some pioneer studies and analyzed in Turkish. Second one is based on dictionary patterns that are extracted from TDK and Wiktionary. We adopted both types of patterns to extract the sentences that include part-whole relations from a Turkish corpus. Third cluster is based on bootstrapping. Some manually prepared seeds were used to induce and score LPSs. Based on reliability scores, we decided to filter out some generated patterns and finally obtained six different significant patterns. Once all three pattern clusters have been evaluated, third cluster of patterns (BP) showed significant performance. Table 1 shows six example patterns in third cluster(BP). All of the experiments in the studies, [37, 38], indicate that proposed methods have good indicative capacity. Features 4/5: Hyponym/Hypernym. Same procedure in meronymy acquisition holds true for hypernymy and hyponymy relation. One relation matrix is built for hypernymy/hyponymy by applying LSPs and same procedure is carried out. The most important LSPs for Turkish [36] are as follows: An Integrated Approach to Automatic Synonym Detection 1. 2. 3. 4. 121 “NPs gibi CLASS” (CLASS such as NPs), “NPs ve diğer CLASS” (NPs and other CLASS) “CLASS lArdAn NPs” (NPs from CLASS) “NPs ve benzeri CLASS” (NPs and similar CLASS) First pattern gives strong indication of is-a hierarchy. Given the syntactic patterns above, the algorithm extracts the candidate list of hyponyms for a hypernym. The method had a good capacity to get higher precision, such as 72.5% [36]. Table 1. Bootstrapped Patterns and Examples Patterns NPy+gen NPx+pos NPy+nom NPx+pos NPy+Gen (N-ADJ) NPx+Pos NPy of one-of NPxs NPx whose NPy NPxs with NPy Examples door of the house / evin kapısı house door / ev kapısı back garden gate of the house / evin arka bahçe kapısı the door of one of the houses / evlerden birinin kapısı the house whose door is locked / kapısı kilitli olan ev the house with garden and pool / bahçeli ve havuzlu ev Features 6–15. The dependency relations are obtained by syntactic patterns (or regular expression). For example, for auto and car pair, possible governing verbs bearing direct-object relations might be drive, design, produce, use, etc. The dimension of word-space model of direct-object syntactic relation consists of verbs and the cells indicate the number of times the selected noun is governed by corresponding verb. The more they are governed by the similar verb profile, the more likely they are synonyms. Likewise, the process is naturally applicable for other syntactic features. The more they are modified by same adjectives, the more likely they are synonym. Although 36 different patterns were extracted, eight were eliminated because of the poor results. Then we grouped them according to their syntactic structures. Representation of groups, number of patterns and examples in English/Turkish are given in Table 2. The essential problem we face in the experiments is the lack of features of some words. Particularly, rare words cannot be represented due to lack of corpus evidence. Even in the corpus that contains about 500M words, all instances of use of Turkish language may not be present. Thus, those instances in train data that do not occur in any of dependency and semantic relations are eliminated. Especially the pairs including low frequent word cannot be represented and evaluated by means of the methodology as the number of missing values in many features increases. Out of 400 instances, about 40–50 are discarded from training data due to insufficiency. 3.4 Binary Classification for Synonym Finally, train data turns out to contain balanced number of negative and positive examples with fifteen attributes. All the cells contain real value between 0–1. We 122 T. Yıldız, S. Yıldırım, and B. Diri know and accept that all features but co-occurrence feature have positive linear relationship with target class. Therefore, the data is considered to exhibit linear dependency. As a consequence of linearity, linear regression is an excellent and simple approach for such a classification. It has been widely used in statistical applications. The most suitable algorithm is logistic regression which can easily be used for binary classification in the domains with numeric attributes and nominal target class. Contrary to the linear regression, it builds a linear model based on a transformed target variable. Another model would be perceptron. If the data can be separated perfectly into two groups using a threshold value or a function, it is said to be linearly separable. The perceptron learning rule is capable of finding a separating hyperplane on linearly separable data. However, our problem looks more suitable for logistic regression (transformed linear regression) than perceptron. Table 2. Dependency Features Features Dependency relation direct object of verb G1 G2 G3 G4 G5 G6 G7 G8 G9 G10 4 # of Patterns Examples 13 I drive a car araba sürüyorum subject of verb 3 waiting car bekleyen araba direct object/subject of verb 3 modified by adjective+(with/without) 2 car with gasoline benzinli araba modified by inf 1 swimming pool yüzme havuzu modified by noun 1 toy car oyuncak araba modified by adjective 1 red car kırmızı araba modified by acronym locations 1 the cars in ABD ABD’deki arabalar modified by proper noun locations 1 the cars in Istanbul Istanbul’daki arabalar modified by locations 2 the car at parking lot otoparktaki araba Results and Discussion To evaluate the impact of semantic and dependency relations in finding synonyms, first, we look at their individual performances in terms of cross-validation. Picking up each feature one by one with target class, we evaluated the performance of logistic regression on the projected data. As long as the averaged fmeasured score of the corresponding feature is higher than 50%, it is considered a useful feature otherwise, independent feature. An Integrated Approach to Automatic Synonym Detection 123 The first aim is to find out which feature is the most informative for detecting synonymy and contributes most to the overall success of the model. When evaluating the result as shown in Table 3, the semantic features are notably better than syntactic dependency models in finding true synonyms. They are called to be good indicators. Table 3. F-Measure of Semantic Relations (SRs) Features F-Measure co-occurrence hyponym hypernym meronym holonym 62.5 60.5 60 68.7 73.7 Among semantic relations, the most powerful attributes are meronymy and holonymy features with f-measure of 68.7% and 73.7%, respectively. The possible reason for the success seems to be the sufficient number of cases matched by lexico-syntactic and syntactic pattern from which semantic and syntactic features are constructed. For example, the model utilizing meronymy relations has a good production capacity and success. The Table 4 shows that meronymyholonymy matrix has the size of 17K x 18K. The total number of instance is 1.7M. Average number of instances for each meronym is 102 and for each holonym is 96. They also show good performance. The averaged number of instances for hypernymy and hyponymy are 50 and 8, respectively. As a result of insufficient data volume, hypernymy/hyponymy semantic relation is relatively weaker than meronymy. Table 4. Statistics for features : Mero:Meronym, Hypo: Hyponym, AVG cpr: average case per row, AVG cpc: average case per column #ofrow #ofcol #ofcases AVG cpr AVG cpc G1 16K 1.7K 3.3M 206 2010 G2 G3 G4 G5 G6 G7 G8 G9 G10 Mero Hypo 18K 10K 13K 7K 13K 20K 6K 1.7K 13K 17K 4.3K 1.7K 1.4K 5K 1.6K 13K 5.6K 1.6K 0.2K 5K 18K 29K 3M 0.5M 1.6M 1M 5.3M 12M 0.1M 0.01M 1M 1.7M 0.2M 164 47 128 140 391 590 23 7 75 102 50 1783 341 319 621 405 2106 86 51 195 96 8 Among dependency relations, G1, G4 and G7 have better performance as shown in the Table 5. Also their production capacities are sufficient as well. The poorest groups, G8 and G9, have low production capacity and their performances are worse. As a consequence of the poor results, they are called independent and useless variables. Co-occurrence feature has negative linear relation with target class and its individual performance is 62.5%. It is acceptable as a useful feature. The successful features are linearly dependent on target class. The most suitable machine learning algorithm is the logistic regression. After aggregating all useful features which have better than the individual performances, the machine learning process was carried out and evaluated. The achievement of aggregated 124 T. Yıldız, S. Yıldırım, and B. Diri Table 5. F-measure of Dependency Relations Features G1 G2 G3 G4 G5 G6 G7 G8 G9 G10 F-Measure 64.7 58 60.5 65 61.6 58.8 63 49.4 48.3 62.6 model was evaluated in terms of cross validation. On the aggregated data where all useful features are considered, the performance of logistic regression is fmeasure of 80.3% and that of voted perceptron is 74%. The achieved score is better than the individual performance of each feature. The number of useful features is obviously the main factor to get higher scores. The proposed model utilizes only a huge corpus and morphological analyzer and it receives an acceptable score. Moreover, other useful resources might be integrated into the model to obtain better result. Dictionary definitions, WordNet, and other useful resources could be used and evaluated in future work. 5 Conclusion In this study, synonym pairs were determined on the basis of co-occurrence statistics, semantic and dependency relations within distributional aspect. Contrary to hypernymy and meronymy relation, simply applying LSPs does not extract synonym pairs from a big corpus. Instead, we extracted other semantic relations to ease detection of synonymy. Our methodology relies on some assumptions. One is that the synonym pairs mostly show similar semantic characteristics by definition. They share the same meronym/holonym and hypernym/hyponym relations. Particular lexico-syntactic patterns can be used to initiate the acquisition process of those semantic features. Secondly, a pair of synonym words mostly shares a particular list of governing verbs and modifying adjectives. The more a pair of words are governed by similar verb profile and modified by similar adjectives, the more likely they are synonym. We built ten groups of syntactic patterns according to their syntactic structures. To apply machine learning algorithm, three annotators manually and randomly selected 200 synonym pairs and 200 non-synonyms. Non-synonym pairs were especially selected from associated (relevant) pairs such as tree-leaf, appleorange, school-student. Otherwise, such negative example selection could lead to false inference. The main challenge faced in the experiments is the lack of features of some words due to their corpus evidence. Thus, such instances were eliminated. Remaining instances was classified by the most suitable algorithm which is the logistic regression. It can easily be used for binary classification in domains with numeric attributes and nominal target class. As long as individual performance of any feature is higher than f-measure of 50%, it is considered as useful features or considered independent feature from target class. The aim was to find out which features are the most informative for detecting synonymy and contribute most to the overall success of the model. When comparing the results, it was clearly observed that the semantic features are notably better than syntactic dependency models in finding true An Integrated Approach to Automatic Synonym Detection 125 synonyms. The most effective attributes are meronymy and holonymy features with weighted average f-measure of 68.7% and 73.7% respectively. The analysis indicated that the possible reason for the success is sufficiency in the number of cases from which semantic and dependency features are constructed. As a consequence of insufficient data volume, hypernymy/hyponymy relation is relatively worse than meronymy. Among dependency relations, G1, G4 and G7 outperformed the others. Likewise, it was also observed that sufficiency in the number of cases was the strong factor. After aggregating all useful features, the same learning process was carried out. The aggregated model shows promising results and performance. Regression model achieved an acceptable f-measure of 80.3%. One of the main contributions of the study is that the system first obtains acquirable semantic relations such as hypernymy, meronymy from corpus by lexico-syntactic patterns to extract subtle relations such as synonymy. The second contribution of the study is considered to be the first major attempt for Turkish synonym identification based on corpus-driven approach. References 1. Mandala, R., Tokunaga, T., Tanaka, H.: Combining Multiple Evidence from Different Types of Thesaurus for Query Expansion. In: 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Berkeley, CA, USA, pp. 191–197 (1999) 2. Bai, J., Song, D., Bruza, P., Nie, J., Cao, G.: Query Expansion Using Term Relationships in Language Models for Information Retrieval. In: 14th ACM International Conference on Information and Knowledge Management, Bremen, Germany, pp. 688–695 (2005) 3. Stefan, R., Liu, Y., Vasserman, A.: Translating Queries into Snippets for Improved Query Expansion. In: 22nd International Conference on Computational Linguistics, COLING 2008, Manchester, UK, pp. 737–744 (2008) 4. Lin, D.: Automatic Retrieval and Clustering of Similar Words. In: 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Montreal, Quebec, Canada, pp. 768–774 (1998) 5. Inkpen, D.: A Statistical Model for Near-synonym Choice. ACM Transactions on Speech and Language Processing 4(1), 1–17 (2007) 6. Barzilay, R., Elhadad, M.: Using Lexical Chains for Text Summarization. In: Proceedings of the ACL Workshop on Intelligent Scalable Text Summarization, Madrid, Spain, pp. 10–17 (1997) 7. Inkpen, D.Z., Hirst, G.: Near-synonym Choice in Natural Language Generation. In: Recent Advances in Natural Language Processing III, Selected Papers from RANLP 2003, Borovets, Bulgaria, pp. 141–152 (2003) 8. McCarthy, D., Navigli, R.: The English Lexical Substitution Task. Language Resources and Evaluation 43(2), 139–159 (2009) 9. Mirkin, S., Dagan, I., Geffet, M.: Integrating Pattern-Based and Distributional Similarity Methods for Lexical Entailment Acquisition. In: Proceedings of the COLING/ACL 2006 on Main Conference Poster Sessions, Sydney, Austraila, pp. 579–586 (2006) 10. Harris, Z.: Distributional Structure. Word 10(23), 146–162 (1954) 126 T. Yıldız, S. Yıldırım, and B. Diri 11. Hearst, M.A.: Automatic Acquisition of Hyponyms from Large Text Corpora. In: 14th International Conference on Computational Linguistics, COLING 1992, Nantes, France, pp. 539–545 (1992) 12. Lin, D., Zhao, S., Qin, L., Zhou, M.: Identifying Synonyms among Distributionally Similar Words. In: IJCAI 2003, Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, Acapulco, Mexico, pp. 1492–1493 (2003) 13. Wang, W., Thomas, C., Sheth, A.P., Chan, V.: Pattern-based Synonym and Antonym Extraction. In: Proceedings of the 48th Annual Southeast Regional Conference, Oxford, MS, USA, p. 64 (2010) 14. Hagiwara, M.: A Supervised Learning Approach to Automatic Synonym Identification Based on Distributional Features. In: 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Student Research Workshop, Columbus, OH, pp. 1–6 (2008) 15. Heylen, K., Peirsman, Y., Geeraerts, D., Speelman, D.: Modelling Word Similarity: an Evaluation of Automatic Synonymy Extraction Algorithms. In: International Conference on Language Resources and Evaluation, LREC (2008) 16. Hindle, D.: Noun Classification from Predicate-Argument Structures. In: 28th Annual Meeting of the Association for Computational Linguistics, Pittsburgh, Pennsylvania, USA, pp. 268–275 (1990) 17. Gasperin, C., Gamallo, P., Agustini, A., Lopes, G., Lima, V.: Using Syntactic Contexts for Measuring Word Similarity. In: Workshop on Knowledge Acquisition and Categorization, ESSLLI (2001) 18. Curran, J.R., Moens, M.: Improvements in Automatic Thesaurus Extraction. In: ACL 2002 Workshop on Unsupervised Lexical Acquisition, Philadelphia, USA, pp. 59–66 (2002) 19. van der Plas, L., Bouma, G.: Syntactic Contexts for Finding Semantically Related Words. In: Meeting of Computational Linguistics in the Netherlands (CLIN), Amsterdam, pp. 173–186 (2005) 20. Barzilay, R., McKeown, K.: Extracting Paraphrases from a Parallel Corpus. In: 39th Annual Meeting and 10th Conference of the European Chapter, Proceedings of the Conference, Toulouse, France, pp. 50–57 (2001) 21. Ibrahim, A., Katz, B., Lin, J.: Extracting Structural Paraphrases from Aligned Monolingual Corpora. In: The Second International Workshop on Paraphrasing: Paraphrase Acquisition and Applications, Sapporo, Japan, pp. 57–64 (2003) 22. Shimohata, M., Sumita, E.: Automatic Paraphrasing Based on Parallel Corpus for Normalization. In: Third International Conference on Language Resources and Evaluation, Las Palmas, Canary Islands, Spain, pp. 453–457 (2002) 23. van der Plas, L., Tiedemann, J.: Finding Synonyms Using Automatic Word Alignment and Measures of Distributional Similarity. In: 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia, pp. 866–873 (2006) 24. Blondel, V.D., Sennelart, P.: Automatic Extraction of Synonyms in a Dictionary. In: SIAM Workshop on Text Mining, Arlington, VA (2002) 25. Wang, T., Hirst, G.: Exploring Patterns in Dictionary Definitions for Synonym Extraction. Natural Language Engineering 18(3), 313–342 (2012) 26. Freitag, D., Blume, M., Byrnes, J., Chow, E., Kapadia, S., Rohwer, R., Wang, Z.: New Experiments in Distributional Representations of Synonymy. In: Ninth Conference on Computational Natural Language Learning (CoNLL), Ann Arbor, Michigan, pp. 25–32 (2005) 27. Terra, E., Clarke, C.: Frequency Estimates for Statistical Word Similarity Measures. In: HTL/NAACL 2003, Edmonton, Canada, pp. 165–172 (2003) An Integrated Approach to Automatic Synonym Detection 127 28. Turney, P.D., Littman, M.L., Bigham, J., Shnayder, V.: Combining Independent Modules in Lexical Multiple-choice Problems. In: Recent Advances in Natural Language Processing III, Selected Papers from RANLP 2003, Borovets, Bulgaria, pp. 101–110 (2003) 29. Turney, P.D.: A Uniform Approach to Analogies, Synonyms, Antonyms, and Associations. In: 22nd International Conference on Computational Linguistics, Coling 2008, Manchester, UK, pp. 905–912 (2008) 30. Yates, A., Goharian, N., Frieder, O.: Graded Relevance Ranking for Synonym Discovery. In: 22nd International World Wide Web Conference, WWW 2013, Rio de Janeiro, Brazil, pp. 139–140 (2013) 31. Hagiwara, M., Ogawa, Y., Toyama, K.: Selection of Effective Contextual Information for Automatic Synonym Acquisition. In: 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia, pp. 353–360 (2006) 32. Yazici, E., Amasyali, M.F.: Automatic Extraction of Semantic Relationships using Turkish Dictionary Definitions. In: EMO Bilimsel Dergi, Istanbul (2011) 33. Sak, H., Güngör, T., Saraçlar, M.: Turkish Language Resources: Morphological Parser, Morphological Disambiguator and Web Corpus. In: Nordström, B., Ranta, A. (eds.) GoTAL 2008. LNCS (LNAI), vol. 5221, pp. 417–427. Springer, Heidelberg (2008) 34. Serbetci, A., Orhan, Z., Pehlivan, I.: Extraction of Semantic Word Relations in Turkish from Dictionary Definitions. In: ACL 2011 Workshop on Relational Models of Semantics, Portland, pp. 11–18 (2011) 35. Orhan, Z., Pehlivan, I., Uslan, V., Onder, P.: Automated Extraction of Semantic Word Relations in Turkish Lexicon. Mathematical and Computational Applications (1), 13–22 (2011) 36. Yildirim, S., Yildiz, T.: Automatic Extraction of Turkish Hypernym-Hyponym Pairs From Large Corpus. In: COLING (Demos), pp. 493–500 (2012) 37. Yildiz, T., Diri, B., Yildirim, S.: Analysis of Lexico-syntactic Patterns for Meronym Extraction from a Turkish Corpus. In: 6th Language and Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, pp. 126–138 (2013) 38. Yıldız, T., Yıldırım, S., Diri, B.: Extraction of Part-Whole Relations from Turkish Corpora. In: Gelbukh, A. (ed.) CICLing 2013, Part I. LNCS, vol. 7816, pp. 126–138. Springer, Heidelberg (2013)