KannadaLex: A lexical database with psycholinguistic information

A large set of Kannada sentences was pooled from the resources under various domains such as culture, arts, science, social, sports, food, health, politics, education, business, economy, world/local news, entertainment, and so on, by scraping websites of these domains. Since language evolves and spoken language varies widely from standard formal language, it is more logical to consider newspapers, periodicals, fiction, and non-fiction textual contents. So, the archives of Kannada news articles from the past decade were used to source the data, because they contain a consistent collection of vocabulary from diverse fields and reflect the most current form of language. The two tasks, data scraping and pre-processing, were performed to collect the data. Figure 2 shows the steps involved in data collection. In scraping data from news article archives, for each date given as an input, a list of article links was returned as a response, and articles were read from each of these links. Dates were input such that data collected is not centered on a particular news piece. Once data has been sourced, a pre-processing or data cleaning is an essential step. The process of cleaning text involves eliminating unusable sections, while tokenization involves breaking down the text into individual units such as words or phrases. Identifying the phonological and syllabic structures for each word's written form is also part of the process. The articles read were stripped into sentences on encountering sentence delimiters. Each of these sentences is then processed by further tokenizing them into words and performing a preliminary check to ensure that the sentence is entirely in Kannada script. Words that are acceptable in this manner are written into flat files, indexed by the date field. The complete sentences are also stored separately, keeping in mind the need for sentence context during the part-of-speech (POS) tagging of words. The output of this phase is an orthographic representation of a large set of words and the sentences that they were obtained from. In total, around 35.4 MB of uncleaned data corresponding to six months of newspaper articles were obtained. Other word representations such as phonological and syllabic representations were also obtained. The words with less than four phonemes, which are usually caused due to abbreviations or stop words, are excluded from the database.

KannadaLex database contains two main tables, Base and RootAndFamily, and three supporting tables that not only help obtain few features of the former two, but also provide independent lexical statistics useful for psycholinguistic research. A detailed description of individual fields in the Base table and the RootAndFamily is provided in Table 1 and Table 2.

The tables are obtained as follows:

The lemma, phonological neighborhood, word length, and frequencies of the word are the most important linguistic features used in the psycholinguistic study. The word length and frequency count in a corpus are two fundamental psycholinguistic variables that have been found to affect word recognition tasks in many languages [4, 6, 7], particularly in English [3]. For the Kannada language, the word length is nothing but the number of bytes in the word's UNICODE representation. Other measures, such as the number of syllables and phonemes, the number of complex syllables, and so on, describe the complexity of Kannada words. They are calculated from their phonological and syllabic representations. The frequencies of syllables and bigrams are maintained in the database. The summed syllable frequency, the summed bigram syllable frequency, and their averages determine the frequency of word parts. They are calculated by extracting syllable/bigram syllable frequencies from the database. We extract 13 psycholinguistic features in total. These have been described in the following sections.

Orthographic representation: The orthographic representation of the word is the UNICODE representation of the word adhering to the Unicode block (128 code points ranging from U+0C80 to U+0CFF) that contains characters for Kannada, Tulu, and Kodava languages. Words sourced from the archive are initially in this representation. Figure 3 shows a word, corresponding orthographic representation, and its Unicode.

Phonological representation:  Phonological neighbors are defined as a set of words that differ by a single sound (which is more relevant than orthographic neighborhood for the Kannada language). For every word, the Phonological Neighbors table in the database is first looked up. If it is not present in the table, then all words with one phoneme difference are found, non-words are filtered out, and phonological neighbors are recorded. The number of neighbors gives the phonological neighborhood density, and the mean sum of frequencies of all the neighbors gives the phonological neighborhood mean frequency. It is the description of the combination of sounds that comprise words. In Kannada, the vowels (svara), consonants (yvanjana), and yogavahas (part-vowel, part-consonant) is the phonemic letters. The phonemes in each word are segmented and represented with “-” as a separator. The consonants are described with a diacritic (Virama), vowels stripped and represented separately. For an orthographic representation of a word as an input, the phonological representation module outputs its phonological representation. The phonological representation of the word in Figure 3 is shown in Figure 4.

Syllabic representation and CV pattern: Syllables, units of pronunciation that form words, are considered to play a major role in psycholinguistic tasks. In Kannada, each word is composed of aksharas or kagunita, which directly correspond to syllables. Like in many Indian languages, Kannada also follows a CVCVCV pattern for words with syllables in the formats V (Vowel in its primary orthographic form), CV (Consonant-Vowel, secondary form of vowel added to primary orthographic form of consonant), CCV (Consonant-Consonant-Vowel: Ottaksharas, first consonant in primary form, second below the first in its secondary form, and vowel indicated by a first consonant), and CCCV (Consonant-Consonant-Consonant-Vowel, usually in loan words from Sanskrit); the last two being classified as complex syllables. Figure 5 shows the four basic types of syllables.

The phonological representation of the word is taken as an input and the output is a CV string, each syllable separated by a “-” and represented as one of the above CV patterns. The phonological representation is also separated at syllable endings by a “;”, thus providing the syllabic representation of the word. The syllabic representation of the word is in Figure 3, and its CV pattern is shown in Figure 6.

Word length: Word length is the number of segments in the Unicode representation of the word or the number of bytes used to represent the word in Unicode. Word length is obtained from the orthographic representation of the word. Word length for the word in Figure 3 is thus eight.

Number of phonemes: Obtained from the phonological representation, it is the number of phonemes that the word consists of. In Figure 4, the example word or “kSetrada” (using WX notation), which means ‘of the field’, contains 8 phonemes. This count is more accurate in describing the length of a Kannada word than a mere Unicode segments’ count, as it accurately determines the number of sounds in the word.

Number of syllables: The number of units of pronunciation vaguely describes the complexity of a Kannada word. From the syllabic representation or the CV pattern, a count of the number of syllables constituting the word is obtained. Considering Figure 6, the number of syllables in the example word is three.

Number of complex syllables: In various languages, it has been proven that the complexity of syllables directly affects the difficulty of recognizing a word. Thus, a count of the number of complex vowels in Kannada is worth noting as a feature. In Kannada, vowels taking the form CCV or CCCV are considered complex syllables. Considering Figure 6, the number of complex syllables in the example word is two.

Word frequency: Frequency is an important factor in psycholinguistic studies. Word frequency is the number of times the word occurs in the data that was sourced as a function of the total number of words sourced. Words in each file (that correspond to each date) are processed to find the unique words and their frequencies within each file. These words are then compared to words in other files and frequencies are updated accordingly. These frequencies are then modified based on the total number of words that were sourced.

Summed syllable bigram frequency and mean: In Kannada, bi-syllable frequencies contain more value than orthographic bigram frequencies (bisyllable and bigram frequencies are used interchangeably in the text that follows). The frequency of every successive sequence of two syllables is extracted and summed up for obtaining Summed Bigram Frequency, and it is divided by a total number of bigrams in the word to obtain the mean. To do so, on encountering each word for obtaining it syllabic representation, all bigrams (bisyllables) in the word are inserted into a separate database: A new entry is made in case of the absence of bigram in the database or the frequency of the existing entry updated if bigram is found. After one round of processing all words in the database, a second pass calculates the summed bigram frequency as well as its mean.

Summed syllable frequency and mean: Syllables in Kannada have a major role to play in word recognition and other psycholinguistic tasks, since they are represented as one unit or akshara. A summed syllable frequency provides an estimate of how familiar the word's syllables are and thus corresponds to the ease of recognizing the word. The approach to finding this is similar to that used to find Summed Bigram Frequency and its mean.

Phonological neighborhood density: A phonological neighbor of a word is another word that might differ from the former by maximally one phoneme (through deletion, addition, or substitution). To find a word's neighbors, each other word in the corpus is checked for its similarity (minimum phonological edit distance of one unit) to the query word and then added to a list of neighbors. A total count of these neighbors gives the word's phonological neighborhood density.

Phonological neighborhoods mean frequency: Neighborhood means the frequency is calculated by adding up the frequencies of all the neighbors and dividing this sum by the density. In the process of finding the neighborhood density, on finding a valid neighbor, its frequency is extracted and added.

Lemma: Stemmers generally derive the word stem from the inflected form of a word, and the word stem may or may not be the same as the actual morphological root. Kannada has a complex morphology, characterized by its agglutinative nature. There are more than 10K root words and more than a million morphed variants, and stemming is particularly challenging due to the several morphophonemic changes that take place during suffix attachment. However, with a sufficiently large dataset covering almost all roots, morphemes, inflections, and suffixes, one can attempt to build a stemmer. A supervised stemmer employing an SVM classifier is built to extract the lemma. Additional lemma information is stored in the database. The overall design of the stemmer is depicted in Figure 7. The stemmer is internally built in two parts: The first predicts the split point, and the second predicts if any additional changes are to be made to the root. The algorithm for designing a stemmer is explained below:

For each word in the lexical database, stemming is performed, and the lemma is stored as a feature in the base table. Further, all lemmas are identified, and their cumulative frequencies (summed frequency of the inflectional family) are calculated and stored in the databases.

Different psycholinguists and speech therapists require to use the contents in the database differently to form stimuli for their experiments and treatment. A user interface for the front end has been developed to assist users in querying the database according to their preferences, using a combination of recorded features. The selection of features was informed by research conducted across various other languages. However, not all of them may significantly impact the frequency of use or the ease of comprehension and learning in Kannada. These measures must be tested before being used for treatment or other advanced experiments. Preliminary validation of these features is attempted by correlating them with the frequency of use. The user interface has two main features: one is to generate stimuli based on a combination of characteristics or features, and the other feature is to download the entire database to enable researchers to work on the data directly and other developers to add to the existing database. To validate the features, the relationship between the frequency count of a word and the different numerical features is found. KannadaLex was thus developed and analyzed in three phases, beginning with collecting data that made up a large and diverse corpus and moving on to extracting all the features and tabulating them and finally performing a preliminary validation of the features before making it available to the public with means to query the data.

The result of the work is KannadaLex, a validated, free lexical database, with features that not only help in the psycholinguistic study but also research in the field of natural language processing. With 170,000 words of diverse domains extracted (and a target of 5 million words) and its features tabulated, it is the first of its kind in Kannada and poses as a tool that signals the start of research in the language. With 92.5% accuracy for splitting a word into root and suffix (and 75% for identifying the changes required), the stemmer that was developed presents itself as an important aid for many linguistic researches and applications in the language and many other Indian languages, like Telugu, that bear resemblance with Kannada. Lemma information gives an overview of the inflectional family a word can take and can nurture research in the area of morphology.

The abundant syllable and bigram syllable information captured and stored can be used for studies related to syllabic structures. The minimum edit distance algorithm implemented to find neighbors of a word can be used in many applications that focus on the difference in the usages of words (such as discriminating between accents or versions of the language and studying the evolution of the language). The online tool provides a way to access this data easily, enabling even computer-illiterate researchers to access and query data. An option to download the data in different formats makes it more appealing to developers, too. Finally, the results of validation also show how each of the features that were logged is indeed an important psycholinguistic feature for the language.

There were a few deviations from the expected results, especially in the capacity of the stemming algorithm. Although the stemmer performed with high accuracies on the training and test data and during 5-fold cross-validation, it did not respond very well to unseen suffixes. Also, some words were partially stemmed. A Multilayer Perceptron classifier may give better results in learning and predicting unseen suffixes.

The results of the work can be summarized as having developed a lexical database for around 170K words (with plans to extend it to 5 million soon), with features that cover lengths, various frequencies (standard, summed syllabic, summed bigram syllabic, and phonological neighborhood), and morphology (lemma information), each of which has been preliminarily validated. As a side product, a stemmer has also been developed with 92.5% accuracy (using 5-fold cross-validation) for splitting a word into root and suffix (and 75% for identifying the changes required to be made to the root word). An online tool, i.e., a front end, has been built to freely access and query the database.

KannadaLex, a lexical database in the Kannada language, contains five major tables with features such as lengths, frequencies, and grammar information intended to help psycholinguists and speech therapists to prepare lists of stimuli with ease and accuracy. It will also serve as a database of great interest to other researchers vested in natural language processing and linguistics. Being based on text sourced from a local newspaper and several blogs, this database currently consists of about 170K words from diverse domains and is planned to house a total of about five million words after extending and sourcing more data. Features extracted include orthographic, phonological, and syllabic information along with information regarding phonological neighborhood and inflectional family and summed syllable and bigram syllable frequencies. The database also contains syllable and bigram syllable frequencies for further research in the field. KannadaLex's online tool provides a front end for researchers to query and access the validated data.

Kannada WordNet, a part of Indo-WordNet, is a lexical database for semantic relationship between words such as synonymy, antonymy, homonymy, hypernymy-hyponymy, entailment relations depending on the POS category of the word. Kannada nouns, verbs, adjectives, and adverbs are organized into synonym sets called the synsets, each representing one underlying concept. However, KannadaLex contains orthographic, phonological, syllabic, and morphological information that are of interest to psycholinguists and speech therapists. In addition, it contains features such as syllable count, complex syllable count, and phonological neighborhood. Kannada WordNet consists of 12,765 nouns, 3,119 verbs, 5,988 adjectives, and 170 adverbs. The synsets normally exist separately for each of the categories. Each word in Indo-WordNet has a Synset ID, Synonyms, gloss, example statement, and the gloss. So, the purpose of KannadaLex is different from Kannada WordNet.

We assessed the performance of the proposed stemmer on KannadaWordNet. The latest version of KannadaWordnet [43] contains approximately 29K Nouns and 5.5K verbs. In our experiment, as a preprocessing step, we excluded the words that are hyphenated or the words having “_” symbol. The stemmer achieved an accuracy of 83.42% in identifying the root words using the SVM models mentioned in Section 3.1. Some instances of both successful and unsuccessful identification of root words are presented in Table 3. In the table, the word and its WX notation are given. We observe that the KannadaWordNet has relatively many words in the root form compared to our dataset. However, the verbs in KannadaWordNet have more morphological modifications. Our stemmer can identify the root word when the word itself is a root word or the word contains the root word without structural changes in most of the cases, e.g., ಹೊಂದಿಸುವಿಕೆ, ಮದುವೆಯಾಗು, and so on. But it is also problematic in certain cases, such as ಮಡಿವಾಳ, ಕೊಡದೆಯಿರು, and so on. The stemmer is able to handle root words that have undergone additional changes in the word, for example, ಅಲುಗಾಡಿಸಿಸು. Overall, the stemmer performs very well in most of the cases.

We outline two possible use cases of KannadaLex in the following paragraphs. A web-based user interface (UI) is provided for accessing the KannadaLex. Using ASP.NET with C#, a web-interface is built to allow psycholinguists, speech therapists, and other users to access the lexical databases; or to query the databases by choosing a combination of features. The screenshots of the developed web-based UI are shown in the Appendix section. The web-interface has two main features: (i) to generate stimuli based on a combination of characteristics or features and (ii) to download the entire database to enable researchers to work on the data directly and other developers to add to the existing database The two features are explained here in detail.

This database, comprising 170K words from various domains, is a pioneering tool in Kannada. It serves not only for psycholinguistic studies but also aids research in natural language processing. This tool proves beneficial for linguistic research and applications in Kannada and related languages such as Telugu. Additionally, the lemma information provided by the database offers insights into the inflectional family of a word, fostering research in morphology.

Case study-1: In speech therapy, a lexical database like KannadaLex can be used to identify phonological neighbors, which are words that differ by only one phoneme (sound). Identifying phonological neighbors is beneficial in assessing and treating speech disorders, such as phonological or articulation disorders [38]. A speech-language pathologist (SLP) may use a lexical database to compile a list of target words relevant to a client's speech goals. Throughout the therapy process, the SLP can track the client's progress by comparing their production accuracy and consistency with the target word and its phonological neighbors. Phonological neighborhood information helps the SLP in designing therapy activities and interventions. The SLP may incorporate the phonological neighbors into articulation or phonological therapy sessions to target specific sound substitutions, deletions, or distortions. Using words from the phonological neighborhood can facilitate auditory discrimination, sound production practice, and generalization of speech skills. The SLP can assess if the client has successfully acquired the target sound by evaluating their ability to produce it accurately within the phonological context of the word and differentiate it from the neighboring sounds.

Case study-2: KannadaLex provides detailed information about the lexical properties of words, including phonological representations, syllable structure, stress patterns, and phonetic transcriptions. Researchers can use KannadaLex to study phonological processes, analyze word structures, and investigate linguistic phenomena related to pronunciation and phonetics. For example, the analysis for the Kannada word hinneleyalli (in WX notation) as shown in Figure 10, the syllable details and the syllable bi-gram details below can be obtained with the KannadaLex but not with the KannadaWordNet.

Thus, KannadaLex is a unique lexical database in the Kannada language annotated with lexical and phonological information. There are no databases of this kind available for the Kannada language. The popular KannadaWordNet does not provide the psycholinguistic information such as the syllable frequency, orthographic/phonological neighborhood characteristics, and morphological information.