Publication IEICE TRANSACTIONS on Information and SystemsVol.E103-DNo.10pp.2228-2236 Publication Date: 2020/10/01 Publicized: 2020/07/14 Online ISSN: 1745-1361 DOI: 10.1587/transinf.2019EDP7330 Type of Manuscript: PAPER Category: Natural Language Processing Keyword: new word detection, BiLSTM,
Full Text: PDF(1MB)>>
Summary: With the widespread popularity of a large number of social platforms, an increasing number of new words gradually appear. However, such new words have made some NLP tasks like word segmentation more challenging. Therefore, new word detection is always an important and tough task in NLP. This paper aims to extract new words using the BiLSTM+CRF model which added some features selected by us. These features include word length, part of speech (POS), contextual entropy and degree of word coagulation. Comparing to the traditional new word detection methods, our method can use both the features extracted by the model and the features we select to find new words. Experimental results demonstrate that our model can perform better compared to the benchmark models.
