Analyzing Regional Variation of English Dialects in Central Asia Using Text Similarity Algorithm - An Exploration of Similarity Measure and Clustering Features

Abstract

With the acceleration of globalization, English has become a universal language, but its localized use in different regions has given rise to numerous regional variations with systematicity. The lack of large-scale annotated corpus for English in Central Asia makes traditional supervised categorization difficult to implement, for this reason, this paper reveals regional text clustering based on context and mapping relationships through text similarity algorithm. Then, the traditional Topic Frequency-Inverse Document Frequency (TFIDF) is improved from the perspective of inter- and intra-category discretization, and the improved ITFIDF method is used to calculate the feature weights. The results show that in the self-constructed English corpus of Central Asia, the average difference between the resultant values of the proposed method for calculating text similarity and Miller's human-determined values is 0.0583, and the overall deviations are all controlled within 0.15, which is basically in line with Miller's human-determined values. In terms of accuracy rate, Recall and F1 value, the proposed word weight calculation method has certain superiority. The study can identify the lexical and syntactic preferences of Central Asian English for business and diplomatic people, reduce misunderstanding, and improve the efficiency of cross-cultural communication.