MODIFICATION OF THE METHOD OF LARGE TEXT SETS CLUSTERING

Abstract

In this paper, a comparative analysis of common clustering methods such as k-means, Latent Dirichlet Distribution or LDA, Hierarchical Clustering Algorithm or HC, Density-based spatial clustering of applications with noise or DBSCAN, and Gaussian Mixture Model or GMM was conducted. The analysis was performed according to the selected criteria, such as scalability, computational complexity, presence (or absence) of a predefined number of clusters, and the evaluation approach (absolute with a clear relation to the cluster or relative using probabilities). According to the results, the DBSCAN method was chosen for further consideration due to a number of advantages, and the modification mod_DBSCAN was proposed, which reduces the number of potential calculations at each iteration, as a result, reduces computational complexity, and also increases system performance in conditions of limited resources. The modification consists of two changes: the vectorization stage, which is based on the annotation and keywords of the text specified by the author instead of the full text, and distance estimation for the so-called noisy points, which is performed in two steps. Popular datasets for the clustering task were analyzed. The proposed modification was tested on own Academ Lib Set dataset, formed on the basis of materials in the electronic catalog of the NURE scientific library. The analysis of the results showed an improvement in Precision by 5.6%, Recall by 12.5%, and F-score by 9.65%, which proves the effectiveness of the proposed modification. Further developments include testing combinations of methods and modules into larger functional blocks to identify and eliminate potential problems, as well as further optimization of such blocks. A separate work will investigate the approach to re-clustering after the dataset is updated. The quality of the new distribution is planned to be assessed based on the Rand index.