DEVELOPMENT OF A NODE BALANCING ALGORITHM USING MACHINE LEARNING AND DYNAMIC WEIGHT CALCULATION

Abstract

The paper presents a method and software solution for node balancing in multi-node systems using machine learning techniques and dynamic weight computation. The relevance of this research is determined by the bottleneck effect that arises during big data processing, when one node becomes overloaded while others remain underutilized, resulting in inefficient resource usage. The aim of the research was to design a software system capable of real-time load distribution across nodes to maintain stable performance under high data volumes. The proposed algorithm combines an online machine learning model, which estimates query complexity based on input parameters (length, number of headers, type of operations, presence of JOIN clauses), with a dynamic weight calculation mechanism that regulates node load.A distinctive feature is the ability of the system to learn directly while processing requests, eliminating the need for prior data preparation and reducing deployment time. The research implemented the Python River library, which provides tools for online learning. To evaluate the efficiency of the algorithm, a simulated three-node system was tested with a dataset of 10,000 queries. The proposed method provides an average efficiency improvement of 2–5 % due to a more balanced load distribution between nodes compared to the baseline algorithms. The experimental results demonstrated that the proposed method achieved efficiency comparable to classical approaches (Round Robin, Random, Join Shortest Queue), while ensuring a more balanced query distribution due to complexity-aware processing. Comparative analysis confirmed that even under controlled conditions, the algorithm reached the performance level of baseline methods, whereas in real-world scenarios, with more heterogeneous request characteristics, its advantages are expected to be more significant.Future work will focus on improving testing conditions, deploying the algorithm in real distributed environments, and decoupling the machine learning logic from the overall architecture to enhance flexibility and scalability.