ADVANCED CONTROL STRATEGIES USING DEEP REINFORCEMENT LEARNING FOR IMPROVING ENERGY EFFICIENCY IN SMART BUILDINGS
DOI:
https://doi.org/10.35546/kntu2078-4481.2025.2.2.11Keywords:
reinforcement learning, energy efficiency, load, building, control, agent, optimizationAbstract
This research investigates building control strategies for decreasing the energy consumption. A review of existing literature indicates that prior studies employed either complex, model-based approaches or relatively less efficient Q-learning techniques. To address these limitations, we propose a novel control method for Heating, Ventilation, and Air Conditioning (HVAC) systems in medium-sized office buildings, utilizing an intelligent reinforcement learning controller grounded in the model-free Proximal Policy Optimization algorithm (PPO). Our approach incorporates simulations conducted in the EnergyPlus environment, which allows for a precise and responsive depiction of HVAC system dynamics under various control conditions. A key aspect of the study is the regulation of supply air temperature. To support the implementation and enhancement of our control strategy, we developed a co-simulation framework using the Gymnasium library. This environment served as a robust foundation for testing and fine-tuning reinforcement learning models. The performance of the proposed deep reinforcement learning-based controller was benchmarked against a conventional control strategy that adjusts supply air temperature based on outdoor air conditions. Experimental results showed a significant 27.8 % reduction in energy use, all while preserving indoor comfort in terms of temperature, humidity, and carbon dioxide levels. This work highlights the potential of reinforcement learning to simplify the deployment of advanced control techniques in real-world building energy management systems. Unlike traditional optimization methods that require detailed mathematical models of the system, deep reinforcement learning infers optimal control actions by analyzing the relationship between system states and the outcomes of selected actions.
References
Walnum H. T., Sartori I., Ward P., Gros S. Demonstration of a low-cost solution for implementing MPC in commercial buildings with legacy equipment. Applied Energy. 2025, 380, 125012. https://doi.org/10.1016/j.apenergy.2024.125012
Perera A. T. D., Kamalaruban P. Applications of reinforcement learning in energy systems. Renewable and Sustainable Energy Reviews. 2021, 137, 110618. https://doi.org/10.1016/j.rser.2020.110618
Cai W., Sawant S., Reinhardt D., Rastegarpour S., Gros S. A Learning-Based Model Predictive Control Strategy for Home Energy Management Systems. IEEE Access. 2023, vol. 11, pp. 145264–145280. https://doi.org/10.1109/ACCESS.2023.3346324
Bayer D., Pruckner M. Enhancing the Performance of Multi-Agent Reinforcement Learning for Controlling HVAC Systems. 2022 IEEE Conference on Technologies for Sustainability (SusTech). 2022, pp. 187–194. https://doi.org/10.1109/SusTech53338.2022.9794179
Razzano G., Brandi S., Piscitelli M. S., Capozzoli A. Rule extraction from deep reinforcement learning controller and comparative analysis with ASHRAE control sequences for the optimal management of Heating, Ventilation, and Air Conditioning (HVAC) systems in multizone buildings. Applied Energy. 2025, vol. 381, 125046. https://doi.org/10.1016/j.apenergy.2024.125046
Heidari A., Maréchal F., Khovalyg D. An occupant-centric control framework for balancing comfort, energy use and hygiene in hot water systems: A model-free reinforcement learning approach. Applied Energy. 2022, vol. 312, 118833. https://doi.org/10.1016/j.apenergy.2022.118833
Wen Z., O’Neill D., Maei H. Optimal Demand Response Using Device-Based Reinforcement Learning. IEEE Trans. Smart Grid. 2015, 6(5), 2312–2324. https://doi.org/10.1109/TSG.2015.2396993
Huang X., Zhang D., Zhang X. Energy management of intelligent building based on deep reinforced learning. Alexandria Engineering Journal. 2021, 60(1), 1509–1517. https://doi.org/10.1016/j.aej.2020.11.005
Mugumya K. L., Wong J. Y., Chan A., Yip C.-C., Ghazy S. Indoor haze particulate control using knowledge graphs within self-optimizing HVAC control systems. IOP Conf. Ser.: Earth Environ. Sci. 2020, 489(1), 012006. https://doi.org/10.1088/1755-1315/489/1/012006
Kim S., Lim H. Reinforcement Learning Based Energy Management Algorithm for Smart Energy Buildings. Energies. 2018, 11(8), 2010. https://doi.org/10.3390/en11082010
Korkidis P., Dounis A., Kofinas P. Computational Intelligence Technologies for Occupancy Estimation and Comfort Control in Buildings. Energies. 2021, 14(16), 4971. https://doi.org/10.3390/en14164971
Mahapatra C., Moharana A., Leung V. Energy Management in Smart Cities Based on Internet of Things: Peak Demand Reduction and Energy Savings. Sensors. 2017, 17(12), 2812. https://doi.org/10.3390/s17122812
Mocanu E., et al. On-Line Building Energy Optimization Using Deep Reinforcement Learning. IEEE Trans. Smart Grid. 2019, 10(4), pp. 3698–3708. https://doi.org/10.1109/TSG.2018.2834219
Coraci D., et al. A scalable approach for real-world implementation of deep reinforcement learning controllers in buildings based on online transfer learning: The HiLo case study. Energy and Buildings. 2025, vol. 329, 115254. https://doi.org/10.1016/j.enbuild.2024.115254
Spangher L., et al. Prospective Experiment for Reinforcement Learning on Demand Response in a Social Game Framework. Proceedings of the Eleventh ACM International Conference on Future Energy Systems. 2020, pp. 438–444. [In English]. https://doi.org/10.1145/3396851.3402365
Forootani A., Rastegar M., Jooshaki M. An Advanced Satisfaction-Based Home Energy Management System Using Deep Reinforcement Learning. IEEE Access. 2022, vol. 10, pp. 47896–47905. https://doi.org/10.1109/ACCESS.2022.3172327
Park B., Rempel A. R., Lai A. K. L., Chiaramonte J., Mishra S. Reinforcement Learning for Control of Passive Heating and Cooling in Buildings. IFAC-PapersOnLine. 2021, 54(20), pp. 907–912. https://doi.org/10.1016/j.ifacol.2021.11.287
Almughram O., Slama S. A. B., Zafar B. A. A Reinforcement Learning Approach for Integrating an Intelligent Home Energy Management System with a Vehicle-to-Home Unit. Applied Sciences. 2023, 13(9), 5539. https://doi.org/10.3390/app13095539
Wang J., Wang Y., Qiu D., Su H., Strbac G., Gao Z. Resilient energy management of a multi-energy building under low-temperature district heating: A deep reinforcement learning approach. Applied Energy. 2025, vol. 378, 124780. https://doi.org/10.1016/j.apenergy.2024.124780
Tortorelli A., Sabina G., Marchetti B. A Cooperative Multi-Agent Q-Learning Control Framework for Real-Time Energy Management in Energy Communities. Energies. 2024, 17(20), 5199. https://doi.org/10.3390/en17205199
Vazquez-Canteli J., Kampf J., Nagy Z. Balancing comfort and energy consumption of a heat pump using batch reinforcement learning with fitted Q-iteration. Energy Procedia. 2017, vol. 122, pp. 415–420. https://doi.org/10.1016/ j.egypro.2017.07.429
Cordeiro-Costas M., Villanueva D., Eguía-Oller P., Granada-Alvarez E. Intelligent energy storage management trade-off system applied to Deep Learning predictions. Journal of Energy Storage. 2023, vol. 61, 106784. https://doi.org/10.1016/j.est.2023.106784
Du Y., Li F., Kurte K., Munk J., Zandi H. Demonstration of Intelligent HVAC Load Management with Deep Reinforcement Learning: Real-World Experience of Machine Learning in Demand Control. IEEE Power and Energy Mag. 2022, 20(3), pp. 42–53. https://doi.org/10.1109/MPE.2022.3150825
Liu C., Xue Z. Adaptive Optimization Design of Building Energy System for Smart Elderly Care Community Based on Deep Deterministic Policy Gradient. Processes. 2023, 11(7), 2155. https://doi.org/10.3390/pr11072155
Towers M., et al. Gymnasium: A Standard Interface for Reinforcement Learning Environments. ICLR 2025 Conference, 2025. https://doi.org/10.48550/ARXIV.2407.17032
Liang E., et al. RLlib: Abstractions for Distributed Reinforcement Learning. Proceedings of the 35th International Conference on Machine Learning. 2018, vol. 80, pp. 3053–3062. https://doi.org/10.48550/arXiv.1712.09381
