SAFE RELIABILITY-ADAPTIVE MODALITY FUSION FOR STREAMING FORECASTING OF MULTIMODAL TIME SERIES UNDER TEMPORARY DATA DEGRADATION
DOI:
https://doi.org/10.35546/kntu2078-4481.2026.2.48Keywords:
machine learning, data analysis, information systems, decision support systems, multimodal time series, streaming forecasting, data degradation, modality reliability, dynamic fusion, clean gateAbstract
Streaming decision support systems operate causally and under low-latency constraints; therefore, multimodal time series must be analyzed online. In real pipelines, individual data sources temporarily degrade (missing values, elevated noise, and scale shifts), making static fusion strategies unstable: a degraded modality continues to affect the forecast, while any additional control risks harming quality in segments without degradation. This paper considers a reliabilityadaptive dynamic modality-fusion method for streaming forecasting, where online modality-reliability estimates are used as a control signal for reweighting modal predictors with smoothing, and a safe “clean regime” is implemented via a clean gate: in clean windows, the output matches a strong early-fusion baseline, which excludes quality regression in degradation-free segments. Effectiveness is evaluated in the prequential streaming protocol on controlled streams with deterministic degradation injections and on real data (UCI Appliances Energy Prediction). It is shown that, in the clean regime, the method matches the early-fusion baseline ( MAE = 0.56 ± 0.13 , RMSE = 0.70 ± 0.17 ); under alternating missingness it reduces error ( MAE = 0.64 ± 0.11 versus 0.68 ± 0.08 ); it demonstrates higher robustness at «missing_ level» = 0.9 (MAE 0.64 versus 0.70 ); and on degraded UCI Appliances segments it achieves MAE = 63.69 ± 5.82 versus 381.91± 95.22 . Modality reliability is treated as a causal probabilistic estimate of being in a non-degraded state and is used as a control interface for streaming fusion. The results are positioned as part of a coherent online pipeline that satisfies causality, low latency, and bounded computational resources in streaming decision support. Per-step inference latency remains in the microsecond range (on average 10.07μs ), which makes the approach applicable to low-latency streaming decision support
References
Baltrušaitis T., Ahuja C., Morency L.-P. Multimodal machine learning: A survey and taxonomy // IEEE Transactions on Pattern Analysis and Machine Intelligence. 2019. Vol. 41(2). P. 423–443. DOI: 10.1109/TPAMI.2018.2798607.
Lahat D., Adali T., Jutten C. Multimodal data fusion: an overview of methods, challenges, and prospects //Proceedings of the IEEE. 2015. Vol. 103(9). P. 1449–1477. DOI: 10.1109/JPROC.2015.2460697.
Dawid A. P. Present position and potential developments: Some personal views: Statistical theory: the prequential approach // Journal of the Royal Statistical Society. Series A (General). 1984. Vol. 147(2). P. 278–292. DOI:10.2307/2981683.
Hyndman R. J., Koehler A. B. Another look at measures of forecast accuracy // International Journal of Forecasting. 2006. Vol. 22(4). P. 679–688. DOI: 10.1016/j.ijforecast.2006.03.001.
Candanedo L. M., Feldheim V., Deramaix D. Data driven prediction models of energy use of appliances in a lowenergy house // Energy and Buildings. 2017. Vol. 140. P. 81–97. DOI: 10.1016/j.enbuild.2017.01.083.
UCI Machine Learning Repository. Appliances Energy Prediction. 2017. URL: https://archive.ics.uci.edu/dataset/374/appliances+energy+prediction.
Ramachandram D., Taylor G. W. Deep Multimodal Learning: A Survey on Recent Advances and Trends // IEEE Signal Processing Magazine. 2017. Vol. 34(6). P. 96–108. DOI: 10.1109/MSP.2017.2738401.
Jain R. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley-Interscience, 1991. URL: https://www.cse.wustl.edu/~jain/books/perfbook.htm.
Khaleghi B., Khamis A., Karray F. O., Razavi S. N. Multisensor data fusion: A review of the state-of-the-art //Information Fusion. 2013. Vol. 14(1). P. 28–44. DOI: 10.1016/j.inffus.2011.08.001.
Hall D. L., Llinas J. An introduction to multisensor data fusion // Proceedings of the IEEE. 1997. Vol. 85(1). P. 6–23. DOI: 10.1109/5.554205.
Atrey P. K., Hossain M. A., El Saddik A., Kankanhalli M. S. Multimodal fusion for multimedia analysis: a survey //Multimedia Systems. 2010. Vol. 16(6). P. 345–379. DOI: 10.1007/s00530-010-0182-0.
Jacobs R. A., Jordan M. I., Nowlan S. J., Hinton G. E. Adaptive mixtures of local experts // Neural Computation. 1991. Vol. 3(1). P. 79–87. DOI: 10.1162/neco.1991.3.1.79.
Kittler J., Hatef M., Duin R. P. W., Matas J. On combining classifiers // IEEE Transactions on Pattern Analysis and Machine Intelligence. 1998. Vol. 20(3). P. 226–239. DOI: 10.1109/34.667881.
Shim J. P., Warkentin M., Courtney J. F., Power D. J., Sharda R., Carlsson C. Past, present, and future of decision support technology // Decision Support Systems. 2002. Vol. 33(2). P. 111–126. DOI: 10.1016/S0167-9236(01)00139-7.
Gama J., Žliobaitė I., Bifet A., Pechenizkiy M., Bouchachia A. A survey on concept drift adaptation // ACM
Computing Surveys. 2014. Vol. 46(4). P. 1–37. DOI: 10.1145/2523813. 16. Webb G. I., Hyde R., Cao H., Nguyen H.-L., Petitjean F. Characterizing concept drift // Data Mining and Knowledge Discovery. 2016. Vol. 30(4). P. 964–994. DOI: 10.1007/s10618-015-0448-4.
Che Z., Purushotham S., Cho K., Sontag D., Liu Y. Recurrent neural networks for multivariate time series with missing values // Scientific Reports. 2018. Vol. 8. Article 6085. DOI: 10.1038/s41598-018-24271-9.
Little R. J. A., Rubin D. B. Statistical Analysis with Missing Data. 2nd ed. Wiley, 2002. DOI: 10.1002/9781119013563.
