ITERATIVE FORMALIZATION OF EXPERT KNOWLEDGE INTO SYSTEM INSTRUCTIONS FOR LARGE LANGUAGE MODELS
DOI:
https://doi.org/10.35546/kntu2078-4481.2026.2.33Keywords:
large language models; LLM; knowledge formalization; prompt optimization; automated document auditing; iterative instruction tuning; gender-sensitive recoveryAbstract
This paper addresses the problem of logical inference degradation in large language models (LLMs) during semantic analysis of strategic documents exceeding 100 pages in length. The study focuses on developing a framework for the iterative formalization of tacit expert knowledge into system instructions to automate the auditing of community development plans for compliance with gender-sensitive recovery standards. Methodology. A multi-level logical inference approach, structured as an Orchestrator–Executor architecture, was implemented using models of the gemini-2.5-flash class. The mathematical model of the system is represented as a tuple of operators comprising: an Initializer-Extractor for reverse engineering of expert reasoning; an Executor; a QA-Auditor for hallucination detection; a Judge; and a Trainer responsible for constructing an explanatory model of errors. Optimization is realized as an iterative cycle incorporating a Human-in-the-Loop (HITL) mechanism to prevent semantic drift and the loss of domain-specific terminology. To eliminate anachronisms, a semantic version control mechanism was introduced that adapts analytical criteria to the historical context of the document under review. Results. Verification was conducted through blind testing on strategic plans of 15 territorial communities. The results confirmed the high generalization capacity of the optimized prompt, achieving with a validation level approximating that of professional human assessment on a 0–10 point scale. The application of the Orchestrator–Executor architecture enabled a substantial reduction in operational costs while preserving the accuracy of fact extraction. Novelty. The conceptual contribution lies in reconceptualizing knowledge distillation as a process of "crystallizing" expert heuristics into the semantic space of system prompts, without requiring neural network fine-tuning. Practical Significance. The practical relevance of the research is demonstrated within the framework of the GenPlan project, where the developed toolkit provides screening of strategic documentation in support of gendersensitive community recovery planning
References
Новаковський А., Яловега І. Визначення ролей агентів великих мовних моделей (LLM) у моделі дизайн-процесу. Сучасний стан наукових досліджень та технологій в промисловості. 2025. № 3 (33). С. 58–72. DOI:10.30837/2522-9818.2025.3.058.
Про затвердження Методичних рекомендацій щодо оцінювання гендерного впливу галузевих реформ :Наказ Міністерства соціальної політики України від 07.02.2020 № 86.
Wei J., Wang X., Schuurmans D. et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems. 2022. Vol. 35. P. 24824–24837.
Shojaee P., Mirzadeh I., Alizadeh K. et al. The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity. Apple, 2025. 23 p. DOI: 10.48550/arXiv.2506.06941
Khattab O. et al. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv preprint. 2024. DOI:10.48550/arXiv.2310.03714
Yang C., Wang X., Lu Y. et al. Large language models as optimizers. arXiv preprint. 2024. DOI:10.48550/arXiv.2309.03409
Hinton G., Vinyals O., Dean J. Distilling the knowledge in a neural network. arXiv preprint. 2015. DOI:10.48550/arXiv.1503.02531
Wu X., Xiao L., Sun Y., Zhang J., Ma T., He L. A survey of human-in-the-loop for machine learning. Future Generation Computer Systems. 2022. Vol. 135. DOI: 10.1016/j.future.2022.05.014
