DESIGNING LOSS FUNCTIONS FOR TRANSFORMER-BASED TIME ESTIMATION IN AI PROJECT

Abstract

Accurate time estimation is critical for planning and managing artificial intelligence (AI) projects. However, traditional approaches often fall short due to the domain-specific nature of the task, causing high variability and complexity in such projects. This study dives into how transformer-based models can be improved for project duration prediction by designing a custom loss function. A dataset containing structured project metadata – such as domain, stage, and difficulty – was used to train a transformer model using multiple loss functions. These include standard Mean Squared Error (MSE), a domain-weighted loss with custom metadata mapping, and an uncertainty-aware loss. The final design of the loss function integrates all benefits of having a domain knowledge baseline and the structure of the uncertaintybased loss model. Evaluation results demonstrate that this combined approach outperforms standard loss functions based on MAE, MSE, and RMSE metrics. This study addresses the challenge of improving time estimation accuracy in artificial intelligence (AI) projects by focusing on designing custom loss functions for transformer-based regression models. Traditional estimation approaches often fail to capture the complexity and variability inherent in AI development processes, where project duration is influenced by factors such as high differences in domains and sudomains, targeted development stages, or just project difficulty. To overcome these limitations, this research looks into how loss function design can be used for better model performance. A structured dataset of AI projects with metadata-based features like primary domain, additional domain, project stage, and difficulty score. Based on this dataset, a transformer-based model was trained and evaluated using multiple loss function strategies. These include the standard mean squared error (MSE), a domain-weighted MSE that uses and relies on expert-driven metadata mappings, an uncertainty-aware loss function, and a final stabilised loss that combines domain weighting with probabilistic constraints. Experimental results demonstrate that the final design of a custom loss function outperforms traditional approaches across standard accuracy metrics, including mean absolute error (MAE), mean squared error (MSE), and root mean squared error (RMSE). The findings confirm that incorporating domain knowledge and uncertainty modeling directly into the loss function leads to more accurate, stable, and interpretable predictions. Overall, this study highlights the importance of loss function design in applied machine learning tasks and sets a framework for improving transformer-based time estimation models in AI project management contexts.