Abstract

This study investigates the application of Large Language Models (LLMs) for structured information extraction in the context of simulation modelling. Specifically, the performance of Meta-Llama-3-70B-Instruct was evaluated in extracting simulation model components — such as sources, process steps, and parameters — from unstructured textual descriptions of logistics systems. The evaluation compared zero-shot and few-shot prompting strategies and analysed model consistency across 50 repeated runs per prompt configuration. Metrics such as precision, recall, and F1-score were computed using fuzzy string matching. Results show that few-shot prompting improves extraction performance overall, with statistically significant gains in categories such as model parameters and sources.  
However, inconsistent effects were observed for other categories, including a decrease in key metrics identification. Variability analysis using the coefficient of variation revealed differing levels of output stability across categories and prompts.  
The findings indicate that few-shot prompting with care-fully chosen examples enhances model reliability but may introduce risks of hallucinated outputs and prompt-specific overfitting. Recommendations are provided for prompt engineering in simulation modelling applications. Future work will consider other LLM architectures, dynamic prompt selection, and integration of external validation mechanisms.