AI5Production – Test Before Invest Project with Objentis
Exploring LLMs to Support Test Processes
OBJENTIS Software Integration GmbH teamed up with SBA to explore the potential of Large Language Models (LLMs) for translating natural language test cases into executable languages. Motivated by data protection requirements, the focus was placed on open models and self-hosting solutions.
The evaluation of various LLMs (e.g., Llama 2, Mistral, Vicuna) in a test scenario demonstrated that GPT-4 currently outperforms the others, though Llama 2 70B comes close to the baseline. Despite having fewer parameters, Mistral 7B also delivered strong results. Fine-tuning and prompt engineering were investigated as methods to improve performance.
© SBA/Objentis
What is a “Test before Invest”?
A “Test before Invest” is a small-scale project focused on digitalization within a company. These projects typically range in budget from €30,000 to €40,000 and are carried out in collaboration with an AI5production partner. The costs are fully covered by the European Commission and the FFG (Austrian Research Promotion Agency). AI5Production handles the preparation and submission of the “Test before Invest” proposal.
AI5production offers free access to the infrastructure and expertise of our partners, which include pilot factories, competence centers, research institutes, universities, and industrial partners. Experts from these partner institutions cover nearly all aspects of digitalization in manufacturing companies. Together with the respective company, challenges are addressed, and tailored solutions are developed.
Are you an SME or a mid-cap company with up to 2,999 employees? If so, you can receive free support for your digital transformation! For further information please contact Andreas Ekelhart.
Further information
Andreas Ekelhart – Management Board, key researcher and Applied Research Lead at SBA Research
EDIH AI5production
Test before Invest