Implementation of an AI Support Chatbot Based on Microsoft Azure OpenAI with Special Consideration of Quality

EPiC series in computing, vol. 107, pp. 237–226

Abstract

This article describes the development and evaluation of an AI powered chatbot that was developed specifically to improve IT support at the IT-ServiceDesk (IT-SD) within the IT Center of RWTH Aachen University. The implementation is carried out using Microsoft Azure OpenAI and a Retrieval-Augmented Generation (RAG) approach. Given the varying complexity of support requests among different customer groups—namely, students, employees, and IT administrators—the AI chatbot will be a valuable supplement to established support channels such as email, telephone, chat, ticket portals, and personal interaction. The article describes the key requirements for the chatbot and explains how the quality of the response is ensured by a structured feedback system. It further addresses the challenges that arise from the fact that there are two different scenarios with different requirements. On one hand, standardized and publicly available RWTH support content is provided for RWTH members as well as interested parties of the IT services and, on the other, specific internal support content for the supporters at the IT-SD. This necessitates that specific measurable quality requirements and criteria must be identified implemented and adapted.

Authors 3

1. Bernd Decker Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University , Germany

2. Sarah Grzemski Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University , Germany

3. Ingo Hengstebeck Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University , Germany