AI-Enabled MLOps Framework for Enterprise Machine Learning on Modern Lakehouse Architectures

Abstract

With increasing data volume and frequent implementation of machine learning in enterprises, the need for scalable infrastructure and compliance arises. The traditional MLOps pipelines run independently of the storage infrastructure and cause inefficiency, lack of governance, and high costs of operations. This paper introduces the concept of the AI-enabled MLOps pipeline which can be used together with the existing lakehouse infrastructure to solve the problems in traditional MLOps pipelines. Through the use of the Design Science Research Methodology, the prototype of the framework which involves the stages of data ingestion, model training, deployment, and monitoring layers using artificial intelligence for automated orchestration was designed. The performance testing of the framework was conducted with simulated enterprise workloads in comparison with the traditional MLOps pipelines without lakehouse infrastructure. Improvement in results was achieved with 42% less deployment latency, 89% accuracy of data drift detection, 35% higher resource utilisation, and 28% cost reduction. In relation to practitioner evaluation, there is great user satisfaction, whereby 82% of the participants have provided positive feedback regarding the practicality of using the application. In conclusion, in relation to all the above findings, it is evident that the integration of AI automation in the lakehouse architecture has been very practical, in terms of technology, economics, and human aspects. This paper, therefore, provides an empirical basis of a unified framework solving the fragmented frameworks in existing literature.