the Power to change anything.

Mastering CI/CD for Machine Learning: Challenges, Solutions, and Insights from Mind Mastering Machines 2023

At the “Mind Mastering Machines 2023” conference in Germany, I met with IT experts from various companies to discuss the latest trends and challenges in the field. In my session on continuous integration and deployment (CI/CD) for machine learning, I shared insights based on extensive research I conducted prior to the conference.

During the conference, there were numerous engaging discussions among experts on a wide range of topics, including the recent advancements in natural language processing. Particularly intriguing was the discourse surrounding the new Language Learning Model (LLM) developed by ChatGPT. This state-of-the-art model showcased remarkable capabilities in understanding and generating human-like text, opening up new possibilities for language-related applications in various domains.


Siegfried Eckstedt Aiku at Minds Mastering Machines 2023 Kalrsuhe
Minds Mastering Machines 2023, Kalrsuhe, Germany
Photo Credits to Minds Mastering Machines Conference, Heisse


While my session focused on CI/CD for machine learning, the discussions on the new LLM from ChatGPT caught my attention. It was fascinating to witness the enthusiasm and curiosity surrounding the potential applications and implications of this cutting-edge technology. With the upcoming trends, the question to ask is, “What does CI/CD for Machine Learning mean for LLMs in production?”

Running a LLM in production should be done in line with the CI/CD best practices, e.g. implementing automated testing, continuous delivery, and performance monitoring. The technical enhancements of a CI/CD pipeline will surely boost the success of any models, including LLMs. But firstly, we need to understand the common challenges the industry faces when implementing CI/CD for the more traditional ML applications.

Before the event, I delved into the realm of CI/CD for machine learning by collecting and analysing 20 comprehensive cases. These cases collectively represented and exemplified different aspects of practising CI/CD for ML applications. The findings from this research shed light on the challenges faced by organisations and highlighted key areas that needed attention when implementing CI/CD in the context of machine learning.

Siegfried Eckstedt Aiku at Minds Mastering Machines 2023 Kalrsuhe_Research Cases

Siegfried Eckstedt Aiku at Minds Mastering Machines 2023 Kalrsuhe_Research Result

Testing and model governance emerged as critical areas, with nearly all the cases indicating the importance of addressing these challenges. Additionally, team collaboration and managing experiments were identified as key factors in successful CI/CD implementation.

By understanding and addressing these challenges, organisations can navigate the complexities of implementing CI/CD for machine learning more effectively. It is essential to establish robust testing procedures, prioritise model governance, foster collaborative team dynamics, and develop efficient experiment management frameworks.

While there are many approaches to tackle the wide set of challenges, I want to highlight four areas to focus on when practising CI/CD for Machine Learning:


Automation Blueprint
  • Make use of an end-to-end workflow architecture that enables automation and triggers retraining when necessary.
  • Draw from industry-leading blueprints that have proven good practice when handling ML operations, ensuring synchronisation and execution efficiency (example).


Pipeline Orchestration
  • Create environment symmetry between experimentation pipelines and production pipelines.
  • Set up both experimental and production pipelines with aligned steps to ensure smooth synchronisation and process optimisation.

Do you want to know how we change everything?

Schedule a call and we can show you how to boost your business.

Automated Testing
  • Develop and implement additional tests for ML applications, including data tests, model tests, and ML infrastructure tests.
  • Automate testing processes to get more feedback, iterate faster, and ensure reliable performance.


Automated Training
  • Regularly retrain models to avoid model staleness and accommodate changes in the real-world data.
  • Align retraining with serving strategy, consider costs, and keep stakeholders informed.


When focusing on LLMs, specific challenges arise in implementing the CI/CD pipeline. Testing, in particular, can be vexing as it should include verification of the model itself. Evaluating the text output requires reading, comprehending, fact-checking, and assessing formatting and punctuation. And evaluation processes can lead to subjective results. Additionally, depending on the availability of the dataset, testing LLMs can be time-consuming due to the large, contextual, and diverse nature of the data.

Ideally, the entire testing checklist should be covered by the automated testing system. However, best practices are still evolving, and further experimentation is required. One approach is to use a second, trustworthy LLM for testing the performance of a newly developed version of your LLM.

In conclusion, the “Mind Mastering Machines 2023” conference provided a great platform for insightful discussions on various topics in the field of data and AI. Alongside my session on CI/CD for machine learning, the exploration of the new LLM from ChatGPT demonstrated the rapid advancements being made in natural language processing. By leveraging these advancements and addressing the challenges of CI/CD, we can pave the way for accelerated development and deployment of reliable machine learning applications.

Let’s get to know each other.

Schedule a call and we can show you how we can boost your business.

This website uses cookies to ensure you get the best experience.


Many subscribers already enjoy our premium stuff. Subscribe now.