Do you want to know how we change everything?
Schedule a call and we can show you how to boost your business.
Menu
Menu
Siegfried Eckstedt
At the “Mind Mastering Machines 2023” conference in Germany, I met with IT experts from various companies to discuss the latest trends and challenges in the field. In my session on continuous integration and deployment (CI/CD) for machine learning, I shared insights based on extensive research I conducted prior to the conference.
During the conference, there were numerous engaging discussions among experts on a wide range of topics, including the recent advancements in natural language processing. Particularly intriguing was the discourse surrounding the new Language Learning Model (LLM) developed by ChatGPT. This state-of-the-art model showcased remarkable capabilities in understanding and generating human-like text, opening up new possibilities for language-related applications in various domains.
While my session focused on CI/CD for machine learning, the discussions on the new LLM from ChatGPT caught my attention. It was fascinating to witness the enthusiasm and curiosity surrounding the potential applications and implications of this cutting-edge technology. With the upcoming trends, the question to ask is, “What does CI/CD for Machine Learning mean for LLMs in production?”
Running a LLM in production should be done in line with the CI/CD best practices, e.g. implementing automated testing, continuous delivery, and performance monitoring. The technical enhancements of a CI/CD pipeline will surely boost the success of any models, including LLMs. But firstly, we need to understand the common challenges the industry faces when implementing CI/CD for the more traditional ML applications.
Before the event, I delved into the realm of CI/CD for machine learning by collecting and analysing 20 comprehensive cases. These cases collectively represented and exemplified different aspects of practising CI/CD for ML applications. The findings from this research shed light on the challenges faced by organisations and highlighted key areas that needed attention when implementing CI/CD in the context of machine learning.
Testing and model governance emerged as critical areas, with nearly all the cases indicating the importance of addressing these challenges. Additionally, team collaboration and managing experiments were identified as key factors in successful CI/CD implementation.
By understanding and addressing these challenges, organisations can navigate the complexities of implementing CI/CD for machine learning more effectively. It is essential to establish robust testing procedures, prioritise model governance, foster collaborative team dynamics, and develop efficient experiment management frameworks.
While there are many approaches to tackle the wide set of challenges, I want to highlight four areas to focus on when practising CI/CD for Machine Learning:
Schedule a call and we can show you how to boost your business.
When focusing on LLMs, specific challenges arise in implementing the CI/CD pipeline. Testing, in particular, can be vexing as it should include verification of the model itself. Evaluating the text output requires reading, comprehending, fact-checking, and assessing formatting and punctuation. And evaluation processes can lead to subjective results. Additionally, depending on the availability of the dataset, testing LLMs can be time-consuming due to the large, contextual, and diverse nature of the data.
Ideally, the entire testing checklist should be covered by the automated testing system. However, best practices are still evolving, and further experimentation is required. One approach is to use a second, trustworthy LLM for testing the performance of a newly developed version of your LLM.
In conclusion, the “Mind Mastering Machines 2023” conference provided a great platform for insightful discussions on various topics in the field of data and AI. Alongside my session on CI/CD for machine learning, the exploration of the new LLM from ChatGPT demonstrated the rapid advancements being made in natural language processing. By leveraging these advancements and addressing the challenges of CI/CD, we can pave the way for accelerated development and deployment of reliable machine learning applications.
Schedule a call and we can show you how we can boost your business.
The power to change anything.
This website uses cookies to ensure you get the best experience.
Many subscribers already enjoy our premium stuff. Subscribe now.
