Let’s hone in on Model Operations (ModelOps), an emerging category of companies within the broader artificial intelligence/machine learning (AI/ML) market. As organizations deploy AI/ML models across their business, governance, monitoring and trust are crucial.
Trust becomes a key issue for organizations to gain enough confidence in the infrastructure to deploy models for business-critical use cases. While CSAA Insurance Group’s AI efforts are still nascent, its path to becoming more data-centric and AI-centric can be expedited by leveraging ModelOps alongside Machine Learning Ops (MLOps).
The missing piece
At the start of the AI/ML wave, the referenced summary of how models are built, regardless of the type or methods being used, was the CRISP-DM cross-industry standard process for data mining:
Noticeably missing from this process is what happens after deployment. If models are being built and deployed at a small scale without frequent updates, then the CRISP-DM process and summary are likely sufficient for the business’ needs. However, AI model deployment into production at increasing scale presents a different and more complex environment:
- New AI model development tools enable business analysts and domain experts rather than data scientists to generate hundreds or thousands of models.
- As the number of deployed models scale, those that must be updated often become unwieldy to manage and monitor.
- As AI model results and predictions become increasingly relied upon for critical business decisions or driving automated processes that may also be subject to regulatory scrutiny, the stakes for the accuracy, management and monitoring of those models rise significantly.
A deeper examination of organizations that have successfully scaled AI and are mature in their implementation and operationalization of AI surfaces a key component they’ve implemented that’s missing from the traditional CRISP-DM framework: ModelOps.
MLOps? ModelOps? What’s the difference?
Given the nascency of the ModelOps field, there are numerous variations on the definition and capabilities of ModelOps, primarily different flavors of the same concept. For this post, let’s define MLOps as the data science tools and processes that involve speeding up the development, testing and deployment of AI/ML models. ModelOps is the enterprise governance and operations layer for models in production and encircles the entire model-building process to provide model lifecycle automation. ModelOps applies monitoring, management and governance layers on top of MLOps.
Key Differences: MLOps vs. ModelOps
As outlined in the table above, ModelOps and MLOps go hand-in-hand. It’s necessary to have MLOps in place before bringing in ModelOps. The business need for ModelOps is low in most organizations today, as widespread production deployment of AI models is still nascent. However, companies deploying tens to hundreds of models, especially in highly regulated industries, increasingly purchase ModelOps software. Most organizations will operationalize or implement their AI stack in stages. And for many companies, ModelOps will be the last piece encompassing everything else, as depicted in the diagram below:
The tipping point of ModelOps
Organizations won’t experience the pain of lacking a ModelOps tool on day one in the AI maturation and implementation journey. When the AI governance pain is felt depends to an extent on the industry and the organization. However, ModelOps is currently gaining the most significant foothold in organizations further along in the AI maturation spectrum with business-critical AI/ML models in production. There’s also been some initial traction in organizations in highly regulated or technical industries that have started to scale up their deployed AI models (~10 models or more in place) while not massively scaling up their AI/ML teams. The pain becomes more apparent then, especially if there’s a desire to deploy models into critical functions/use cases. Avanta Ventures has spoken to executives at QBE Insurance, Fidelity and Bank of New York, who have grown their AI/ML deployments to the scale where ModelOps is necessary. Most organizations with mature AI systems using ModelOps have homegrown ModelOps solutions. But those solutions are generally Frankenstein-esque, hard to scale, and expensive to support as model adoption within the organization grows.
The ModelOps market is still emerging. Most organizations don’t derive enough value from AI today to justify the cost or aren’t mature enough to have a lack of model governance be a sufficient pain point for them to search for a ModelOps solution. One driver that will help accelerate the need for ModelOps is regulation and increased accountability of and auditing of AI/ML models, especially in the insurance industry. Gartner’s Hype Cycle for AI, below, identifies the different components of AI and their journey to maturation.
What it means for the future
Ultimately, ModelOps builds trust and reliability for organizations in their AI/ML models so they can deploy models at a greater scale, more quickly and in higher-value use cases. To truly become a data-first, AI-centric company, both the concept of ModelOps and ModelOps tools need to be central to an organization’s roadmap so they can confidently deploy models at a large scale, as well as manage, monitor and update.
