Companies are increasingly adopting machine learning (ML) to solve real-world problems. With this comes the challenge of putting machine learning systems into production. For a machine learning model to be deployable, factors such as configuration, automation, feature engineering, server infrastructure, testing, monitoring and process management have to be addressed.
In conventional software engineering, DevOps bridges development and operations seamlessly. Similarly, in machine learning, MLOps can be applied to support rapid deployments in production by enhancing better collaboration of teams and solid integration of machine learning tasks.
MLOps can either be manual or automated. Choosing the right strategy depends on the needs of your company and the output that you would want to generate.
Manual MLOps
With manual MLOps, all activities are manually implemented and no automation is embedded into the machine learning pipeline. Data procurement, data preprocessing, data validation, model training and model validation are all manually executed. The model is trained at the deployment site and the pre-trained model generated is pushed to production. Further, with this approach, no retraining or changes in the model implementation are expected in the near future.
Because every step has to be manually initiated and executed, manual MLOps can slow down the building and serving of a machine learning system and can consume a lot more time.
Moreover, data scientists who build the model might not be in sync with the engineers who serve (or deploy) the model. For instance, data scientists who construct the model could hand over the sandboxed machine learning system to the engineer who deploys it on the target system, leading to variations in the model’s performance. This could be due to changes in the data attributes or differences in the methodologies embraced to handle the machine learning system in development and production.
Additionally, a model’s release iterations aren’t tracked in manual MLOps and thus versioning doesn’t happen in an automated fashion, which would create discrepancies in the model’s reproducibility. No continuous integration and continuous delivery (CI/CD) are ensured as the model is generally assumed to be static where no frequent changes are introduced. Monitoring the performance of a model doesn’t happen all the time, which could lead to overlooking the model’s degradation statistics.
As a result, manual MLOps is a strategy best suitable for non-tech companies whose machine learning models remain static for a long period of time (say, a year or so). It’s a standard methodology used when machine learning is in the early stages.
And for all those companies that have had an experienced ML team and are planning to take up manual MLOps, proper monitoring along with regular training of a model would help in being on par with the changing dynamics concerning data and algorithms.
Automated MLOps
Automating an out-and-out ML pipeline constitutes automated MLOps. Data validation, the verification of data credibility, and model validation, a check on the model’s adaptability in the production environment, largely penetrate the machine learning pipeline owing to the ability to retrain a machine learning model in production.
In automated MLOps, data has to be properly validated to identify the skewness persisting in it. If the data is not in accordance with the data schema defined, the ML team needs to intervene. The model should also be trainable online using a fresh batch of data. It should be able to compare the current metrics with the previous metrics (baseline model) and choose the best model to be moved to production.
To retrain and test the model, a development environment has to be available in the production stage. Modularity has to be induced into the code for better sustainability of the machine learning system. Monitoring the machine learning system has to happen on both a micro and macro level.
Additional components include feature stores and metadata management. A feature store serves as a repository for feature-related data collected throughout the user’s interaction. The values can be reused for retraining and serving the model reliably. Metadata management is the storage of metadata pertaining to the model’s versions (or release iterations), the time consumed to build and run the pipeline, hyperparameters and statistics. Metadata helps in easier anomaly detection.
Altogether, when retraining of a machine learning model has to happen at any point in the production environment by performing data and model validation — and without manual intervention — automated MLOps should be your choice. Unless yours is a company where machine learning isn’t a significant contributing factor, you’re likely going to benefit more from automated MLOps.
However, if you’re a big tech company extremely reliant on machine learning, CI/CD orchestration in automated MLOps is your best bet. A huge server infrastructure alongside continual updates to the machine learning system should be compelling enough to adopt this strategy. Continuous integrations of the source code together with A/B testing and continuous deliveries to the production environment while verifying the system’s compatibility with the production infrastructure results in sophisticated feature engineering, model construction and validation.
A full-fledged machine learning pipeline isn’t just about training and evaluating a model — it involves deploying and retraining the model online as well. The deployment strategy you choose must rely on the costs you’re willing to incur, tenure of the product you’re planning to build, size of your firm, headcount, expertise and the productivity you would want to generate. Only the best MLOps strategy befitting your company’s needs can generate fruitful results.
