What are the main stages of an end-to-end MLOps workflow?

An end-to-end MLOps workflow describes how a machine learning idea becomes a working application and how that application is maintained after deployment.

Think of it as a journey: we start with a problem, build a model to solve it, make it available to users, and keep improving it as new data arrives.

Let's understand with an example

Imagine a bank wants to predict whether a customer might default on a loan.

Building a prediction model is only part of the solution. The bank must also ensure that the data is reliable, the model performs well, predictions reach the right application, and problems are detected after deployment.

An MLOps workflow helps manage this entire journey.

The 8 main stages

1. Define the problem: Decide what the model should predict and how success will be measured. For example, the bank wants to identify potentially risky loans while keeping incorrect predictions under control.
2. Collect and validate data: Gather relevant historical information and check for missing values, incorrect formats, and other data quality problems.
3. Prepare the data: Clean and transform the data into a form the model can use, while keeping training and evaluation data properly separated.
4. Train and track experiments: Train candidate models and record their settings, results, and versions so the team can understand what worked.
5. Evaluate and approve the model: Check performance on suitable evaluation data, compare it with a baseline, and verify that it meets the required standards before release.
6. Register and deploy the model: Store the approved model with its version information and make it available for predictions, perhaps through an API used by the bank's application.
7. Monitor the production system: Watch for prediction-quality issues, unusual input data, errors, and slow responses. A running application doesn't always mean its model is performing well.
8. Improve and retrain when necessary: Investigate problems, use appropriate new data to train another model, and repeat the evaluation and release process.

How do these stages connect?

Define the problem → Collect and prepare data → Train and evaluate → Register and deploy → Monitor and improve → Repeat when needed.

Does every stage need to be automated?

Not necessarily. Teams often begin with manual steps and gradually automate repetitive tasks such as data validation, training, testing, and deployment.

Also, retraining doesn't automatically mean releasing a new model. The new version must pass the necessary checks before it replaces the existing one.

In short: MLOps is not a one-time process that ends when a model goes live. It's a continuous cycle of developing, deploying, monitoring, and improving machine learning systems.