Airflow is the most popular solution, followed by Luigi. There are newer contenders too, and they’re all growing fast. (source)
Task orchestration tools and workflows
Recently there’s been an explosion of new tools for orchestrating task- and data workflows (sometimes referred to as “MLOps”). The quantity of these tools can make it hard to choose which ones to use and to understand how they overlap, so we decided to compare some of the most popular ones head to head.
Overall Apache Airflow is both the most popular tool and also the one with the broadest range of features, but Luigi is a similar tool that’s simpler to get started with. Argo is the one teams often turn to when they’re already using Kubernetes, and Kubeflow and MLFlow serve more niche requirements related to deploying machine learning models and tracking experiments.
Before we dive into a detailed comparison, it’s useful to understand some broader concepts related to task orchestration.
What is task orchestration and why is it useful?
Smaller teams usually start out by managing tasks manually – such as cleaning data, training machine learning models, tracking results, and deploying the models to a production server. As the size of the team and the solution grows, so does the number of repetitive steps. It also becomes more important that these tasks are executed reliably.
The complex ways these tasks depend on each other also increases. When you start out, you might have a pipeline of tasks that needs to be run once a week, or once a month. These tasks need to be run in a specific order. As you grow, this pipeline becomes a network with dynamic branches. In certain cases, some tasks set off other tasks, and these might depend on several other tasks running first.
This network can be modelled as a DAG – a Directed Acyclic Graph, which models each task and the dependencies between them.
A pipeline is a limited DAG where each task has one upstream and one downstream dependency at most.
Workflow orchestration tools allow you to define DAGs by specifying all of your tasks and how they depend on each other. The tool then executes these tasks on schedule, in the correct order, retrying any that fail before running the next ones. It also monitors the progress and notifies your team when failures happen.
CI/CD tools such as Jenkins are commonly used to automatically test and deploy code, and there is a strong parallel between these tools and task orchestration tools – but there are important distinctions too. Even though in theory you can use these CI/CD tools to orchestrate dynamic, interlinked tasks, at a certain level of complexity you’ll find it easier to use more general tools like Apache Airflow instead.
Overall, the focus of any orchestration tool is ensuring centralized, repeatable, reproducible, and efficient workflows: a virtual command center for all of your automated tasks. With that context in mind, let’s see how some of the most popular workflow tools stack up.
Just tell me which one to use
You should probably use:
Apache Airflowif you want the most full-featured, mature tool and you can dedicate time to learning how it works, setting it up, and maintaining it.
Luigiif you need something with an easier learning curve than Airflow. It has fewer features, but it’s easier to get off the ground.
Prefectif you want something that’s very familiar to Python programmers and stays out of your way as much as possible.
Argo if you’re already deeply invested in the Kubernetes ecosystem and want to manage all of your tasks as pods, defining them in YAML instead of Python.
KubeFlowif you want to use Kubernetes but still define your tasks with Python instead of YAML. You can also read about our experiences using Kubeflow and why we decided to drop it for our projects at Kubeflow: Not ready for production?
MLFlow if you care more about tracking experiments or tracking and deploying models using MLFlow’s predefined patterns than about finding a tool that can adapt to your existing custom workflows.
Comparison table
For more Machine Learning Tips - Get our weekly newsletter
For a quick overview, we’ve compared the libraries when it comes to:
Maturity: based on the age of the project and the number of fixes and commits;
Popularity: based on adoption and GitHub stars;
Simplicity: based on ease of onboarding and adoption;
Breadth: based on how specialized vs. how adaptable each project is;
Language: based on the primary way you interact with the tool.
These are not rigorous or scientific benchmarks, but they’re intended to give you a quick overview of how the tools overlap and how they differ from each other. For more details, see the head-to-head comparison below.