Data pipeline orchestration sits at the intersection of two hard problems: scheduling and dependency management. You have tasks that must run in a specific order, on a schedule, with retries for failures, and visibility into what ran, when, and why it failed. Airflow, Prefect, and Dagster all solve this problem—but with significantly different philosophies about what a pipeline orchestrator should be. For event-driven and long-running workflow orchestration outside the data engineering world, see Temporal vs AWS Step Functions.
Apache Airflow
Airflow is the incumbent. Released by Airbnb in 2014, open-sourced in 2015, and now a top-level Apache project with 200+ providers covering AWS, GCP, Azure, Databricks, dbt, Snowflake, and most data tools your organization uses. If you need an integration, Airflow probably has one.
The DAG model: Airflow pipelines are Python files that define a Directed Acyclic Graph (DAG) of tasks and their dependencies. DAGs are discovered by Airflow’s scheduler by scanning a DAG folder. Operators define what each task does: PythonOperator, BashOperator, SparkSubmitOperator, and hundreds of provider-specific operators.
What Airflow does well:
- Largest provider ecosystem by far
- Mature UI for monitoring DAG runs, task logs, and retries
- Extensive scheduling options (cron, data-interval-aware, external triggers)
- Production-tested at massive scale (Airbnb, Lyft, Twitter)
- Managed options: MWAA (AWS), Cloud Composer (GCP), Astronomer
For the underlying messaging infrastructure that often feeds Airflow pipelines, see Kafka Streams vs Flink vs Spark Streaming and AWS SQS vs Kafka.
Airflow’s persistent frustrations:
- DAG authoring is slow: Python files must be importable by the scheduler—global code in DAG files runs at parse time. Dynamic task generation, templating, and parameterization require careful patterns
- Testing is painful: Testing a DAG requires either spinning up an Airflow instance or mocking extensively. Unit testing business logic is possible but awkward when it’s wrapped in operators
- Backfills and reruns: The execution date model (tasks run for a specific data interval) is powerful but confusing to new users. Debugging why a task ran for the wrong date is a rite of passage
- Operational overhead: Running Airflow yourself means managing the scheduler, webserver, workers, and metadata database. MWAA/Astronomer reduce this significantly
Airflow is the right choice for teams with diverse integration requirements, a data engineering team comfortable with its patterns, and the operational capacity to run it—or budget for a managed service.
Prefect
Prefect emerged as an explicit Airflow alternative, founded by the author of a well-known critique called “The Prefect Manifesto.” Its design choices reflect lessons learned from Airflow’s limitations.
The flow model: Prefect pipelines are Python functions decorated with @flow and @task. There’s no DAG file, no special folder, no parse-time execution. A flow is just a Python function that calls task functions. Dependencies are inferred from how tasks call each other.
@task
def extract(url: str) -> dict:
return requests.get(url).json()
@task
def load(data: dict):
db.insert(data)
@flow
def etl_pipeline(url: str):
data = extract(url)
load(data)
What Prefect does well:
- Developer experience: Flows are just Python. Testing is just testing Python functions. No framework-specific testing patterns required.
- Dynamic task generation: Creating tasks dynamically based on runtime data is first-class, not a workaround
- Fast iteration: Run a flow locally, deploy it, run it in production—the same code everywhere
- Prefect Cloud: The managed control plane provides scheduling, a UI, and observability with minimal infrastructure. Workers run anywhere (local, Docker, Kubernetes, serverless)
Prefect’s trade-offs:
- Provider ecosystem is smaller than Airflow’s, though growing
- The Prefect 1.x to 2.x migration broke many organizations—version stability has been a concern
- Less mature than Airflow for complex scheduling scenarios
- Prefect Cloud (SaaS) is the easiest path; self-hosting the server adds operational complexity
Prefect is the right choice for teams prioritizing developer experience, rapid iteration, and Python-native workflows—especially teams building data pipelines alongside application engineers who want to use standard Python tooling.
Dagster
Dagster takes the most opinionated approach of the three. Its central abstraction is not a task or a flow—it’s a software-defined asset: a piece of data (a table, a file, a model) that your pipeline produces. Pipelines in Dagster are defined by the assets they produce and their dependencies on upstream assets, not by the execution steps.
@asset
def raw_orders(context) -> pd.DataFrame:
return pd.read_csv("s3://bucket/orders.csv")
@asset
def cleaned_orders(raw_orders: pd.DataFrame) -> pd.DataFrame:
return raw_orders.dropna()
@asset
def order_summary(cleaned_orders: pd.DataFrame) -> pd.DataFrame:
return cleaned_orders.groupby("customer_id").agg({"amount": "sum"})
Dagster infers the pipeline structure from asset dependencies. The UI shows a lineage graph of assets and their upstream/downstream relationships—which data assets exist, which are up to date, which are stale.
What Dagster does well:
- Asset-centric model: Teams that think in terms of “what data do we produce?” rather than “what tasks do we run?” find this model natural
- Data lineage: Built-in lineage graph across all assets; visibility into what produced each piece of data
- Type system: Assets have declared types; Dagster validates inputs and outputs
- Testing: Assets are functions; testing is straightforward. IO managers abstract storage so tests run against local data
- dbt integration: Dagster’s dbt integration is best-in-class—dbt models become Dagster assets automatically
Dagster’s trade-offs:
- Steeper learning curve than Prefect; requires buying into the asset model
- Smaller provider ecosystem than Airflow (though growing fast)
- The asset abstraction requires rethinking how you structure pipelines; teams with established Airflow or Prefect patterns face a non-trivial mental model shift
- Dagster Cloud (managed) is the easiest operational path; self-hosting requires running the Dagster daemon and webserver
Dagster is the right choice for teams building data platforms where data lineage, asset cataloging, and data quality are first-class concerns—particularly teams working heavily with dbt and wanting a unified orchestration layer.
The Decision Framework
Choose Airflow if:
- You have an existing Airflow investment and migration cost is too high
- You need maximum provider coverage and ecosystem maturity
- Your team is familiar with Airflow’s DAG model
- You have a managed Airflow service (Astronomer, MWAA) handling operations
Choose Prefect if:
- Developer experience is the priority—your data engineers want to write Python, not framework-specific patterns
- You’re starting fresh and want fast iteration
- Dynamic workflows (runtime-generated task graphs) are important
- Your team has limited ops capacity and Prefect Cloud’s managed control plane is valuable
Choose Dagster if:
- Data lineage and asset visibility are important to your team’s workflow
- You’re building a data platform where understanding “what data do we have and is it fresh?” matters
- Your dbt models are central to your pipeline and you want integrated orchestration
- You want the strongest data testing and validation model
What All Three Share
All three tools have converged on several features that were differentiators a few years ago: sensor-based triggering, cross-pipeline dependencies, dynamic task generation, managed cloud options, and Python-native definitions. The differences that remain are in default abstractions, maturity of specific integrations, and operational model.
The tool you can actually get your team to adopt consistently and maintain over time is the right one. A half-adopted Dagster migration is worse than a well-maintained Airflow installation.