Apache Airflow in Fabric: When Code-First Orchestration Earns Its Keep
When should you reach for Apache Airflow Job instead of Data Pipelines in Microsoft Fabric? Matthias and Fabia break down pool types, cost traps, dbt fragility, and the architectural threshold where visual orchestration breaks down.
A real-world mistake from a pre-Fabric eraThe one question that reframes the architectural debateHow we got here — predecessor products and evolutionWhy the "obvious" answer is often wrongA real Reddit/Microsoft Q&A question unpackedThe concrete recommended architectureF-SKU realism — what this actually costsWhen the rejected approach is actually rightRisks of the recommended pathWhat Microsoft is shipping that changes the calculusThe architectural principle to take homeI'm stealing that. But yeah — match the orchestrator to the orchestration complexity. If your workflow fits on a visual canvas, keep it there. Airflow is for when Python is genuinely the clearest way to express your pipeline logic.For maybe... sixty, seventy percent of Fabric workloads, Data Pipeline is the right answer. Airflow earns its spot when you need cross-cloud orchestration, dbt models, complex dependency graphs, or your team already thinks in Python DAGs....You just paid for a permanent orchestra conductor to wave a baton at six musicians who already know the song.What is Apache Airflow Job?Run a Fabric item using Apache Airflow DAGsApache Airflow compute in FabricApache Airflow job pricingTransform data using dbtMigrate to Apache Airflow job in Microsoft FabricCI/CD for Apache Airflow in FabricApache Airflow Job region availabilityQuickstart: Create an Apache Airflow JobApache Airflow Job workspace settingsAccess Apache Airflow Job LogsSync a GitHub repository in Apache Airflow JobRun Hello-world DAG in Apache Airflow JobFabric region availabilityMS LearnBuilt on ElevenLabs voice synthesis. Matthias — cloned voice. Fabia — designed AI co-host. See Matthias live on YouTube (Fabric Friday), at his meetups, and at conferences like FabCon.
Hosted by Matthias Falland — Microsoft Data Platform MVP and community architect behind the Fabric Periodic Table. New episodes every Friday.
Have an architecture decision you are wrestling with? DM Matthias on LinkedIn — find him as Matthias Falland. Three to five sentences about the decision, your team size, and your current stack. We anonymize before airing.
Built on ElevenLabs voice synthesis. Brand design based on fabricperiodictable.com.