M365.FM - Modern work, security, and productivity with Microsoft 365

Fabric Rewrote Data Engineering


Listen Later

Microsoft Fabric didn’t make data engineering easier.
It made ambiguity cheaper to ship. This episode explains why teams feel faster and more out of control at the same time after adopting Fabric and Copilot—and why that isn’t a tooling problem. Fabric removed the ceremony that used to slow bad decisions down. Copilot removed the typing, not the consequences. The result is architectural erosion that shows up first as cost spikes, conflicting dashboards, and audit discomfort—not broken pipelines. If your Fabric estate “works” but feels fragile, this episode explains why. What You’ll Learn 1. What Fabric Actually Changed (and What It Didn’t) Fabric didn’t rewrite data engineering because of better UI or nicer tools. It rewrote it by collapsing:
  • Storage
  • Compute
  • Semantics
  • Publishing
  • Identity
into a single SaaS control plane. This removed handoffs that used to force architectural decisions—and replaced them with lateral movement inside workspaces. Fabric removed friction, not responsibility. 2. Why Speed Accelerates Drift, Not Simplicity In older stacks, ambiguity paid a tax:
  • Environment boundaries
  • Tool handoffs
  • Deployment friction
  • Separate billing surfaces
Those boundaries slowed bad decisions down. Fabric removes them. Drift now ships at refresh speed. The result isn’t failure—it’s quiet wrongness:
  • Dashboards refresh on time and disagree
  • Pipelines succeed while semantics fragment
  • Capacity spikes without deployments
  • Audits surface ownership gaps no one noticed forming
3. The New Failure Signal: Cost, Not Outages Fabric estates don’t usually fail loudly.
They fail expensively. Because all workloads draw from a shared capacity meter:
  • Bad query shapes
  • Unbounded filters
  • Copilot-generated SQL
  • Refresh concurrency
surface first as capacity saturation, not broken jobs. Execution plans—not dashboards—become the only honest artifact. 4. Copilot’s Real Impact: Completion Over Consequence Copilot optimizes for:
  • Plausible output
  • Fast completion
  • Syntax correctness
It does not optimize for:
  • Deterministic cost
  • Schema contracts
  • Security intent
  • Long-term correctness
Without enforced boundaries, Copilot doesn’t break governance—it accelerates its absence. Teams with enforcement get faster.
Teams without enforcement get faster at shipping entropy. 5. Why Raw Tables Become a Cost and Security Liability When raw tables are queryable:
  • Cost becomes probabilistic
  • Schema drift becomes accepted behavior
  • Access intent collapses into workspace roles
  • Copilot becomes a blast-radius multiplier
Fabric exposes the uncomfortable truth:
Raw tables are not a consumption API. 6. Case Study: The “Haunted” Capacity Spike A common Fabric incident pattern:
  • No deployments
  • No pipeline failures
  • Dashboards still load
  • Capacity spikes mid-day
Root cause:
  • Non-sargable predicates
  • Missing time bounds
  • SELECT *
  • Copilot-generated SQL under concurrency
Fix:
  • Views and procedures as the only query surface
  • Execution plans as acceptance criteria
  • Cost treated as an engineered property
7. Lakehouse → Warehouse Contract Collapse Lakehouses are permissive by design.
Warehouses are expected to enforce structure—but they can’t enforce contracts that never existed. Without explicit schema enforcement:
  • Drift moves downstream
  • Semantic models become patch bays
  • KPIs fork silently
  • “Power BI is wrong” becomes a recurring sentence
The Warehouse must be the contract zone, not another reflection layer. 8. Why Workspace-Only Security Creates an Ownership Vacuum Workspaces are collaboration boundaries—not data security boundaries. When organizations rely on workspace roles:
  • Nobody owns table-level intent
  • Service principals gain broad access
  • Audit questions stall
  • Copilot accelerates unintended exposure
The fix isn’t labels or training.
It’s engine-level enforcement: schemas, roles, views, and deny-by-default access. 9. The Modern Data Engineer’s Job Didn’t Shrink—it Moved Fabric shrinks visible labor:
  • Pipeline scaffolding
  • Glue code
  • Manual SQL authoring
But it expands responsibility:
  • Contract design
  • Boundary enforcement
  • Cost governance
  • Failure-mode anticipation
The modern data engineer enforces intent, not just movement. 10. The Only Operating Model That Survives Fabric + Copilot This episode outlines a survivable operating model:
  • AI drafts, humans approve
  • Contracts before convenience
  • Execution plans as cost policy
  • Views over raw tables
  • CI/CD gates for schema and logic
  • Assume decay unless enforced
Governance must be mechanical—not social. Core Takeaway Fabric is a speed multiplier. It multiplies:
  • Delivery velocity
  • Ambiguity
  • Governance debt
at the same rate. The platform doesn’t break.
Your assumptions do. Call to Action Ask yourself one question:
When something feels wrong, which artifact do you trust?
  • Execution plan
  • Capacity metrics
  • Violation count
  • Lineage view
Whatever you answered—that’s what your governance model is actually built on. Subscribe for the next episode:
“How to Design Fabric Data Contracts That Survive Copilot.”

Become a supporter of this podcast: https://www.spreaker.com/podcast/m365-fm-modern-work-security-and-productivity-with-microsoft-365--6704921/support.
...more
View all episodesView all episodes
Download on the App Store

M365.FM - Modern work, security, and productivity with Microsoft 365By Mirko Peters (Microsoft 365 consultant and trainer)