The argument over table formats has largely converged on Iceberg as the shared interchange format, and the expensive part of the decision moved somewhere else.
In May 2026, Databricks listed managed Iceberg tables in Unity Catalog as generally available. On May 26, Snowflake did the same for writes from outside engines into its own managed Iceberg tables. Microsoft had already shipped Iceberg interoperability between OneLake and Snowflake, Google had made the Iceberg REST catalog in its Lakehouse runtime catalog generally available in April, and AWS lets Iceberg clients such as Spark and PyIceberg read and write through the Glue Iceberg REST endpoint.
For years, platform evaluations circled one worry: where does the data live, and what does it cost to leave? On table formats, the honest answer is now “less than it used to”, and the debate has moved on without announcing it.
Where it moved matters if you are building AI applications. In partner conversations, I see platform debates start with formats and feature lists and end, a few meetings later, on three questions nobody put on the original scorecard: what is this agent allowed to see, who pays when it gets curious, and what does it mean by “revenue”? All three live in the layer above the tables. I think that is where the lock-in sits now.
The same stack, five platforms
Here is the stack compared on the same dimensions. Status labels such as Beta, preview and generally available are as of November 2026 and change often, so re-verify them before you rely on them. These are not equivalent products: some entries are complete platforms, others are architectural components, and AWS is assembled from several services.
| Dimension | Databricks | Snowflake | BigQuery | Fabric | AWS |
|---|---|---|---|---|---|
| AI and agent capabilities | Genie Agents, Agent Bricks, Unity AI Gateway | Cortex Agents, Snowflake Intelligence | Gemini in BigQuery, AI functions | Copilot, data agents | Bedrock models, AgentCore |
| Compute and billing model | DBUs | Credits per warehouse-second | TiB scanned or slot-hours | Capacity Units (shared pool) | Several meters, one per service |
| Catalog and metadata | Unity Catalog | Horizon Catalog | Lakehouse runtime catalog (formerly BigLake metastore) | OneLake catalog | Glue Data Catalog |
| Governance and access enforcement | Unity Catalog policies. Cross-engine policy for external Iceberg is in Beta | Roles, masking and row policies. Scan planning extends them to external engines | IAM and native fine-grained security. REST-catalog tables have no row or column security | Entra identity and OneLake security. Row and column security for third-party engines is in preview | Lake Formation. Finer enforcement applies to AWS-integrated engines |
| Storage and table formats | Delta tables, managed Iceberg | Snowflake-managed, managed Iceberg | Google-managed, Lakehouse Iceberg tables | OneLake (Delta, Parquet) | S3, S3 Tables |
| Open table interoperability | Managed Iceberg, foreign Iceberg and Delta with UniForm differ in external read and write support | External reads and writes for Snowflake-managed Iceberg tables | Iceberg REST catalog. v2 generally available, v3 in preview | Iceberg through metadata virtualization | Glue Iceberg REST endpoint |
| Typical starting hypothesis | ML and engineering-heavy teams that want open formats | SQL-first teams that value isolation | GCP estates that do not want to size clusters | Entra and Power BI estates | AWS estates that want to own each layer |
Databricks is the platform most clearly built for people who write code against data. Genie Agents and Agent Bricks sit on Unity Catalog, and with managed Iceberg now generally available it is a natural fit for teams that want open formats and heavy ML. Be precise about the table type, though. Managed Iceberg tables are owned and governed by Unity Catalog, foreign Iceberg tables are registered from an external catalog, and Delta tables with UniForm expose Iceberg-readable metadata. External read and write support differs across the three, so “Iceberg support” means little until you name the table type and the access path.
Snowflake is often the cleanest starting point for SQL-first organizations that value isolation. Separate warehouses give workload isolation and more granular compute attribution, although storage and shared service costs still have to be accounted for separately. Since May 26, external engines can write to its managed Iceberg tables through Horizon.
BigQuery suits teams that never want to size a cluster. Google’s published US rates start at $6.25 per TiB scanned on demand, with the first TiB each month free, and at $0.04 per slot-hour for capacity pricing. These are two different pricing models, not directly comparable, and rates vary by region and edition. Actual cost depends on bytes processed, partition pruning and caching. It is a simple model until an agent starts exploring.
Fabric extends a Microsoft estate naturally. Fabric data agents generate read-only queries, and an external orchestrator can still call them inside a larger workflow that has action-taking components. Their release status is in conflict: different versions of Microsoft’s own documentation describe them as preview and as generally available, while third-party coverage reports general availability, so confirm it before you plan around them. OneLake reaches Iceberg through metadata virtualization rather than native storage. That is an interoperability mechanism, not a replacement for native Delta storage, and the direction matters: Iceberg-compatible readers access Delta tables, and Fabric workloads read Iceberg tables, through different paths.
AWS is the odd one out, because it does not sell a single product that maps onto the others. Compare it as a composed architecture, with each service doing a different job. Amazon SageMaker Unified Studio is the workspace and development environment, not a query engine or a catalog. The Glue Data Catalog holds metadata, Lake Formation manages data permissions, S3 is object storage and S3 Tables is managed Iceberg table storage. Athena, EMR, Redshift and Glue ETL are the engines that query and process the data. For agents, Bedrock AgentCore provides runtime, memory, gateway, identity and observability as separate services. You are assembling a platform. If you want to choose each layer, that is freedom. If your platform team has to own every seam, it is a cost. Databricks and Snowflake both run on AWS and can attach to the same Glue and S3 Tables catalogs, with the read, write and policy limits discussed below, so many AWS shops are really choosing between the native stack and one of those two.
Agents are a different kind of tenant
Feature comparisons are written with human analysts in mind. Agents break systems in different places, and the pricing and governance documentation shows where. A human analyst runs a handful of queries and reads the results. A person tends to stop when satisfied; an agent keeps going until a limit stops it. An agent loops. Early in a task it explores: list the tables, sample some rows, try a join, back up, try another.
Start with the bill. On a scan-priced engine, exploration becomes a line item. A query that scans 10 TiB at BigQuery’s on-demand rate costs about $62.50 each time it runs, whether it returns a million rows or five hundred (an identical repeat can be served free from the query result cache, but an exploring agent rarely repeats itself exactly). On shared capacity the cost shows up as contention instead: an agent burst on a single Fabric capacity or a shared slot reservation can slow the dashboards your finance team is watching. Isolated compute, such as separate Snowflake or Databricks SQL warehouses, gives clean attribution, but only if someone sets it up that way. None of these models is wrong. Each needs its own guardrail, and the question for any vendor is whether you can cap, attribute and isolate what one agent spends, and how far a limit can be overshot. Budgets and capacity controls exist, but Databricks, for one, documents its budgets as near-real-time estimates rather than hard caps, so measure the overshoot instead of assuming a ceiling.
Then identity, where the stakes are highest. Vendors increasingly position their agents as operating within existing identity and governance controls, but the degree of enforcement varies by platform, integration and execution path. Some of the mechanisms are strong: Databricks documents that Unity AI Gateway runs MCP calls with the requesting user’s permissions, and Snowflake documents that row access and masking policies on underlying tables propagate to semantic views and apply to agent queries. Both describe the native path, and neither says anything about the other doors. Passing a user’s identity through an agent is necessary for delegated access. It does not by itself guarantee that the data layer enforces that user’s permissions. That depends on which identity makes the data request, whether the downstream service recognizes it, whether the access path supports the policy and whether the engine applies it. The question is whose permissions the agent runs under, and whether the policy still holds when it reaches the data through a different door. Data policy also answers only one question, which rows and columns an identity may read. An agent with correct permissions can still be steered by prompt injection, misuse a tool, or retry a failed action until it does damage, so test the execution boundary as well as the data boundary.
External Iceberg access shows how much the path matters, using one configuration as an example. Row filters and column masks defined on a table can reach an outside engine when the engine asks the catalog to plan the scan, and Snowflake documents that approach. An engine that does not request it receives table-level access through vended credentials, without row or column enforcement. There are two safe outcomes: the engine enforces the policy, or it refuses the request because it cannot. Handing an engine that cannot enforce a row filter unrestricted access as a fallback is the failure to design out. In practice that will often mean refusal today, because few engines request server-side scan planning, so tables with row or column policy are usually queried through the owning platform’s own engine. Databricks also bills that server-side filtering as serverless compute, so budget for it. The standard is catching up: Iceberg 1.12.0 (October 2026) adds read restrictions to the REST specification, but the reader has to apply them, so a standard is a contract that each catalog and engine still has to implement. Databricks cross-engine fine-grained policy is in Beta as of November 2026, and views are a separate case: external access to Unity Catalog views is generally available through the Unity REST API and in Beta through the Iceberg REST catalog. Databricks’ cross-engine documentation also notes that external engines can only read while fine-grained controls are enforced, so any writing principal has to be exempted from the policy, which is a design decision rather than a footnote. Other catalog and engine combinations differ, so test the exact path instead of assuming. Access control that depends on which door the agent walks through is a zero-trust finding, and you want to find it before an auditor does.
AWS approaches the same problem with Lake Formation vending scoped, temporary credentials through the Glue endpoint. Outside engines receive them through a role with full table access, so the finer row and column enforcement applies only to AWS-integrated engines. For an outside engine the outcome is the same table-level access described above. AgentCore Identity lets an agent act for a user or under its own identity, and it does not by itself enforce downstream data policy.
Context is another form of lock-in
Last is context. An agent is only as right as its definitions: what counts as an active customer, which revenue figure is net. Databricks, Snowflake and Microsoft now each keep those definitions in their own constructs, from Genie Ontology to semantic views to Fabric IQ and Power BI semantic models. On BigQuery and AWS the definitions usually live in a BI tool such as Looker, in transformation code or in a separate semantic-layer product, so the same switching question applies. Atlan, which sells metadata tooling, reads Microsoft’s documentation as saying that data agent quality follows the quality of semantic model metadata more than the model behind it. That matches what I would expect. Iceberg standardizes table metadata and data interoperability, not the business-semantic layer. Definitions may live in platform semantic models, BI tools, transformation code or external metadata services. They can be recreated or migrated, but preserving their meaning, relationships, permissions and behavior is a real switching cost.
The best objection
The sharpest pushback I would expect from an architect is simple: with Iceberg everywhere, pick the best engine for each workload and stop agonizing. For storage I agree, and I would build that way, provided one catalog is authoritative for each table. Two things keep it from being the whole answer.
First, Iceberg support is a matrix of version, operation and engine, and the matrix keeps moving. A Snowflake release note from May 7 2026 said external writes to its v3 tables were not yet supported. The current documentation lists read and write for v2 and v3. Test the exact combination you need. Reading the same table from two engines is the easy part. Running both against it in production, especially when more than one of them writes, is where deletes, concurrent commits, compaction and snapshot retention differ, so test your real read and write pattern. Reading is also easier than writing. Two engines that can both read an Iceberg table are not necessarily safe to write it together, so settle delete and merge behavior, concurrent writes, schema and partition evolution, compaction and snapshot retention before you let two engines share a table in production.
Second, someone has to own the catalog. Snowflake’s own comparison of Horizon and Unity Catalog, a vendor’s view of course, shows capabilities that depend on which catalog owns the table. The catalog owner determines which catalog-native capabilities and governance integrations are available, although what you actually get also depends on the consuming engine and the protocol features it supports. Choosing the catalog is the platform decision, whatever you call it. It deserves its own line in the scorecard, because open tables lower the cost of leaving storage, not the cost of leaving governance: policies, lineage and audit history live in the catalog.
Five questions, in this order
Residency, private networking, identity federation and contract terms eliminate platforms before any of this starts. Questions 1 and 2 build a shortlist, questions 3 to 5 are tested in the proof of concept, and the results feed the final decision. The platform leanings are starting hypotheses, not rules.
1. Where do your identity, data and application teams already live, and what skills and support model do you have?
An AWS estate points to the native stack, or to Databricks or Snowflake running on AWS. A GCP estate points to BigQuery, or to Databricks or Snowflake running on Google Cloud. A Microsoft estate points to Fabric, or to Azure Databricks or Snowflake on Azure. For many organizations this settles it, and that is a fine outcome. If you already run two or three platforms, test the agent path that crosses them as its own case, since delegated identity does not carry across platforms.
2. Who will build?
ML and engineering-heavy teams that value open formats often start with Databricks. SQL-first teams that value sharing and isolation often start with Snowflake. Application developers who need tight control over agent runtime identity may start with AWS, where AgentCore is one way to manage agent identity and runtime, not the only one. Test each hypothesis against your workloads, governance needs and integrations.
3. Does the blast radius hold?
Put an agent in front of a table with row-level policy and record what a restricted user, a privileged user, a delegated identity and a service identity can read through the native path, an external Iceberg engine and an MCP tool. Include prompt injection and unauthorized actions, and any vector index, cache or memory the agent builds from the data, where row policy does not follow the copy, on synthetic or masked data. On every path the pass condition is the same: the policy is enforced, or the request is refused because it cannot be. Treat any other outcome as a security and compliance finding, and do not sign while one is open.
4. What does one correct answer cost?
Replay a week of realistic agent exploration under each billing unit, including the shared-capacity case. Run it cold and warm, at your negotiated rates, and record how far each spend limit can be overshot. If the agent does not exist yet, build the traces from analyst query logs and say so, and count model inference and agent runtime alongside scan cost, since they can exceed it. Compare cost per correct answer: total attributable cost, covering model inference, query execution, agent runtime, memory and retrieval, gateway, observability, storage and network, divided by the answers that meet your agreed accuracy threshold.
5. What does it cost to rebuild your definitions elsewhere?
Export your semantic layer, including metrics, relationships, permissions and tests, and estimate the hours to recreate it on the runner-up. Then run the gold question set on both and compare the answers, because a recreated metric can run cleanly and still return a different number, for example from a different fiscal calendar or currency rate. The hours, plus any answers that still disagree, are your real switching cost.
Before you sign
Run a 30-day structured evaluation with two finalists, one of them your current platform if you have one, on one real use case. It helps you choose a platform. It does not certify production readiness. Agree the pass marks before the first query runs, treat the security tests as vetoes rather than weighted points so a high score elsewhere cannot offset a leak, and for every test record the scenario, expected behavior, actual result, pass or fail, evidence captured and an owner. A capability in beta or preview is not the same as a generally available one.
Score each finalist on:
- Task completion and answer correctness against a gold question set (at 100 questions a gap under about 10 points is noise, so score both finalists on the same questions).
- Identity fidelity from the agent to the data.
- Policy enforcement for restricted and privileged users on every access path, including external Iceberg engines and MCP tools.
- Compatibility of the exact Iceberg version and operation you need with your external engines.
- Semantic portability: engineering hours to rebuild definitions on the runner-up.
- Resilience and recovery from failures and retries.
- Cost per correct answer, and how far costs can overrun a limit.
- p95 latency under concurrent workloads.
- Operational effort to monitor and troubleshoot.
- Catalog ownership and exit: which catalog is authoritative for each table, and whether policies, lineage and audit history can be exported.
Where I land
Pick the platform whose controls you have watched cap, attribute and enforce on every path your agents will use. Table formats are the easy part now. Building an AI application keeps getting cheaper, and governing one has not followed it down.
Sources
The document lists these sources by name:
Evaluating Enterprise Data Platforms for AI & Agents?
Abilytics partners with enterprise technology teams to evaluate lakehouse catalogs, design zero-trust agent permissions, enforce cross-engine governance, and benchmark cost-per-correct-answer.
