Versions and support statements in this article are as of 1 October 2026. INVISOM has no commercial relationship with any vendor or project named here.
Every new feature is a compatibility decision
Amazon Redshift supports Apache Iceberg v3 tables. Its documentation also says, in plain terms, that a v2 table upgraded to v3 cannot be downgraded, and that Redshift cannot read or write v3 tables containing struct, list, map or variant columns, nor v3 tables with equality deletes. Databricks' documentation carries a similar warning for Delta Lake: enabling deletion vectors upgrades the table protocol, after which clients without deletion vector support cannot read the table.
Neither statement is a criticism of the vendor. They describe how open table formats actually work. A table format is a contract between writers and readers. Each new capability that changes the bytes on disk or the metadata layout raises the minimum reader version, and any engine below that line silently drops out of your architecture.
Our earlier articles made the business case for open formats (warehouse, lake, lakehouse or mesh) and showed that lock-in has moved to logic and policy (Snowflake and Databricks). This article goes one level down. Its argument: the format war is over, and the work that remains is managing a feature contract per table and choosing a catalog protocol. Enterprises that adopt "Iceberg" or "Delta" as a one-word standard, without specifying versions, features and catalog, have not standardized on anything.
Four layers, four different standards
"Open data format" refers to four layers that evolve independently, under different governance, at different speeds.
Figure 1
Four layers, evolving at four speeds
Status of the open data stack on 1 October 2026. Each layer has its own governance and compatibility rules.
- SemanticsMetrics, dimensions, business definitionsApache Ossie (OSI), incubating since July 2026Early
- Catalog protocolDiscovery, commits, credentials, policy enforcementIceberg REST spec; Apache Polaris 1.8.0; Unity Catalog OSS 0.6.0Decisive layer
- Table formatTransactions, snapshots, schema evolution, row-level changesIceberg spec v3 complete, v4 in development; Delta Lake 4.4.0; Hudi 1.2.1Converged
- File formatColumnar encoding, types, statisticsParquet format 2.14.0; variant and geospatial typesStable
Object storage sits beneath all four layers and is the one part that is already fully portable.
Parquet is the file format underneath almost everything. Its compatibility rules are explicit: forward-compatible features (statistics, Bloom filters, new logical type annotations) can be ignored by older readers, while forward-incompatible ones make data unreadable to them. Parquet added geometry and geography types in format 2.11.0, and its variant encoding, with "shredding" of frequently used fields into typed columns, is now the shared binary representation that Iceberg v3 references and that Spark 4.1 ships as generally available.
Table formats (Iceberg, Delta Lake, Hudi) add transactions, snapshots, schema evolution and row-level changes on top of Parquet files. Catalog protocols decide how an engine finds a table, commits to it and obtains permission to read its files. Semantic specifications describe what the numbers mean. Most enterprise roadmaps still treat these as one decision. They are four.
What the new table features actually do
Iceberg's specification now lists versions 1, 2 and 3 as complete and adopted, with version 4 under active development. Version 3 is the release that matters for enterprises in 2026, and most of its features have direct equivalents in Delta Lake 4.x and Hudi 1.x.
| Feature | What it does | What it means for an enterprise | The catch |
|---|---|---|---|
| Deletion vectors | Record deleted row positions in a compact bitmap per data file, instead of rewriting the file | Cheaper MERGE, UPDATE and DELETE; viable CDC and GDPR-style deletes on large tables | Deleted rows remain physically present until a rewrite; older readers are locked out |
| Row lineage | Gives every row a stable _row_id and the sequence number of the commit that last changed it | Incremental pipelines can find changed rows without full-table comparison | Not maintained for rows changed through equality deletes |
| Variant | Stores semi-structured, JSON-like values in a typed binary encoding, optionally shredded into columns | Event, log and API payloads queryable without rigid schemas | Engine support is uneven; Redshift does not read v3 variant columns |
| Default values | Lets a new column carry a default for old rows without rewriting files | Schema changes on very large tables become metadata-only | Defaults on variant and geospatial columns must be null |
| Branches and tags | Named references to snapshots, with their own retention rules | Write-audit-publish workflows; tagged snapshots for audit or month-end | Retained snapshots retain data, including data you meant to delete |
| Geospatial types | Native geometry and geography columns with bounds for pruning | Location analytics without custom encodings | Newer still; check every engine |
| Table encryption keys | Track wrapped encryption keys in table metadata | A standard place for key references, rather than per-engine conventions | Key management and engine support remain your responsibility |
Two of these deserve a closer look, because they change operational behaviour.
Deletion vectors replace a scaling problem. Version 2 of Iceberg introduced position delete files, which could accumulate into thousands of small files that every reader had to merge. Version 3 prohibits writing new position delete files and allows at most one deletion vector per data file in a snapshot, stored as a Roaring bitmap inside a Puffin file. Writers must merge new deletes into the existing vector. Databricks describes the deletion vector encoding as shared between Delta and Iceberg, which is part of why the two formats can now point at the same files.
Figure 2Illustrative
Three ways to delete ten rows
Worked example: removing 10 rows from a 1 GB Parquet data file in an open table
Any version
Copy-on-write
- On write
- Rewrite the whole 1 GB data file without the 10 deleted rows
- On read
- Read one clean file
Heavy writes, light reads
Iceberg v2 (prohibited for new writes in v3)
Position delete files
- On write
- Add a small file listing file path and row positions
- On read
- Merge every delete file that may apply; many small files accumulate
Light writes, reads degrade over time
Iceberg v3; Delta Lake
Deletion vector
- On write
- Merge positions into the single bitmap kept for that data file
- On read
- Apply one bitmap per data file
Light writes, predictable reads
In all three cases the deleted rows survive in older snapshots until those snapshots expire and the files are removed. Soft deletion is not erasure.
Row lineage moves change tracking into the format. Before v3, finding "what changed since the last run" meant comparing snapshots or relying on engine-specific change feeds. In v3, every newly written row carries _row_id and _last_updated_sequence_number, assigned by inheritance at commit time, so retried commits do not rewrite files. Redshift's documentation shows the practical consequence: after upgrading a v2 table, both fields return null for old rows until the first write, which then assigns lineage for the whole table as a metadata operation.
Delta, Iceberg and Hudi: convergence with different centres of gravity
The three formats now offer similar row-level capabilities. Where they differ is what each project is optimizing next.
- Delta Lake 4.x has focused on the catalog. Version 4.1 brought catalog-managed tables to production status, so a catalog such as Unity Catalog brokers every commit; 4.2 made
REPLACE TABLE AS SELECTand dynamic partition overwrite atomic on those tables and made variant generally available in Delta Kernel; 4.4.0, released on 20 August 2026, added Spark 4.2 support and wider integration with the Unity Catalog Delta API. - Apache Hudi 1.x has focused on database-style write paths. Version 1.0 added secondary and expression indexes, partial updates, non-blocking concurrency control for multiple concurrent writers, and an LSM-tree timeline. Version 1.1 introduced a pluggable table format framework intended to run Hudi's indexing and table services over other formats, with Iceberg support described as future work. Version 1.2 added vector and blob types and support for the Lance file format.
- Apache Iceberg has focused on the specification and the catalog protocol. The version 4 work restructures metadata; the draft already adds relative paths in metadata, which make tables easier to copy or move between storage locations, and the development specification prohibits adding new equality deletes. Databricks has proposed that a future Delta 5.0 adopt the same metadata tree structure as Iceberg v4, which would bring the two formats closer still.
The practical reading: choose by write pattern and catalog, not by feature list. High-frequency upserts from many streaming writers fit Hudi's design; a Unity Catalog-centred estate fits Delta; a multi-engine estate with several vendors fits Iceberg and an Iceberg REST catalog.
Translation layers copy metadata, not semantics
Delta UniForm and Apache XTable let one set of Parquet files be read as more than one format. They are useful, and both document what they leave out.
Figure 3
Translation keeps the files, not the newest features
What survives when one table is exposed in another format
Delta UniForm
Generates Iceberg metadata for a Delta table in Unity Catalog
- Data files and schemaCarried
One copy of Parquet, two sets of metadata
- Writes from Iceberg clientsNot carried
Iceberg access is read-only
- Deletion vectorsDepends
Not with IcebergCompatV2; supported with IcebergCompatV3
Apache XTable
Converts metadata between Hudi, Delta, Iceberg (plus Paimon and Parquet sources)
- Data files, column stats, partitionsCarried
Incremental or full sync
- Merge-on-read tablesNot carried
Copy-on-write and read-optimized views only
- Deletion vectors and Hudi log filesNot carried
Documented as not captured
UniForm generates Iceberg metadata asynchronously for a Delta table registered in Unity Catalog. Iceberg clients can read but not write. Under the IcebergCompatV2 protocol, deletion vectors are not allowed; IcebergCompatV3 lifts that, provided the Iceberg readers support v3. XTable, whose 0.4.0-incubating release in August 2026 added Delta Kernel as a target and Paimon and Parquet as sources, states that it supports copy-on-write and read-optimized tables only, and does not carry Hudi log files or Delta and Iceberg deletion vectors.
The pattern is consistent. Translation works for the subset of features both sides understand, at the moment of translation. The newest features, the ones that motivated the upgrade, are the first to fall outside that subset. Translation is a bridge for migration and read access, not a substitute for agreeing a shared feature level.
The catalog is becoming the platform
If tables are shared, the catalog decides who can find them, who can commit to them and who can read their files. The Iceberg REST catalog specification has become the most widely implemented answer, and its scope has grown well beyond listing tables.
Figure 4
How a REST catalog brokers access
The sequence an engine follows to read a governed Iceberg table. Step 3 is optional in the specification and decisive for security.
- 1AuthenticateEngine obtains a token for its principalOAuth token endpoint or external identity provider
- 2Load tableEngine asks for table metadata and signals it accepts delegated accessloadTable with X-Iceberg-Access-Delegation
- 3Plan the scan on the serverCatalog evaluates row filters and column masks and returns only permitted filesplanTableScan, fetchScanTasks
- 4Receive scoped accessShort-lived credentials for the planned files, or per-request signingvended-credentials or remote-signing
- 5Read filesEngine reads Parquet directly from object storageNo standing bucket access needed
Three capabilities in the specification matter most for enterprise architecture:
- Credential vending and remote signing. A client signals through the
X-Iceberg-Access-Delegationheader that it can acceptvended-credentialsorremote-signing. The catalog then returns short-lived, table-scoped storage credentials, or signs each storage request itself. Engines no longer need standing access to the whole bucket. - Server-side scan planning. The
planTableScanendpoints let the catalog, rather than the engine, decide which files a query reads, and return storage credentials scoped to that plan. Databricks uses this to extend Unity Catalog row filters and column masks to external Iceberg engines, a capability it labelled Beta in May 2026. This is the first credible mechanism for consistent fine-grained security across engines. - Atomic multi-table commits, views and functions. The specification includes a transaction endpoint for committing changes to several tables at once, and endpoints for views and functions, which begin to address the business-logic portability gap.
Two open implementations compete for this role. Apache Polaris, donated by Snowflake, entered the Apache Incubator in August 2024 and graduated to a top-level project in February 2026; release 1.8.0 shipped on 28 September 2026. It implements the Iceberg REST API, vends credentials, applies role-based access through principal and catalog roles, and can federate other catalogs. Unity Catalog OSS, contributed by Databricks to the LF AI & Data Foundation as a sandbox project in June 2024, reached 0.6.0 on 20 August 2026 with SQL views and metric views as governed catalog objects. Its own README still cautions that its APIs are evolving and should not be assumed stable.
The enterprise implication: evaluate a catalog the way you would evaluate a database. Check which REST endpoints it implements, whether it vends credentials or signs requests for your storage, whether policies are enforced through server-side planning, and how it federates with the catalogs your vendors run.
Semantics: the newest open layer
Shared tables do not guarantee shared meaning. The Open Semantic Interchange, whose specifications were finalized in January 2026, entered the Apache Incubator as Apache Ossie in July 2026. Ossie defines a YAML-based, vendor-neutral format for metrics, dimensions and their relationships, with contributors including Snowflake, Salesforce, Databricks and dbt Labs.
It is early. It does not standardize access policy, and its value depends on BI tools, engines and AI agents consuming it. Still, it is the first serious attempt to make metric definitions as portable as tables, and data architects should track it alongside the REST catalog specification.
Soft deletes are not erasure
Deletion vectors, snapshots, branches and tags all keep old data reachable. That is what makes time travel and audit possible, and it is a problem when a regulator or data subject requires data to be destroyed.
Databricks documents the full sequence for Delta: REORG TABLE ... APPLY (PURGE) to rewrite files containing soft-deleted rows, then VACUUM to remove the old files, noting that ordinary compaction does not guarantee deleted rows are physically removed. Iceberg has the equivalent: rewrite affected data files, then expire snapshots, remembering that tagged snapshots and branches have their own retention rules and keep data alive.
In Saudi Arabia, SDAIA publishes a dedicated guideline on destroying, anonymizing and pseudonymizing personal data under the Personal Data Protection Law. For Gulf entities, the erasure procedure for each table tier should be designed and tested with the table format in mind, and documented as part of the data lifecycle controls described in our NDMO implementation guide. A deletion that only sets a bit in a deletion vector is not destruction.
A table feature contract
The practical tool is a short, written contract per table tier: which format version, which features, which catalog, and which engines must be able to read and write.
Figure 5
A table feature contract by tier
The more engines depend on a table, the more conservative its feature set should be
Strictest contract
Shared tier
Read or written by engines from more than one vendor
- Iceberg via one REST catalog
- New features only when every consuming engine documents support
- Credential vending or remote signing mandatory
Main risk: An unreviewed protocol upgrade silently removes a consumer
Tier
Owned curated tier
One owning engine writes and enforces policy; others read
- Owning engine's native format and latest features
- Read access for others through the catalog or a translation layer
- Row and column policies enforced in one place
Main risk: Readers lose access to newer features when translation lags
Tier
Landing tier
Raw ingestion, short retention
- Whatever the ingestion tool writes best
- Short snapshot retention and automatic expiry
- No direct consumer access
Main risk: Retained snapshots keep personal data longer than intended
The principle behind it: the more engines need a table, the more conservative its feature set. Curated tables owned by one engine can use every new capability. Tables shared across vendors should move to a new feature only when every consuming engine supports it, verified in that engine's documentation rather than in the format's release notes.
What to do next
- Inventory format versions and table features across your estate: format version, deletion vectors, row tracking, column mapping, variant, and which engines read each table.
- Write a feature contract per tier and make it a gate in your table creation templates, so no one enables a protocol upgrade by accident.
- Test v3 upgrades on a copy first, with every consuming engine, since upgrades are one-way.
- Select your catalog on protocol support: REST endpoints, credential vending or remote signing, server-side scan planning and federation.
- Use translation layers for migration and read access, with their documented gaps written into the contract.
- Design and test erasure end to end, including purge, snapshot expiry and tag retention, and record the procedure as evidence for privacy regulators.
Frequently asked questions
Should we standardize on Apache Iceberg or Delta Lake?
Standardize on a feature contract and a catalog protocol rather than a format name. Iceberg is the better default where several vendors' engines must write; Delta fits estates centred on Unity Catalog. With deletion vectors and variant now aligned, many organizations will run both.
Is it safe to upgrade tables to Iceberg v3?
For tables read by engines that fully support v3, usually yes, and the gains in delete performance and change tracking are real. Upgrades cannot be reversed, and support varies by engine and data type, so test with every consumer first.
Do Delta UniForm and Apache XTable remove the need to choose a format?
No. They expose one copy of data in another format's metadata, but they are read-oriented or limited to certain table types, and both document gaps around deletion vectors. Use them as bridges, not as the architecture.
What is the difference between Apache Polaris and Unity Catalog?
Both are open catalogs. Polaris is an Apache top-level project centred on the Iceberg REST specification. Unity Catalog OSS, in the LF AI & Data Foundation, governs tables in several formats, files, functions and models, and now views and metric views. Choose based on the engines and protocols you need.
Sources and further reading
- Iceberg table specification and REST catalog OpenAPI specification, release 1.12.0, Apache Iceberg; Branching and tagging.
- Variant type in Apache Parquet, Apache Parquet, 27 February 2026; Parquet format versions.
- Delta Lake releases: 4.1.0, 4.2.0, 17 April 2026; 4.4.0, 20 August 2026.
- Apache Hudi: 1.0, 16 December 2024; 1.1, 25 November 2025; 1.2, 7 June 2026.
- UniForm documentation, Databricks; XTable features and limitations, Apache XTable.
- Apache Polaris graduation, February 2026; Polaris incubation record.
- Unity Catalog 0.6.0 release notes, 20 August 2026; LF AI & Data welcomes Unity Catalog, 20 June 2024.
- Apache Ossie (Incubating), Snowflake, 8 July 2026.
- Iceberg v3 features in Amazon Redshift, AWS; S3 Tables support for all Iceberg v3 data types, AWS, 30 September 2026.
- Iceberg v3 GA and cross-engine ABAC, Databricks, 28 May 2026 (vendor material); Deletion vectors, Microsoft Learn.
- Personal Data Protection knowledge center, SDAIA.