AWS Glue 6.0 arrives with Spark 4.1, Apache Iceberg v3 support and a 30 percent price cut

A table format gives object storage the properties of a database. Object stores such as S3 hold files but know nothing about rows, schemas or transactions. Iceberg adds a metadata layer that records which files belong to a table, what schema they follow and a chain of snapshots representing each commit. Engines coordinate through that metadata, which is how several tools can read and write one table without corrupting it, and how time travel to an earlier snapshot works.

VARIANT with shredding keeps flexibility and speed. Semi-structured data like JSON varies from record to record, so forcing it into fixed columns breaks whenever a new field appears. A VARIANT column stores the value in a binary encoding that preserves its structure, while shredding extracts frequently used fields into ordinary columns behind the scenes. Queries on those fields run at columnar speed and rare fields remain queryable without a schema change.

Arrow removes a hidden tax on Python. Spark runs on the JVM, and calling a Python function historically meant converting each batch of rows to a format Python understands and back again. Apache Arrow defines a columnar memory layout that both sides can read directly, so data can be handed over without copying or re-encoding. A user-defined function that operates on Arrow batches processes whole columns at once instead of one row at a time, which also lets libraries such as pandas or NumPy apply vectorised operations across the batch.

Declarative pipelines separate intent from scheduling. In an imperative script the author decides the order in which tables are built and must handle dependencies manually. A declarative approach lets the author state what each table is derived from, and the engine builds a dependency graph, runs independent steps in parallel and re-runs only what changed. This mirrors how SQL views or build tools such as Make work, and it reduces a common class of ordering bugs.

Precision types exist because rounding is data loss. A timestamp stored to the millisecond silently discards detail from a sensor that reports every microsecond, and two events can appear simultaneous when they were not. Nanosecond timestamps and exact geometry types preserve what the source measured. Choosing the type that matches the instrument avoids analytical errors that are very hard to detect after the fact, because once precision is lost at ingestion no downstream query can recover it.