Spark 4  user defined functions (UDFs) have a bunch of new knobs for acceleration. This talk will look at the history of Spark user defined functions, from resilient distributed datas (RDDs) land where everything* was a lambda to today where we have ~5 types of UDFs (depending on how you count). We'll explore how to upgrade your UDFs for maximum performance (vectorization, code generation*, oh my!)

The final part of this talk will look the future as well (transpilation, RPCs, etc.)

An open, publicly available Iceberg REST Catalog (IRC) paired with well‑curated, open datasets will help drive the next tranche of Iceberg adoption. This will be the new paradigm to get started with Iceberg.

This talk will walk through the architecture, how it works end‑to‑end, and the benefits for both new and existing Iceberg users.

We’ll cover a few exciting use cases, such as

- Simple UX (easiest way to get started with Iceberg on any engine with IRC)- Fair Benchmark (same dataset, different engines)
- Cloud agnostic (supports all object storage)
- Data sharing (simplest way to share data across engines, clouds, vendors)

The past few years have brought a Cambrian explosion of new columnar formats challenging Parquet. Each promising optimizations for modern workloads.
Entering this AI-dominated era, some argue data infra designed before vector embeddings and GPU-optimized layouts won't cut it moving forward. While these projects incorporate genuine innovations, this framing overlooks what made the original successful. Parquet's main contribution isn't just technical.
The real opportunity is not in choosing between old and new, but in leveraging the communities of these established projects to absorb innovations while maintaining interoperability.

Brief about Fluss (Streaming storage with Table as first-class citizen), features (Union Read, Zero ETL, Integration with Lakehouse & Table format), Architecture shift (Lambda to Kappa), Future work

Apache Polaris is designed as a next generation open metadata and governance layer for modern lakehouses. While Polaris started with strong roots in Apache Iceberg, real world data platforms rarely live in a single table format. Apache Hudi remains a critical choice for ingestion heavy, streaming, and incremental processing workloads.
In this talk, we walk through how Polaris integrates with Apache Hudi to provide a unified catalog, consistent governance, and centralized metadata management across table formats. We will cover the motivation behind Hudi support, the architectural approach taken in Polaris, and the key challenges.
Attendees will see how Polaris enables format aware access control, discovery, and interoperability without forcing users to migrate or compromise on Hudi specific capabilities. We will also share lessons learned, current limitations, and what true multi format lakehouse governance looks like in practice.

Apache Gluten is an open-source project that brings the power of native execution to the modern data lakehouse. By offloading query execution from the JVM to high-performance C++ backends like Velox — and leveraging GPU acceleration — Gluten delivers major speedups for compute-intensive workloads while remaining fully compatible with popular engines such as Apache Spark and Flink.
In this session, we’ll introduce what Gluten is, how it works, and why it matters. You’ll learn about the architecture behind native and GPU-accelerated query execution, how Gluten integrates seamlessly into existing Spark and Delta Lake pipelines, and the performance gains users have seen in production.
Whether you're running analytics at scale, optimizing Delta Lake workloads, or just curious about the future of high-performance query engines, this talk will help you understand where Gluten fits in the open data ecosystem and how to get started.

There is a lot of excitement around the lakehouse architecture today, which unifies two mainstream data architectures - data warehouse & data lakes - promising to do more with less. All the major data vendors have embraced the use of open table formats, due to demand for the flexibility & openness promised by supporting an open format. Projects such as Apache Iceberg, Apache Hudi, and Delta Lake have been at the center of this shift. In addition, newer table formats like DuckLake and Lance have emerged. Together, these efforts have helped establish an open and adaptable foundation for data, enabling enterprises to choose compute engines based on their workload needs rather than being locked into proprietary storage formats.

However, as terms like “open table format” and “open data lakehouse” are often used interchangeably, there is a growing need for clarity and a deeper technical understanding of what openness actually means in this context. In this session, we will do a technical breakdown of the lakehouse architecture & and examine what truly brings openness to a data platform.

As AI systems evolve from models to autonomous agents, data platforms need more than storage and compute—they need a control plane. This session explains why metadata is becoming that control plane and the foundation of an AI-native data platform. Using Apache Gravitino as an example, we’ll show how a metadata-centric architecture unifies multi-cloud, multi-format, and multi-engine stacks, connecting formats like Iceberg, Hudi, and Lance with engines such as Spark, Trino, Ray, and Daft. You’ll learn how this approach enables AI agents and modern data platforms to safely discover, govern, and access data at scale.

While many developers can download a model from Hugging Face, few grasp why latency spikes the moment concurrent users hit an endpoint. This talk explores the "physics" of LLM inference, emphasizing that engineering a production-grade service requires understanding hardware constraints rather than just adding more GPUs. We will analyze the fundamental split between the prefill phase, which is compute-bound (GEMM) due to parallel prompt processing, and the decode phase, which is severely memory-bound (GEMV) as it generates tokens one by one. Central to this is the KV Cache Crisis; while caching transforms generation from $O(n^2) to $O(n) per token, it introduces massive memory management challenges, often consuming more space than the model weights themselves.

To solve these bottlenecks, we will examine how continuous batching (iteration-level scheduling) and PagedAttention (block-based memory allocation) saturate GPUs by eliminating the "waiting waste" and fragmentation typical of static batching. We then bridge these single-GPU "physics" to distributed systems using Ray Serve, which provides the programmable infrastructure to manage heterogeneous replicas and offload CPU-bound work like tokenization to separate worker pools. The session concludes with a live demo featuring a production-ready deployment of Ray Serve and vLLM. Attendees will see how to move from single-node bottlenecks to a distributed, elastic system that maintains high throughput and low Inter-Token Latency (ITL) even under heavy concurrent load.

More and more enterprises are post-training AI models to be custom tailored for their domain and their users. Success in model training depends critically on establishing the right data flywheel to improve iteration speed. From exploration to curation, from feature engineering to training, current data infrastructure solutions are insufficient to support the full iteration loop. Instead, model development teams are wrestling with a hodge podge of single purpose tools and having to copy different parts of their data to different systems for different steps in their workflow. This is one of the most important reasons why so many enterprise AI model training efforts are delayed or fail. For this talk, we’ll cover how LanceDB solves this problem by managing all of your multimodal training data in a unified table and streamlining the entire exploration to GPU-loading path. Via customer case studies, we’ll outline what this new architecture looks like, practical recipes for success, and how it all fits into your existing enterprise data architecture.

The open lakehouse was built around structured data leveraging file formats such as Apache Parquet and data formats like Apache Avro. The lakehouse table formats built on top of them were designed in an era before embeddings, multimodal training datasets, and AI-driven applications became mainstream. Today, enterprises building AI pipelines are forced to stitch together a fragmented stack: a vector database for similarity search, an object store for storing large blobs of image and video data, a document store for semi-structured data, and a lakehouse for everything else. The result is duplicated data, broken lineage, and no unified governance — all of which slow down the teams building AI data applications.

‍

The solution is not to abandon the lakehouse but to evolve it with the core primitives that AI workloads actually need. That means first-class data types: a typed VECTOR column that encodes dimension and element type; a BLOB type with inline or out-of-line storage and managed lifecycle semantics; and a VARIANT type for schema-flexible semi-structured data with optional type shredding for high-performance analytics. It also means embracing file formats designed for AI — like Lance, which support faster random access for writing/reading vector embeddings and blob data than Parquet.

‍

In this talk, we'll walk through the work we've done in Apache Hudi to make this a reality. We'll cover how VECTOR, BLOB, and VARIANT types are represented in Hudi's schema layer and stored in Parquet, how we now have supported the Lance file format support. We will also discuss how we added support for VECTOR_SEARCH via a new Spark SQL function —which enables k-nearest-neighbor search with cosine, L2, and dot product distance metrics directly against your lakehouse, no external vector database required. Attendees will leave with a concrete understanding of what an AI-native open lakehouse looks like and how to adopt these capabilities today for the next generation of data and ai workloads.