Options on Kafka sink to open table Formats: Apache Iceberg and Apache Hudi

Confluent just announced Tableflow. Apache Iceberg and Apache Hudi already have solutions in the marketplace.

Apache Kafka reigns supreme when it comes to real-time data pipelines. But where does that data go for further processing and analysis? This is where Apache Iceberg and Apache Hudi step in, offering powerful data lake storage solutions.

The question then becomes: how do you efficiently move your data from Kafka to these data lakes? The answer lies in Kafka Connect, a framework that bridges the gap between Kafka and various data stores. But here’s the twist: Kafka Connect offers multiple sink connectors for both Iceberg and Hudi, each with its own strengths. Let’s dive into the options:

Apache Iceberg Kafka Connect Sink:

https://github.com/tabular-io/iceberg-kafka-connect

• Focus on Schema Evolution: Iceberg shines when dealing with evolving data schemas. Its schema flexibility allows you to seamlessly adapt to changes without data loss.
• Optimized for Analytics: Iceberg prioritizes fast and efficient querying of your data lake. If complex analytics are your endgame, Iceberg’s query performance is a strong contender.
• Connector Availability: Several Iceberg Kafka Connect sink connectors are available, including those from GetInData and Tabular. These connectors provide configuration options for fine-tuning your data ingestion process.

Apache Hudi Kafka Connect Sink Connector:

https://github.com/apache/hudi/blob/master/hudi-kafka-connect/README.md

• Transactional Guarantees: Hudi prioritizes data integrity. Its transactional capabilities ensure exactly-once delivery and no missing records, crucial for critical data pipelines.
• Upserts and Deletes: Hudi allows for fast upserts and deletes, making it well-suited for scenarios where data updates are essential.
• Emerging Connector Landscape: While Hudi connector options are still evolving (like Onehouse’s offering), they provide the core functionality for streaming data into Hudi tables.

Confluent Tableflow:

• Runs on Confluent Cloud: It’s a supported connector hosted by Confluent.
• Only supports Apache Iceberg: It’s an early release as of March 2024 and currently only supports Apache Iceberg.

Now, for the exciting part: analyzing your data! Here’s where StarRocks comes in. StarRocks is a powerful distributed columnar storage engine designed for real-time analytics on massive datasets. The beauty lies in its ability to natively query both Apache Iceberg and Hudi tables.

This means you can leverage StarRocks’ blazing-fast performance and rich SQL functionality to analyze data residing in either Iceberg or Hudi tables, regardless of the Kafka sink connector you used.

Benefits of using StarRocks:

• Unified Analytics: Query Iceberg and Hudi tables seamlessly without worrying about underlying formats.
• Real-time Performance: StarRocks delivers low-latency analytics on your streaming data, enabling near real-time insights.
• Scalability: Handle ever-increasing data volumes with StarRocks’ distributed architecture.

Query data on top of the lake, support performant JOINS at scale, support 1000s of users doing adhoc queries

Run StarRocks on top of raw data and then create views or materialized views as needed.

AirBnB with StarRocks: 4 JOINS with billions of rows in under 4 seconds

Tencent Games with StarRocks: 400+ users doing ad hoc queries on xx+ petabytes of data on Apache Iceberg files.

Building a robust data lake involves choosing the right tools for each stage of the pipeline. Kafka excels at real-time data ingestion, while Iceberg and Hudi provide flexible storage options. StarRocks, with its unified support for both formats, empowers you to unlock the full potential of your data lake for real-time and historical analytics. So, leverage Kafka, Iceberg/Hudi, and StarRocks to build a powerful data-driven architecture!