As a Data Warehouse user, which open table format should I use, and who is backing who?
Spicy info. Word on the street. That’s what I deal in.
By the way, I’m Albert, and I’m Head of Community and Developer Relations at CelerData. CelerData is a $60 million VC-funded startup that is building StarRocks, an open-source replacement for Snowflake, Big Query, RedShift, and Databricks SQL Warehouse.
Links for reference
- https://medium.com/@kywe665/dont-worry-about-data-lakehouse-features-trust-in-google-search-5d8d13675680
- https://tableformats.sundeck.io/
What are my takeaways?
Interesting is who is backing who.
I am not that surprised by Databricks (primary sponsor of Delta Lake) and Azure. As always, follow the money (and investments).
I’m a bit surprised. I thought Iceberg would have more contributions outside of Tabular. I also wish that Onetable.dev was part of this matrix. Maybe I should make one :-)
Which reminds me, I’m a big fan of Onetable.dev. If you’re not familiar, Onetable allows you to convert one open table format to another without database re-ingestion. Another big ability is to help register Delta Lake, Hudi and Iceberg files into Apache Hive Metastore (HMS) so that databases like StarRocks and Trino can query the data.
I’ve got a demo of writing to hudi and then converting the data to iceberg and delta lake. Check it out at https://github.com/StarRocks/demo/tree/master/documentation-samples/datalakehouse.
