If you’ve read some of the articles on Apache Hudi, Apache Iceberg, and Delta Lake, all 3 formats were built with specific use cases in mind and thus there are feature differences (see https://atwong.medium.com/my-favorite-articles-to-understand-the-differences-between-open-table-formats-apache-hudi-apache-de0bd760eead to understand more). …

You can do flat (also known as denormalized), but once you move from flat, you need the ability to do JOINS. Many OLAP databases can’t do JOINS at scale (eg. high queries per second / QPS or across GB, TB or PB of data in single seconds or sub-seconds query…

So my friend Kelvin told me I did not know. He told me that Southern California Edison (SCE) sponsors, via a rebate, an energy monitoring device for your home (read more at https://www.emporiaenergy.com/utilities). The device retails for $40 on Amazon and with the $25 rebate, it goes down to $15. Much, much cheaper than a Sense home monitoring device (of course you get more deeper data). Still, if you live in Orange County, CA, this is a must-buy for your house.

So there is a comment from a database vendor that says you need all these different types of databases for all these different types of use cases. Read more at https://www.reddit.com/r/dataengineering/comments/164x1qp/comment/jyl7w0n/?utm_source=share&utm_medium=web2x&context=3. Maybe. Yes, pick the right tool for the job, but no one has a large data engineering team that…

I get this question a lot from the community that I manage. What would I build and use if I did a modern data stack for data warehouse / data lakehouse. Requirements: OLAP database that can meet sub-second query times and single digit second ingestion times. Oh, you can also…

So it seems like Data Lakehouses are being slotted in 3 different types. Hybrid: This makes sense. The Snowflake, AWS RedShift, GCP BigQuery and Microsoft Azure Synapse Analytic sin the world are adding open table formats like Apache Iceberg into read (first) and then write (later). …

I think this device works for many water districts in Orange County, Los Angeles County and SoCal. So I got this nifty device recently. It’s called Flume Water and it allows residential customers to monitor your water usage. Why? My neighbor messaged me and told me that one of my…

There are the top 4 open source replacements for Microsoft Power BI, Tableau, Looker, MicroStrategy and other Business Intelligence and Data Visualization Tools. Apache SuperSet Metabase Lightdash Streamlit Apache Superset is a modern, open-source data exploration and data visualization platform. It is designed to be fast, lightweight, and easy to…

A database cube, also known as a data cube, is a multidimensional data structure that is used for data analysis and reporting. It organizes data into dimensions and measures, which allows users to slice and dice the data to gain insights into their business. Database cubes were popular in the…