I’ve spent the past decade working at the intersection of AI, systems optimization, and cloud infrastructure. At Google, I helped efficiently scale data center infrastructure on one of the largest and most complex systems in the world. After Google, I started Espresso AI with one goal: to bring that same world-class efficiency to data warehouses.
The Costly Inefficiency of Modern Data Warehouses
Data warehouses and lakehouses have transformed how organizations manage and analyze data. They offer scalable, flexible, and accessible data solutions. However, they often lead to skyrocketing compute costs. For example, Snowflake, which charges based on consumption, can become really expensive really fast, especially when workloads are poorly optimized, load spikes aren’t managed effectively, or the underlying data grows faster than expected. (More on this in our post: explaining Snowflake pricing.)
Databricks SQL is a competing platform that has seen explosive growth, but suffers from the same problems. Without real-time optimization, many data warehouse users leave substantial resources idle: from what we’ve seen, the average data warehouse customer has idle time close to 50%. That means you might be wasting -half- of your data warehousing budget.
Why Are These Platforms So Wasteful?
The core of the problem is static warehouse allocation. Data warehouses assign workloads to individual warehouses, which consistently results in excessive compute usage. This is easy to understand for new users: the dbt job goes on the dbt cluster, the BI job goes on the BI cluster, and so on. It’s also great for orgs lifting-and-shifting from on-prem into the cloud because it maps closely to how they think about analytic workloads.
Unfortunately, it’s 15 years behind the state-of-the-art when it comes to managing compute at scale.
This is where our experience at Google has shaped Espresso AI’s approach. We’ve taken the concepts behind modern data center management – predictive autoscaling, hardware rightsizing, and dynamic real-time routing – and used ML to translate them
to the world of data engineering, enabling platforms like Databricks SQL to become agentic, adaptive, and cost-efficient.
How Espresso AI Transforms Data Lakehouses
Our platform leverages machine learning models trained on a customer’s unique metadata logs to create three core agents:
- Autoscaling Agent: Recognizing fluctuations in workload demands, our models predict spikes and dips and adjust resources in real time. This maximizes efficiency without sacrificing performance.
- Scheduling Agent: Instead of static warehouse placement, our system intelligently analyzes ongoing workloads to route queries to existing resources, reducing idle time and eliminating wasted computing power.
- Query Optimization Agent: SQL is optimized before it hits the data lakehouse. By refining queries upfront, we reduce the computational load, improve query response times, and significantly cut costs.
Winning Together: Bringing Efficiency to both Databricks and Snowflake
With Espresso AI, Databricks SQL users can cut their costs in half. We’ve already shown that this works on Snowflake, and we’re now applying the same techniques to our new Databricks offering. Customers of both platforms now have access to world-class compute efficiency by leveraging Espresso AI.
If you’ve read this far, you’re probably tired of watching your data warehouse bill grow quarter after quarter. Reach out to Espresso AI – we can help.
