The Right Data Foundation Depends on the Question

Working around Snowflake and Databricks reinforced a simple point: architecture has to serve the decisions a business needs to make.

Original line illustration for the right data foundation depends on the question, on a dark signal grid.

There is no one-size-fits-all data foundation.

I saw this firsthand working at both Snowflake and Databricks. The right approach depends on what you are building, your stage, and what you actually need the data for. Sometimes the mistake is simply building too much too early.

Start with the decision

Before turning a platform discussion into a comparison of features, I want to understand what the team is trying to decide.

Are we trying to understand which customers are getting value? Find the part of a sales process where opportunities stall? Make a product workflow more dependable? Each question puts different demands on the information behind it.

A useful starting point is straightforward: who needs this information, what will they do with it, and how often does the answer need to change?

If we cannot answer that, a more elaborate architecture may just make the uncertainty more expensive.

Context changes the answer

An early-stage team and a large organization can use the same vocabulary while solving very different problems. Team capacity, existing systems, and the cost of maintaining another piece of infrastructure all matter.

That is why I am careful with advice that begins, “This worked at my last company.” It may be a useful example. It is not enough of a reason to copy the entire design.

The same caution applies to Snowflake-versus-Databricks arguments. My takeaway from being around both is to spend more time understanding the workload and the people than defending a logo.

Make the foundation earn its complexity

I want a data foundation that helps a team make better decisions and supports what they are actually building. It should have room to grow, but room to grow is different from building every possible future requirement today.

Start with a concrete use case. Make ownership clear. Check whether the resulting information is useful. Then expand as the need becomes real.

The interesting question is not which architecture sounds most impressive. It is which one helps this team do its work.

Written by Babak FarahmandMore field notes →