Back in April, I watched a live discussion between seasoned data folks where the topic of the data function and its value came up. It speaks well of the discipline that the practitioners themselves keep asking if they are actually making a difference. But questions around the ROI of data and whether organizations can succeed without investing in it keeps popping up in discourse. And as if the timing gods were conspiring, I just read the April DBT roundup which has this observation: I think that there is too little scrutiny of the value of data teams today.
And then we have the harsher comparison that happens within the data community — the value of data vs engineering functions. The argument fundamentally goes: engineers build things, you can still build without data, so data isn’t as key a function. It’s compelling on the surface, and if you have axiomatically accepted the power of data, this can shake your core belief.
I’ve helped scale an organization from seed → $1B, consulted with several others, and have both utilized data extensively as an operator and built products working closely with engineers (sometimes with little to no data). More importantly, I’ve been in the hairy exec and board conversations around the different functions, investment levels and resourcing. First, I can tell you there is always scrutiny about the org and levels of investment — and the scrutiny depends on company state, historical outcomes and the vectors for taking the company to a future state. And here’s what I can also say: if you have any meaningful complexity in the business model, it is very hard to directly attribute success to a function or a team (barring sales?), and can even be counter-productive for sustained innovation in the long run. Ammunition just isn’t enough — you need barrels, you need guidance, you need the agility to experiment, you need messaging, you need internal coordination, communication — and all these change over time.
Yes, some organizations can definitely succeed without meaningful investments in data but some organizations can succeed without meaningful investments in engineering or operations or marketing! The simple truth is organizations succeed when they understand their customers well, and deliver a product and business model that is viable long-term. And no singular function has a monopoly on this process, and it’s highly dependent on type and stage of the company. What follows logically is that some companies benefit more from data than others. An application streaming video to a billion users benefits more from engineering than a mom and pop e-commerce store. Ride-sharing or renting apparel benefits more from operational rigor than a database startup. It’s not either or — a core set of functions reconstitute every generation to provide levers for sustained innovation for the time. The relative weights of these functions absolutely needs to vary based on type and stage of company — and that’s the role of leadership to figure out. A 100 year-old fashion company that has thrived historically due to the founding merchandising function and product design ethos may now need to add software engineers, data and operations folks to the mix. On the flip side, organizations that have over-indexed towards engineering as the only vector for innovation may now need to scale down, and instead invest in data or (god forbid!) sales.
A few macro points:
The spreadsheet existed before modern software engineering. We have always attempted to understand the world using data in concert with using that knowledge to build. One doesn’t follow from the other. These are independent disciplines that play off each other.
Software has already eaten the world! Engineers obsess over their tooling and it’s taken as a given that better engineers + better tooling → better product + business. But I distinctly remember a time when the value of engineering was constantly questioned. Still is, but muted.
My 2 cents as to why we are having these data value/ROI conservations or expressing concerns over data teams’ internal tool gazing is that the data function is growing up — it’s going through a similar evolution as engineering — where discussions on tooling, productivity, investment levels, skills, jobs, career paths, reporting structures etc are rippling through the ecosystem of practitioners and leaders — and as it ripples through, we tend to see backlash arguments to check and course-correct, not to get carried away in our perceived spotlight, to remind ourselves that we are only a piece of the puzzle — to ultimately ground the data function in reality and continued value delivery.
This all sounds healthy. But, there is a key open question from a macro perspective: — really, how big is the demand and market ultimately? While I don’t have a precise forecast on this, and it looks today that the data function, the tooling, the labor force etc. is getting big, I do believe this: data will reach a point where it’s also eaten its fair share of the world — where it would become a boring part of the landscape, it’s value self-evident and hopefully, we’ll have far fewer recurring discussions about its value.
