Jevon’s Paradox: While advances in data engineering have reduced the unit production cost of data, the proliferation of data models and dashboards can offset these efficiency gains. This phenomenon, known as Jevon’s Paradox or Effect, was first observed by English economist William Jevons in 1865 while studying the impact of improvements in coal efficiency.
In recent years, significant improvements have been made in the domain of data engineering, from data ingestion to pipeline building, data modeling, and overall ease of debugging and maintenance.
These advancements have lowered the friction to ingest more data sources and build more data models for consumption. Tasks that once required a team of 3-4 engineers can now be accomplished by a solo engineer.
However, with the decrease in production costs, the demand for data and all its variants has increased, leading to a proliferation of assets for end consumers. This is evident in the growing volume of datasets and dashboards being produced.
Maintaining a sprawl of dashboards may end up reversing the efficiency gains as it increases the cost of maintenance. Furthermore, it becomes increasingly challenging for consumers to find the information they need.
The growing tax on consumer discoverability and usability incentivizes the creation of even more datasets and dashboards, perpetuating the cycle.
To break this negative loop, we need to re-evaluate and change how we work:
- Introduction of New Abstractions: We need a new abstraction on top of base datasets. By modeling metrics, entities, and slices/segments as first-class objects in code, we can make them reusable, composable, and adaptable for a wide variety of use cases. Metric layers represent a promising step towards this future.
- A New Framework for Regulation: Even with metric layers, we still need a framework to regulate how data is produced and consumed. Metric trees, serving as a lingua franca of the business, can regulate this producer-consumer dynamic. They align the organization on what matters, and modulate what gets produced and consumed
In essence, by implementing a framework for regulating the producer-consumer dynamic supported ideally by a new abstraction, organizations can navigate the challenges posed by Jevon’s Paradox - and ensure that their data assets remain useful, valuable and manageable in the long run.
