Data Mesh architecture: building a robust data platform with a small team 💪

Data is an invaluable asset for organisations, regardless of the industries they operate in. The sheer volume generated daily is staggering—402.74 million terabytes, according to the latest estimates 🤯

At Carmoola, we're no exception; we're a data-driven company committed to operational excellence through strategic automation. A critical part of this commitment involves investing in a robust data infrastructure that supports automated decision-making frameworks.

Earlier this year, we undertook a significant transformation in our approach to data and analytics. Traditionally, applications generate vast amounts of raw data, often tailored to meet specific application needs rather than delivering direct business value. Without the proper structure and analysis, this raw data often remains underutilised. Many teams, including ours, have historically worked in silos, duplicating efforts by building queries across multiple sources. We recognised these challenges and committed to optimising our processes, closing knowledge gaps, and empowering teams to leverage data more efficiently for faster, more informed decision-making ⚡️

Our research led us to adopt the Data Mesh approach; a modern solution that aligns perfectly with our needs. For those interested in the technical details, the official Data Mesh website offers comprehensive information. What makes our journey remarkable is the significant progress we've made with a small team of engineers, thanks to a strong emphasis on operational efficiency.

We embrace domain-driven development, making Data Mesh a natural fit for our approach. This method aligns seamlessly with the core principle of Data Mesh—domain-oriented decentralisation—which enables more effective management and utilisation of analytical data across the organisation.

On one hand, we have a wealth of structured data tailored for our applications. On the other hand, we have diverse data ‘consumers’ internally—such as marketing, analytics, and credit risk teams. Our goal is to avoid creating a central data team that could become a bottleneck, overwhelmed with queries and causing delays in critical business decisions. Instead, we aim to eliminate inefficiencies and ensure faster, more agile responses 🤸

Implementing a Data Mesh can be a significant challenge without a team that embraces growth and is eager to tackle complex problems. At Carmoola, our core values encourage and necessitate a culture of continuous improvement and curiosity, which is essential for successfully adopting this decentralised data strategy.

To truly empower our teams, we rely on the Data Mesh approach and the support of our Data Engineering team. Though small, this team plays a crucial role in guiding the process. Instead of handling all the data, they focus on enabling domain teams—those who know their data best—to create and manage their own ‘data products’. When changes occur at the application level, these same teams are responsible for updating their data products, ensuring that ownership remains with those who understand the data intimately, and avoiding delays or miscommunication caused by a middle layer.

In contrast, the traditional model relies on a data engineer to mediate between the data owner and the consumer, clarifying data requirements and building queries or dashboards. This extra layer slows down the process and increases business costs by requiring additional resources.

Our win-win scenario with Data Mesh

Data ‘consumers’ gain access to structured data tailored to their needs, without worrying about unexpected changes. Meanwhile, the company avoids the cost of hiring additional data engineers to manage a complex middle layer. Instead, we invest in knowledge and empowerment. This approach has allowed us to reduce BI tool queries by 50%, eliminating misuse, duplication, and unnecessary complexity, leading to faster insights and more efficient operations.

As one of our analysts noted, with Data Mesh, you achieve the same results much quicker without needing to replicate logic across different workflows 💡

Every team member is committed to ensuring the creation of efficient and consistent data products necessary for their work. Data analysts can take ownership of a data product, combining multiple sources to tailor it for their audience or analysis. This accelerates decision-making, allowing business insights to flow quickly and impact the entire organisation more effectively.

One of our engineers highlighted that the Data Mesh approach simplifies our internal data architecture by providing a single contract and source of truth.

An added advantage of the Data Mesh approach is that it's secure by design. In traditional setups, raw data is often spread across multiple sources, requiring broad access to sensitive information for those querying it. With Data Mesh, data product owners can design products that exclude sensitive data from the warehouse, significantly enhancing our overall data security.

In summary

Data Mesh has empowered Carmoola to streamline data processes, enhance security, and reduce costs—all while driving growth in knowledge, efficiency, and user outcomes. With this approach, we're positioned for continued excellence in delivering data-driven outcomes.

And all of this, of course, results in a better experience for the car buyers who are turning to Carmoola to help finance their dream cars 🚗 🚙