Limitless data product potential

A Fortune 500 client and leading roaster and coffee retailer was struggling to tackle data governance challenges with off-the-shelf software.

As the retail organization’s data product offering grew, leaders wanted to keep maintenance responsibilities manageable while ensuring data issues didn’t drive poor decisions or negatively impact customer experiences. The client also needed improved visibility to ensure data was being processed correctly and compliantly.

The organization had tried several off-the-shelf solutions that failed to support the complex needs of their global business.

We have a long history of partnering with the retailer — more than seven years — to develop data transformation pipelines that support analysis, reporting and data science applications.

They asked us to again build a custom solution that would ensure the maintenance of proper data documentation, privacy compliance and fidelity for decision-making and customer-facing applications.

We built custom code libraries using Microsoft Azure. This facilitated containerized data pipeline development, testing and productionalization on the cloud.

• Sensors detect when data has arrived, triggering dependent jobs.

• Self-documenting libraries ensure code is easily discoverable and instantly up to date with every deployment.

• Data lineage tracks interdependencies between every source data field, KPI and reporting artifact.

• Data sensors and a plug-and-play anomaly detection framework work to detect data issues.

• Subscription and registry features mean users are informed when key data is corrupted and receive regular resolution updates.

This software established a robust framework to scale data transformation pipelines. It empowered this global client to quickly, easily and compliantly develop new data products — all while maintaining the fidelity of strategic data:

• 86 data pipelines are currently running, with more than 300,000 data sources and automated lineage.

• The solution enables intelligent strategic decision-making and high-quality customer experiences.

• Documented and highly discoverable data sources enable self-service for new data product development.

• Data product owners can subscribe to their underlying data assets to receive alerts if data fidelity issues surface.