When we assess client data warehouses and their approaches, we find that they are rarely in a place where they are starting from scratch. Typically, teams or organizations are experiencing a history of attempting to deliver analytics in a timely manner and can’t. We’ve seen this result in a lack of stakeholder trust in the accuracy of the data, frustration from technical debt that is impacting new development, and other types of issues that create the necessity to engage a consulting company to perform an assessment and provide results.
When we were engaged by a large medical device maker, they were experiencing the following:
- The client was experiencing a lack of trust from the stakeholders in data warehouse reporting. This led to one-off spreadsheet solutions and perpetuated a “no single source of truth” environment.
- The data warehouse was seen as inflexible and a collection of silos to suit specific needs based on reporting requirements. Dimensions did not conform across data marts and were all single purpose. Development staff and BA’s found themselves having to redesign data marts from the ground up for each new reporting initiative.
- The assessment of the design was hindered by very inconsistent, or complete lack of, project artifacts.
- The data warehouse team was attempting to adopt an Agile approach (like the rest of IT), but Agile alone does not provide enough of a framework for data warehouse teams to thrive.
Our assessment found confirmation of many of these pain points. They needed the coaching associated with the Agile transformation and needed the right approach which we determined to be BEAM* (Lawrence Corr With Jim Stagnitto, Agile Data Warehouse Design, Collaborative Dimensional Modeling, From Whiteboard to Star Schema (Leeds LS28 7UJ, UK: DecisionOne Press, 2013), or see www.modelstorming.com. BEAM* would immediately engage the stakeholders far beyond any past experience, which for them was transformational. It would also transform the team to event-driven analysis and design of discrete events along the client’s value chain. The client established the value chain around the discrete events that represent units and revenue.
In the first sprint, we were tempted to throw everything away and start from scratch. However, because the client had significant database objects and reports around units and revenue, we decided that a blended approach would work best. Meaning, we would perform BEAM* Modelstorming with the stakeholders as we would from scratch to establish the logical design, but then when it came time to physical design and building, we would leverage already existing dimensions and ETL and make changes to have more dimensional conformance. Everything new would then be created (with some reuse as appropriate) in a new data warehouse environment. This would allow the team some level of reuse, and make things conform more to other development down the road.
The biggest challenges faced with making this all work is how legacy work (the temptation of the organization to push continued report driven instead of event-driven analysis and design) and other tasks the data warehouse team performs is prioritized. If there is not enough prioritization given to the learning and adopting the new approach, no matter how much training and coaching occurs, the team will not adopt the practice. This made the transformation duration extend beyond plan.
In the case of our previous client, the costs for the team to adopt the new approach were the dedication to the training, purchasing of Lawrence Corr’s books, the adoption of the new approach through several sprints, and costs for 2 Digineer consultants, one to be a trainer and interim Data Architect, the other as a PM.
The resource costs to the Data Warehouse team and stakeholders included:
- 2 BA’s spending several hours with a Digineer consultant per week over the course of two months to train the trainers
- 1 Data Architect and 2 Developers meeting with the trained BA’s and the Digineer consultant a few times per week over the course of two months to train them and reinforce training for the BA’s (train the trainer)
- Stakeholder adoption of their involvement in Modelstorming required some training of stakeholders. In this case, it would be an audience of selected stakeholders that would require an hour of introduction to Modelstorming.
When we left the client, the Data Warehouse team had undergone a significant transformation. But they had not just gone through a transformation of methodology, they also gained tremendous enthusiasm and drive to live the new framework. Stakeholders also experienced the impact of the transformation in that they owned the design they were building with the Data Warehouse team. Overall, the client will thrive in the long run from this new reality.
Are you interested in learning more about how event-driven analysis can transform your organization? You can continue the conversation by reaching out me at firstname.lastname@example.org.