You can’t spend long in agile project management without encountering the term “user story”. I want to use this two-part article series to summarize how the concept, which originated in software development, can be applied to analytics projects, also known as business intelligence and data warehousing. In the first part, I introduced the basics of stories. Now I would like to explain how we can structure or tailor stories to suit our circumstances in analytics systems and development projects and meet these three principles:
- The story is the smallest planning unit I use to plan a project.
- The story covers a timespan limited to one or two working days.
- A story always has an end-to-end characteristic. In analytics, this often means that a story has a data source as its start and a form of data evaluation as its end.
From feature to story
Let’s take a look at this graphic and go through it point by point:
On the left, we see the simplified architecture stack of an analytics solution. For requirements elicitation and subsequent agile project structuring, the three levels shown are sufficient in most cases:
- BI application: These are the most visible aspects of an analytics solution. They incorporate information products of all kinds, from reports to dashboards and data interfaces to other systems. Often, the finished product at this level is composed of data from several entities on the level below, such as data from a sales data mart and a logistics data mart.
- DWH or data basis: Here I deliberately combine the core data warehouse and data mart levels. Even if they are technically different things, a Core DWH alone hardly adds any value to the practical use of a story. It’s only through the implementation of initial key figures and the application of case-specific business rules that the data really becomes useful.
- Connectivity and infrastructure: Developing a new data source can be a costly business. The same applies to providing the initial infrastructure. Clearly a maximum implementation time for a story of one to two days requires an established infrastructure. But this also means that we need our own stories to develop this foundation.
In many cases, an analytics solution consists of all three levels. At this point, the technology plays a subordinate role. For instance, the DWH aspect might be mapped purely virtually in the BI front-end tool. Nevertheless, it must be considered equally in terms of content (cf. also the model of T-shirt size architecture). These three levels represent for me the macro variant of end to end, from the actual source system to the finished information product. At this macro level, I find it useful to speak of “features”. A feature is formulated analogously to a user story; you can find an example in the upper right corner of the graphic. A feature is often implemented within two to three weeks. But this is too long for the daily monitoring of success in a project. A feature must then be cut further into “epics” and stories on the three levels of BI application, DWH, and connectivity and infrastructure. Epics are merely a means of bracketing stories together. Let’s assume that Feature 1 in the fictitious club dashboard introduced in Part 1 can be outlined in four stories at the BI application level. Then I would gather these four stories into an epic to emphasize the coherence of the stories.
The central aspect of the division into the three architectural levels is that each level also has end-to-end characteristics, but this time at the micro level. Moreover, these three levels overlap somewhat. Let’s look at the club dashboard example and the story there: “As a TWDI back-office employee, I’d like a diagram showing the participants registered per round table in a city that I can select”. To implement this story, the underlying data model is our data source. At best, this may need to be adapted or extended in the application story. The final product is part of the finished dashboard and potentially generates immediate benefits for the end user.
At the DWH level, we build on an established interface to the source system and a functioning data infrastructure. But what does the other end look like, where we also expect some form of evaluation? We don’t need to build a complete dashboard for this either; there’s a story for that. Instead, we can generate a simple analysis on top of any data model, for instance with Excel, and make data visible. And immediately the product owner can start a conversation with the DWH developer about whether these initial results meet expectations.
This approach can even be applied at the connectivity level. Here, the starting point is the source system. In our example, it’s a fictitious, web-based system that allows either a CSV export or a web service query. In a first-user story, a CSV export is loaded into a load layer of the DWH. The result the product owner sees is not just a table definition but a simple evaluation of the raw data. This enables the product owner to detect possible data quality problems early.
As you can see from these example stories, I like to use the traditional story pattern with “I (end user) in my role as want so that ” at the feature and BI application levels. At the DWH and connectivity and infrastructure levels, I prefer an alternative pattern: <action> <result> <after/for/from/in> <object>.
Summary and outlook
In this second article on using stories in analytics, I showed how small stories can be reconciled with the demand for an end-to-end story. It’s important not to lose sight of the big picture, the feature, and to always work towards concrete visible analysis at the epic and story levels.
Even with this structure, it is still a challenge to tailor stories so that they can be implemented in one or two days. There are also many practical approaches to this—and material for another blog 😉