You can’t spend long in agile project management without encountering the term “user story”. I want to use this two-part article series to summarize how the concept, which originated in software development, can be applied to analytics projects, also known as business intelligence and data warehousing.
User stories or general stories are a tool for structuring requirements. Roman Pichler summarizes their use like this:
“(User) stories are intended as a lightweight technique that allows you to move fast. They are not a specification, but a collaboration tool. Stories should never be handed off to a development team. Instead, they should be embedded in a conversation: The product owner and the team should discuss the stories together. This allows you to capture only the minimum amount of information, reduce overhead, and accelerate delivery.”
To keep my explanations a bit less abstract, I’ll introduce a simple, fictitious case study. Let’s assume that a club like TDWI, of which I am an active member, wants a new club dashboard. For this purpose, the project manager has already sketched their first ideas as a mockup:
All user stories follow a particular pattern, of which there are different basic variations. The most common pattern is as follows:
Some projects involve systems that aren’t designed for end users. Let’s assume we build a persistent staging area and a data interface from it that delivers integrated data to another system. In such cases, it seems a bit artificial to speak of “user stories”. Here’s an alternative pattern for formulating a story (source):
One of the central questions when using stories is always how they are tailored. How broad should a story be? I follow three principles:
- The story is the smallest planning unit I can use to plan a project. Of course, you can also divide a story into tasks and technical development steps. But that’s up to the people who implement it. As long as we are at the level where the product owner, and thus a specialist, is involved, we remain at the story level.
- The story covers a timespan limited to one or two working days. This includes the specification of requirements (a reminder to have a conversation about it), their technical implementation, and integrated testing. This forces us to create small stories of a more or less uniform size. This principle makes it possible to measure a flow in implementation daily (by now, it should be clear that I don’t think much of story points, but that’s another story).
- A story always has an end-to-end characteristic, and it should provide something usable in the implementation of the overall system. In analytics, this often means that a story has a data source as its start and a form of data evaluation as its end.
Let’s take a more concrete look at this using the case study. First, we focus on a specific part of the dashboard let’s call it Feature 1, where the number of participants is evaluated:
Independently of technological implementation, we can deduce two things here: a simple, analytical data model, such as a star schema, and the data sources required. From the principles formulated above, the question now arises what “end to end” means. In simple cases, it may well be that the scope of a story extends from the data source to the data model, including the actual dashboard, and that this scope can easily be implemented in one or two working days. But what if the technical development of the data source connectivity alone means several days of work? Or the calculation of a complex key figure requires several workshops?
Particularly in larger organizations, I hear people say: “This end-to-end idea is nice, but it doesn’t work for us. We prefer to write ‘technical’ stories”. This means, for example, that I as data architect model the fact table XY and event dimension, I as ETL developer load the fact table XY and the event dimension, or I as data protection officer define the row-level security for fact table XY. Although each of these examples provides a benefit to the overall system, on its own each does not add value and success is difficult to test. Another common response is: “This end-to-end story makes perfect sense, but it doesn’t work with a lead time of one to working days. We just work on a story for two or three weeks”. That’s just as problematic. If a story takes several weeks to implement, then monitoring success and progress becomes very difficult. In the worst case, we don’t realize that we’re on the wrong track after one or two days but only after three or even four weeks.
Both technical stories and long stories should be avoided. How to do this exactly is explained in the second part of this series.