My life as a BI consultant: Update Spring 2017

Spring 2017 in Provence (France)


Obviously I hadn’t time to write much on my blog during the last nine months. Let me share with you what topics kept me busy:

For the upcoming months I’ll be visiting and speaking at various events:

  • IBCS Annual Conference Barcelona: June 2nd, Discussion of the next version of the International Business Communication Standards.
  • TDWI 2017 Munich: June 26-28, Half-day workshop about practical gathering of requirements for a dashboard.
  • MAKEBI 2017 Zurich: July 3rd, I’ll be presenting a new keynote around alternatives to traditional estimation practices
  • BOAK 2017 Zurich: September 12th, same as with MAKEBI, I’ll be presenting a new keynote around alternatives to traditional estimation practices
  • WhereScape Test Drive / AgileBI introduction Zurich: September 13th: During this practical half-day workshop you learn hands-on how to use a DWH automation tool and you’ll get an introduction to the basics of Agile BI.
  • My personal highlight of today, I’ll be speaking during Agile Testing Days 2017: I’ll do a 2.5 hours workshop regarding the introduction of Agile BI in a sustainable way.

It would be a pleasure to meet you during one of these events – in case you’ll join, send me a little heads-up!

Last but not least, let me mention the Scrum Breakfast Club which I’m visting on a regular basis. We gather once a month using the OpenSpace format to discuss practical issue all around the application of agile methods in all kind of projects incl. Business Intelligence and Datawarehousing. The Club has chapters in Zurich, Bern as well as in Milan and Lisbon.

Advertisements

Teradata & WhereScape Test Environment in the Cloud

In this post I outline how I managed to get a cloud based training environment ready in which WhereScape RED, a data warehouse automation tool,  connects to a Teradata database test machine.

A few weeks ago I had to organize a so called “testdrive” for a local WhereScape prospect. The prospect uses a Teradata database appliance. Hence they evaluated to use WhereScape RED based on Teradata too. As a local Swiss based WhereScape partner we received a virtual machine containing a SQL Server based WhereScape RED environment. The training had to be run onsite at the customer’s location, IT-Logix provided their set of training laptops, each containing 4GB or RAM. These were my starting conditions.

First of all I thought about how to deal with Teradata for a training setup. Fortunately, Teradata provides a set of preconfigured VMs here. You can easily download them as zipped files and run it using the free VM Player.

Based on my previous experience with organizing hands-on sessions, e.g. during our local Swiss SAP BusinessObjects user group events, I wanted to use Cloudshare. This makes it much more easier (and faster!) to clone an environment for multiple training participants compared to copying tons of gigabytes to multiple laptops. In addition, the 4GB RAM wouldn’t be enough to run Teradata and WhereScape properly in a performant way. So I had two base VMs (one from WhereScape, one from Teradata) – a perfect use case to use the VM upload feature in Cloudshare for the first time.

I started with this support note which explains how to prepare your local VM and load it up to my Cloudshare FTP folder. From there you can simply add it to an environment:

01_UploadVM1

After having uploaded both VMs it looks like this in Cloudshare:

02_CloudshareEnvironment

I increased the RAM and CPU power a bit, and more important configured the network between the two machines:

Go to “Edit Environment” -> “Edit Networks”:

03_NetworkSettings

Here I had to specify to which virtual network I’d like to connect the VMs. Please keep in mind that this doesn’t provide an automatic DHCP server or similar. Either you create one within your machine or – as in my case – had to set static IPs within the individual VM (both were delivered by using a dynamic IP provided by the VM Player). Changing the IP wasn’t a big thing, neither on Windows nor on Linux.

04_TD_Setting1

But I quickly found out that the Teradata service didn’t run properly anymore afterwards.

First of all I had to create a simple test case to check if I can connect from the WhereScape VM to the Teradata machine. Besides a simple Ping (which worked) I installed the Teradata Tools & Utilities on the WhereScape machine. As I couldn’t establish a proper connection, I had to google a bit. The following article gave me the hint to add a “cop” entry to the host file:

04_TD_Setting2

After a restart of the machine, Teradata was up and running again. This you can verify with the following command “pdestate -a” by the way:

04_TD_Setting3

The next step in WhereScape was to create a new metadata repository on the Teradata database. For this I created a new schema and user in Teradata first and then created the metadata repository using the WhereScape Administrator:

06_WhereScapeSetup

In WhereScape RED I created a connection to point to the new Teradata database:

05_WhereScapeConnection

… and finally loaded a few tables from the SQL Server to Teradata:

07_WhereScape_Data

Once I finished the work, the most important step is to create a snapshot:

08_Snapshot

Based on this snapshot I finally cloned the environment for the number of participants in the testdrive with just a few clicks. After all, every participant had his own (and isolated) environment consisting of a full stack of source database (SQL Server), WhereScape and the target DWH database (Teradata).

Agile Business Intelligence Maturity Model

As outlined in my previous blog agility in business intelligence projects can’t be produced directly. Instead you should invest into professionalism, standardization and automation. In this post I’m showing an overview of concrete building blocks to support you on this way.

In my Agile Business Intelligence Maturity Model (ABIMM) I’ve collected many building blocks and arranged them in a practical sequence. An overview you can find in the following illustration:

Agile Business Intelligence Maturity Model

Agile Business Intelligence Maturity Model

We can extract a few key messages from this model:

  1. You can’t increase agility directly – you can only reduce the amount of needed upfront design. By doing this agility is increased automatically.
  2. A reduction of upfront design leads inevitably to higher risks – risks you need to deal with actively, e.g. by using a version control system or solutions for test automation (cf. my blog post here). As long as such basic infrastructure elements aren’t available you should be very cautious with introducing iterative, incremental procdures like e.g. Scrum. (A very illustrative presentation about Agility requires Safety you can find here)
  3. All beginnings are difficult: The building block “Agile Basics & Mindeset” represents an enormous hurdle in many cases. As long as an organization doesn’t experience a top down transformation towards agile values and principles (cf. e.g. the Agile Manifesto), it doesn’t make much sense to start with it bottom-up.
  4. The gulf can be overcome by buying the necessary tools for test automation, version control and training for employees. This can typically happen within the boundaries of the already existing infrastructure. But to overcome the chasm, todays often heterogenous, multi layered BI tool landscapes aren’t suited very well. That’s one reason why I’ve become a big fan of data warehouse automation and tools like WhereScape. Products like WhereScape RED institutionalize the usage of design patterns in an integrated development environment. Only for this reason e.g. refactoring on the level of the data model and hence iterative data modeling becomes feasible with realistic effort. At the same time tools like WhereScape provide you with an ultra high degree of automation for the deployment of new and changed artefacts.

A more detailed explanation of the Agile BI Maturity Model can be found in my recent article in the German TDWI journal “BI-Spektrum“.

Here you find the English translation of my article!

Many thanks to my company IT-Logix and all the great staff working with me. You are the indispensable foundation for all my BI related work. Don’t forget, you can hire me as a consultant 😉

Following you’ll find the literature list on which the different building blocks and the model itself is based on:

[AmL12] Ambler Scott W., Lines Mark: Disciplined Agile Delivery: A Practitioner’s Guide to Agile Software Delivery in the Enterprise, IBM Press, 2012

[AmS06] Ambler Scott W., Sadalage Pramod J.: Refactoring Databases: Evolutionary Database Design, Addison-Wesley Professional, 2006

[Bel] Belshee Arlo: Agile Engineering Fluency http://arlobelshee.github.io/AgileEngineeringFluency/Stages_of_practice_map.html

[BiM] Memorandum für Agile Business Intelligence: http://www.tdwi.eu/wissen/agile-bi/memorandum/

[Col12] Collier Ken: Agile Analytics, Addison-Wesley, 2012

[CoS11] Corr Lawrence, Stagnitto Jim: Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema, DecisionOne Press, 2011

[Hug12] Hughes Ralph: Agile Data Warehousing Project Management: Business Intelligence Systems Using Scrum, Morgan Kaufmann, 2012

[HuR09] Humble Jez, Russell Rolf: The Agile Maturity Model – Applied to Building and Releasing Software, http://info.thoughtworks.com/rs/thoughtworks2/images/agile_maturity_model.pdf, 2009

[Kra14] Krawatzeck Robert, Zimmer Michael, Trahasch Stephan, Gansor Tom: Agile BI ist in der Praxis angekommen, in: BI-SPEKTRUM 04/2014

[Sch13] Schweigert Tomas, Vohwinkel Detlef, Korsaa Morten, Nevalainen Risto, Biro Miklos: Agile maturity model: analysing agile maturity characteristics from the SPICE perspective, in Journal of Software: Evolution and Process, 2013 (http://www.sqs.com/de/_download/agile_maturity_wiley_2013_final.pdf)

Parts of this blog have first been published in German here.