Testing for BI & DWH

Since ever testing is part of every IT project plan – that’s true as well for Business Intelligence (BI) & Data Warehouse (DWH) projects. The practical implementation of testing in the BI / DWH environment has confronted me with troubles in the past again and again. Often I’ve had the impression that the BI / DWH world is still back in the Stone Age regarding development processes and environments. At least it is significantly behind the maturity level I know from the software engineering domain. The below chart illustrates this gap:

rbra_testing1 (1)
Cultural differences between the software development and BI community (Source: http://agiledata.org/essays/culturalImpedanceMismatch.html)

If there is something tested at all, typically in the BI frontend area things are tested manually. In the DWH backend we see – besides manual tests – self coded test routines, e.g. in the form of stored procedures or dedicated ETL jobs. However the integration into a test case management tool and systematic evaluation of the test results doesn’t happen. This is heavily contrasting with the software engineering domain where automated regression testing combined with modern development approaches like test driven design are applied. At least for some time we find first inputs regarding BI specific testing (cf. the (German) TDWI book here). Concepts and paper are patient though. Where are we with regard to a possible tool support, namely for the area of regression tests?

Since summer 2014 we at IT-Logix are actively looking for better (tool based) solutions for BI specific testing. We do this together with the Austrian company Tricentis. Tricentis develops the Tosca product suite, one of the worldwide leading software solutions for test automation. In a first step we run a proof of concept (POC) for regression tests for BI frontend artefacts, namely typical reports. One of the architectural decisions was to use Excel and PDF export files as a base for our tests. With this choice of a generic file interface the efforts to develop BI product specific tests were omitted. And this way we reduced the implementation effort to about two days in the POC. The goal was to run “Before-After” tests in batch mode. We took 20 reports for the POC case (these were actually SAP BusinessObjects Web Intelligence reports, but you can imagine whatever tool you like as long as you can export to PDF and / or Excel). A current version of the PDF or Excel output of the report is compared with a corresponding reference file. Typical real life situations where you can use this scenario are:

  • recurringly scheduled regression tests to monitor side effects of ongoing DWH changes: The reference files are created somewhen e.g. after a successful release of the DWH. Imagine there are ongoing change requests on the level of your DWH. Then you want to make sure these changes only impact the reports where a change is expected. To make sure all the rest of your reports aren’t concerned by any side effects, you now run your regression tests e.g. every weekend and compare the hereby produced files with the reference files.
  • BI platform migration projects: If you run a migration project for example to migrate your SAP BusinessObjects XI 3.1 installation to 4.1, you’ll want to make sure reports still work and look the same in 4.1 as they did in XI 3.1. In this case you create the reference files in XI 3.1 and compare them with the ones from 4.1. (As the export drivers vary between the two versions, especially the Excel exports are not very useful for this use case. Still, PDF worked pretty fine in my experience)
  • Database migration projects: If you run a database migration project for example migrating all your Oracle databases to Teradata or SAP HANA, then you want to make sure all of your reports show still the correct data (or at least the data as was shown with the original datasource…)
rbra_testing1 (3)
Sample configuration of a test case template using the GUI of Tosca (Source: IT-Logix POC)

Tosca searches for the differences between the two files. For Excel this happens on a cell by cell basis, for PDF we used a text based approach as well as an image compare approach.

rbra_testing1 (2)
Depending on the chosen test mode the differences can be visualized differently (Source: IT-Logix POC)

Using the solution implemented during the POC we could see very quickly which reports were different in their current state compared to the reference state.

Another important aspect of the POC was the scalability of the solution approach as I work primarily with (large) enterprise customers. If I have not only 20 but hundreds of reports (and therefore test cases), I have to prioritize and manage the creation, execution and error analysis of these test cases somehow. Tosca helps with the feature to model business requirements and to connect them with the test cases. Based on that we can derive and report classical test metrics like test case coverage or test execution rate.

rbra_testing1 (1)
Requirements and test cases are tightly related (Source: IT-Logix POC)

In my eyes an infrastructure like Tosca is a basic requirement to systematically increase and keep the quality in BI / DWH systems. In addition advanced methods like test driven development are only adaptable to BI / DWH undertakings if the necessary infrastructure for test automation is available.

In this blog post I’ve shown a first, rudimentary solution for regression tests for BI frontend tools. In a next article I’ll show the possibilities to implement regression test for DWH backend components.

Event recommendation: Learn about a real life scenario to run a SAP BusinessObjects migration project in an agile manner. Hence test automation is key and explained in some more details. Join me during sapInsider’s BI2015 at Nice by mid of June. Find more information here.

(This blog post was first published by me in German here)

How to speed up connection between Translation Manager, Universe Designer and CMS

At a recent customer project my teammate and I were facing an issue with importing Webi documents and universes into Translation Manager (all on BO4.1 SP4) getting the following error message:

org.apache.axis2.AxisFault: Unable to find servers in CMS, servers could be down or disabled by the administrator (FWN 01014)

There is a KB article describing this error and a solution: http://service.sap.com/sap/support/notes/1879675

One hand we followed the instructions in the KB article and specified the hostname for each server. Although our server is multihomed (that means has more than one network interfaces) we didn’t thought about this before hand because the second network interface goes into a backup LAN and is never used for communication with e.g. the client tools. In addition everything worked fine so far – until we got the issue with Translation Manager. Still, just setting the hostname was not enough. On the server side there is the Windows firewall enabled, therefore we had to assign static request ports to several services. We already did so before our issue with the Translation Manager for the CMS, Input & Output FRS, all Webi Processing Server. We used TCPView to analyze on which ports Translation Manager was opening connections. As long as there were requests on ports which we didn’t specify in the CMC (some random port number e.g. 56487) we narrowed down all the services which Translation Manager establishes a connection. We had to specify a port for all of the following servers:

  • The Adaptive Processing Server (APS) hosting the DSL-Bridge Service
  • The APS hosting the Client Auditing Proxy Service
  • The APS hosting the Search Index Service
  • The APS hosting the Translation Service
  • The WACS (Web Application Container Server)

Besides the issue with Translation Manager we had another issue with creating new universe connections in the Universe Design Tool. When creating a new connection we had to wait up to 30 minutes ( ! ) to get to the wizard page where you can select from the list of available database drivers. Still, after 30 minutes everything worked fine and could create the connection successfully. Based on our experience with Translation Manager we run again TCPView and found that we need to assign a port number to both connection servers (32 and 64 bit) in CMC. Having done this, creating a new connection now works without any waiting time.

After all: If you have firewalls between your BO server and client tools, just assign a port to all servers available and open the port on the firewall (the only exception might be for the Event Server as I’m really not aware of any communication between this server and a component outside the BO server)

A (Webi) dashboard built by a business (power) user

This blog post is inspired by a recent customer request to challenge their decision to use Design Studio for some “dashboard requirements”. Showing how you can create a dashboard in Webi doesn’t mean I told the customer not to use Design Studio. Much more it is to show that finally a dashboard as well as every other type of BI front end solution is made up of requirements and not primarily by the tool you build the solution. Please refer to my Generic Tool Selection Process for more details as well as my post regarding BI specific requirements engineering.

Having said this, let’s have a look at how we can use latest Webi 4.1 features to quickly build an interactive dashboard without the need of (much) scripting. First of all here is what the final result looks like:

01_DashboardOverview1

You can select values from the left side bar (Product Lines), you can select States by directly clicking into the table and you can switch from the bar chart to a line chart. Here you see it in action:

The first step to achieve this, is to create the basic table and the two charts. Until the dynamic switch is implemented, I placed them side by side. Next add a simple input control in the left side bar:

02_SimpleInputControl 03_SimpleInputControlDepend

Next thing is to define the table as an additional input control – right click the table and choose “Linking” and “Add Element Link”,  choose the two chart objects as dependencies:

04_TableAsInputControl 05_TableAsInputControlDepend

Next we need to create the “switch” to toggle the two charts. As I would like to position this switch at the top right corner of the chart, I again use a table input control. To generate the two necessary table values (namely “Bar Chart” and “Line Chart”) I prepared a simple Excel spreadsheet:

08_ExcelContent

In 4.1 you can now finally upload this sheet directly into the BO repository:

07_UploadExcel

If you need to update the Excel sheet later on, this is now feasible as well:

09_UploadExcelReplace

Finally, in Webi add the Excel sheet as a second query:

10_ExcelQuery    10_ExcelQueryDetails

In the report we need now two tables: A visible one to represent the chart switch and a (hidden – see the “Hide always” option) dummy table to act as a dependency for the first:

13_HiddenDummyTable  12_HideDummyTable

The most tricky part is to create a variable to retrieve the selected value:

15_VarSelectedChartType

Here the formula for copy / paste:

=If( Pos(ReportFilterSummary(“Dashboard”);”Chart Type Equal “) > 0)
Then Substr(ReportFilterSummary(“Dashboard”);Pos(ReportFilterSummary(“Dashboard”);”Chart Type Equal “) + Length(“Chart Type Equal “);999)
Else “Bar Chart”

(The idea for this formula I grabed from David Lai’s Blog here)

Finally you need to configure the hide formula for both charts:

16_DynamicallyHideChart

That’s it.

Conclusion

Positive: I’m not too technical anymore (I do more paperwork than I wish sometimes…). Therefore I don’t consider me a “developer” and I like solutions for the so called “business (power) user” more and more. Therefore I like Webi. It took me about 60 minutes to figure out how to create this kind of interactive dashboard. I didn’t need to install anything – I could do everything web based. Except for one single formula (which I didn’t need to write myself)  I could click together the above sample. And I dare to say it looks like some kind of a dashboard 🙂 In addition I have all the basic features of Webi like a broad range of data source support, plenty of export possibilities, Office integration and so on. Even integrating an Excel spreadsheet as a data source is now finally a no-brainer.

Negative: Clearly, Webi is not a “design tool”. For example I wasn’t able to show icons for my chart switch instead of the text lables. Putting a background image to the table doesn’t work well if the table is used as input control. When I discussed this prototype with the customer they also mentioned that there are still too many options end users might get confused with (e.g. that there is a “filter” section showing whether the Bar Chart or the Line Chart value is chosen). In Webi you can’t change that. Toolbars, tabs etc. are just there where they are. Live with it or choose a different tool.

Bottom line: Have a look at my Generic Tool Selection Process and the mentioned hands-on test. The above example is exactly what I mean with this: Create a functional prototype in one or two tools and then do a fact based decision depending on your requirements and end user expectations.

Important remark: This post focused on the technical aspect of the dashboard. The visual representation doesn’t yet fit to best practices mentioned in my earlier articels (e.g. about SUCCESS) In a next blog post I will outline how to optimize the existing dashboard in this regard.

Join my teammate Kristof Gramm during sapInsider’s BI2015 conference at Nice (June 16-18): He will go into much more details about how you can use Web Intelligence as a dashboard tool for business users. Use this link to see more infos and save 300€ on your conference registration!

The Rule of Thumb for BOBJ Tool Selection

What is the right SAP BusinessObjects frontend for a given situation? A question I’m asked nearly every day. When I was confronted first with this topic  a few years ago the taken approach was a highly sophisticated Excel spreadsheet in order to assess all available BOBJ tools based on a feature list. The only problem was: At the bottom line there was never a clear winner. Next approach were the famous decision trees like the following:

BO-Tool-DecisionTree

Not bad as a first guess. And in an ideal world where the basic functionality would be the same for all BOBJ tools such a tree could work indeed. But given the situation that even today – nearly ten year after the aquisition of Crystal by BO – support for universes is still not exactly the same in Webi, Crystal Reports and Xcelsius (aka Dashboards) and especilly the maturity of a tool or a sub component of it is vastly different, there is no clever way to tell you which tool to use for which purpose.

Although you can’t give a distinct answer to the question “which tool to use for what”, I’m convinced that the following rule of thumb will be valid in most situations and for a majority of organisations – the only assumption is that there is no limitation out of licensing. That means I assume you have a license for all or at least the most important frontend tools. The idea behind this rule is that a priority rating is more helpful than a feature or use case driven decision tree.

Here is my rule of thumb:

  1. Try it with Web Intelligence
  2. If Webi didn’t work, try it with Crystal Reports
  3. If Crystal Reports didn’t work, try it with one of the “niche” tools

Let me share some thoughts about this priority list:

Why should we start with Web Intelligence? There are various reasons for this:

  • From a features perspective Web Intelligence provides the most widest range in the BOBJ tool suite. You can use Webi for creating classical standard reports, you can use it for dashboard like applications (think about Input Controls and the ease of use regarding drilling – e.g. compared to Xcelsius…), you can use it for self-service reporting, you can use it as a data pump using XLSX export or interface to other applications using BI Web Services etc.
  • From a maturity perspective it is one of the most stable and mature applications in the BOBJ world. I tell you this as an native “Crystal guy”. But whereas Crystal Reports 2011 runs stable the same way as it did for the last decade, the new Crystal Reports for Enterprise is just crap compared to both, the legacy CR and Webi.
  • From a data source perspective: Webi is the only tool which fully supports all kind of Universe stuff. I’ve never heard of any limitation that Webi would not support something what you can do in a Universe (by design). But let me compare this to Crystal Reports: On one hand you can use only UNX universes in CR4Ent, on the other not all type of queries are supported. Crystal still has the limitation that if a universe query results in multiple SQL statements it fails to handle it as there is no local “micro cube” as with Webi. Of course this whole argument implies that we value a “common semantic layer” to be of high “added value” to an organization and therefore should be supported in its full scope. But there is even more to add: Webi handles not only multiple SQL result sets per query, it can also leverage multiple queries and easily join them. Although I’m not a friend of “merged dimensions”, there are many situations where this capability is the only work around to get the job done at the end of the day (and not three monthes later when the data finally arrived in the DWH…). No clever way to do this in Crystal Reports or Xcelsius directly.
  • From an SAP BW perspective: Two or three years ago we had to decide for Crystal Reports often because of its better connectivity to SAP BW and all around it with hierarchy handling etc. These days are “passé”. My most recent experience with Webi using the BICS interface are very promising. Totally in contrast with CR4Ent which crashes regularly, even with the latest patch level.
  • From a usability perspective: Although SAP currently tries to position Webi to be the tool where business users develop the reports, I think its usability is equivalently valubale for IT folks too. Report development is quick and straight forward – once you’ve got used to the ribbon style menues 😉
  • From an installation footprint perspective: Given the situation that SAP releases new patches nearly every third or four week, patching client installations is an nightmare. The more valuable are fully web based deployment scenarios. Therefore once again, Webi is the favorite.

Still, Web Intelligence has some short comings. That’s why you should evaluate Crystal Reports in a second instance:

  • One of the major differentiators between Crystal Reports and all the other frontend tools is Conditional Formatting. As you may know Crystal Reports has a powerful formula language integrated. This formula language can be used to control neary every property you can set in Crystal Reports. This way you can implement what I call “guided interactivity” at its best: Let the end user choose some parameter values and use these values to control both, the data in the report but especially the layout too. The typical use here is: A customer wants to build 10 similar reports. They are not exactly same regarding the layout, but similar. For example, in Webi there is no straight forward way to show conditionally show or hide some parts of the report. In Crystal Reports such a thing is a no-brainer.
  • Interactive / proactive Alerts: As of today, only Crystal Reports based alerts can be used to send an email notification if they are triggered.
  • Export formats: Crystal Reports has a multitude of available export formats, including Word or XML, which aren’t available in any of the other tools.
  • Hierarchical Grouping for relational data sources: Crystal Reports can dynamically resolve a Child-Id-to-Parent-Id relationship and apply calculations over such a hierarchy.

But before you choose Crystal Reports remember there are two versions of Crystal Reports: The legacy Crystal Reports 2011 and Crystal Reports for Enterprise. The first one is mature and stable, but does not contain new features introduced only to CR4Ent. On the other hand, CR4Ent is a de facto “1.x” product regarding its code maturity. For now I simply cannot recommend to use it as your major reporting tool without intensive testing of your own use cases in your environment. On the other hand – depending on your situation – the legacy Crystal Reports does not support UNX universes at all nor does it support UNV universes as you’d expect it coming from Webi.

What about all the other tools? I call them “niche tools”. This is due to the fact that all of them have quite a narrow scope of application compared to the “generalists” Webi and Crystal, let me name a few:

  • SAP Visual Intelligence: This is a great tool for ad-hoc-analysis. But that’s it. No way (yet) to publish documents online (except over Explorer), schedule them or create more sophisticated standard reports.
  • Explorer: Not the most mature product, especilly in the context of SAP BW and BWA as a datasouce… In general, Explorer is nice for “standard” visualizations. But have you ever tried to customize even basic elements of these charts? Or have you tried to add a simple table into an Exploration View? Or export an Exploration View as a whole? As of today these basic things seem to be impossible…
  • Analysis, Edition for OLAP: Limited to OLAP data sources, no clever integration into scheduling, publishing etc.
  • Analysis, Edition for Microsoft Office: Only BW support…
  • Dashboards / Xcelsius: Limited capabilities in terms of data volume that can be processed, no straight forward way to realize drill downs, no common export formats, no full Universe support, no scheduling capabilities…
  • Design Studio: Not usable for productive environements in the current version 1.0, and even for subsequent versions I’m very sceptical… In addition the scope of the tool is focused on BI App development which as such is clearly a niche.

This doesn’t mean that these tools are not valuable in the context of specific requirements. But assuming that there is a value in reducing the number of used and supported tools to a minimum, these tools should be chosen only after having evaluated Webi and Crystal beforehand. According to my experience chances are quite high that your requirements can be covered by one of these two tools.

What is your experience with tool selection? Would you agree with my rule of thumb? Anything I missed? Looking forward to reading your comments!

Using HANA on Cloudshare Part 1: Setup connectivity

Hi everybody

As you may know I’m a great fan of Cloudshare, you’ll find my previous post about testing in the cloud here. So far we had to use “traditional” databases like SQL Server or Oracle to work in Cloudshare. Finally SAP managed to get its new baby – HANA – to various cloud platforms, including Cloudshare –> see here for an overview. They provide you with a regular Cloudshare environment with 24GB RAM with two machines, the HANA server on Linux and a Win7 client with HANA Studio – you can register for the 30 day trial sponsored by SAP here:

01_Environment

So far so good. But what is the value of an isolated HANA database? It’s pretty small. Usually in Cloudshare, an “environment” is quite isolated network wise, therefore my first idea was to extend the 24GB RAM and add another machine, e.g. with BO4 installed. Unfortunately the maximum RAM per environment is 32GB. Even more sad that BO4 doesn’t really work with 8GB of RAM… What to do? A first inquiry with Cloudshare showed that obviously the HANA environment is somewhat special. After some try and error I found how you can easily connect to your HANA environment both from your local client or another Cloudshare environment. Let me share my findings with you in this blog. As you can read in the title I plan some other posts, especially about how to fill data into HANA using SAP BO Data Services.

First thing we need to do is creating a static vanity URL for the Cloudshare machine. For this switch from “My environments” to “My Account”. There go to “Vanity URLs” and specify whatever you want – the only thing you can’t take anymore is hana 😉

02_VanityURL

As you can see, there are two public URLs available now: the regular with .cld.sr and a second one vm.cld.sr. In the background these two URLs are mapped to different public IPs. Whereas the first one gives you the default access to ports like 80, 8080 etc. the second one seems to redirect also HANA specific ports like 30015. Therefore you don’t need any kind of port forwarding as suggested in forum threads like here. Don’t forget to click “Save changes” at the end of the page.

You can now do a first test within the HANA Studio on Cloudshare itself – add a new system and use <your-name>.vm.cld.sr:

03_AddSystem104_AddSystem2

05_AddSystem306_AddSystem4

As you can see in the last screenshot, the only “issue” with the connectivity is, that somehow the status information of the HANA server cannot be retrieved, therefore you don’t get the green light but a yellow one. But don’t worry, everything works fine.

The next and so far final part is to connect from another Cloudshare environment, e.g. using the Information Design Tool:

Create a new relational connection using the HANA JDBC driver:

07_AddConnection1  08_AddConnection2

And finally you can start to build your data foundation based on this connection:

09_CreateDF

Hope this helps. Wish you a lot of fun playing around with HANA on cloudshare!

How to configure BO Explorer 4.0 to run with BWA

Hi everybody

This is just a quick note about my findings how to get BO Explorer 4.0 to connect to SAP’s Business Warehouse Accelerator (BWA). Besides the possibility of a relational UNX universe based connectivity, this is the only way in BO 4.0 to connect Explorer to an SAP BW.

As it seems many others have the same question – while reading this blog please keep in mind it is not a well researched articel, it is just a write-down of some current findings. They might be incomplete and I’m happy to see comments from your side about what your experience is.

If you look into the official admin guide of BO Explorer you’ll find only the BO side configurations. No word about what’s necessary to configure on the BW side. That’s why so many of you (including myself until a few days ago) never saw this “BWA node” in BO Explorer. For me the key was to find the following documentation:

http://help.sap.com/saphelp_nw73ehp1/helpdata/en/4b/e2ff960ff91323e10000000a15822b/content.htm

Basically I had to configure two main things on the BW side to get the BWA connecting to BO Explorer:

  1. “start program RSDDTREX_ADMIN_MAINTAIN In ABAP Editor (transaction SE38) with OBJECT = ‘POLESTAR_SYSTEM’ and VALUE = ‘2’.”
  2. “Enter transaction code RSDDTPS in the input field. The Explorer Object Selection screen appears. On the left of the screen, there is a list of all BW objects that can be activated for display in SAP BusinessObjects Explorer. The objects are displayed under the InfoAreas that they belong to. The icon in the Explorer Status column indicates that the corresponding object has already been activated.” (more infos here)

After having applied these and some other properties described in the documentation above, some restarts of Tomcat and the BO Explorer services we finally could access the BWA indexes from within Explorer.

A helpful page is the following wiki (although I couldn’t find the info above on it): http://wiki.sdn.sap.com/wiki/display/BOBJ/BWA+and+Explorer+homepage

And a last remark: I got several times a Tomcat stack error including the following statement:

com.businessobjects.datadiscovery.web.beans.DataDiscoveryWebSession.initializeFromRequest

I first thought this might be due to some misconfiguration I did by chance when trying to setup the BWA thing. It was not. It’s some kind of login / SSO problem. Simply close all browser instances (e.g. Internet Explorer) and login to BI Launchpad again and then open Explorer. It should work again.

How to promote a Crystal Reports with Dynamic Cascading Prompts in BI4

This (and most probably some future) blog post will detail on my experience using Promotion Management (LCM) in BusinessObjects release 4.0. The following explanations are mostly based on the description I’ve just handed in to SAP support. I will do my best to keep this post current regarding answers from SAP support…
Infrastructure: I did all my testes on Cloudshare (see my blog here). Currently using BI 4.0 SP4 Patch 4.

Source-Sytem: Cloudsrv012
Target-System: Cloudsrv016

Promotion Management is primarily used on the Source System.

Update from SAP support

SAP support was quite quick and told me that the issue described in this post will be fixed in patch 4.7 (including the problem of promoting BusinessViews residing in subfolders)

Terms

Dynamic Cascading Prompt (DCP): A parameter object in Crystal Reports 2011 which contains a dynamic list of value (LOV).

List of Value (LOV): List of Value object based on a Business View (BV). Can be created manually in the BV-Manager.

Business View (BV): Business View’s are created in the BV-Manager (which is part of the Client Tools setup of the BI Platform). BusinessViews are based on Business Elements. Business Elements are based on Data Foundation objects. And Data Foundation Objects are based on Data Connection objects. These items are generally considerd as “Repository Objects” (at least in XI 3.1 Import Wizard this was the case).

Initial Setup

Create a LOV with its underlying BVs based on the Xtreme database (using ODBC-Connection to local Access file). Save them in a subfolder (in my example “rbra_Test”):

Create a simple Crystal Report (in CR 2011) containing a parameter with a DCP:

Save this report in the source system. In the BI Launchpad the parameter looks like this:

Problem Description

Goal: Simply promote the above created report from source to target system using promotion management.

Steps taken:

Create new promotion job in Source System including all dependencies:

Then promote:

Result: Partial Success:

My guess: The problem is that the BV-objects are in a subfolder. Therefore, I move the BV-objects in source system to root folder:

Report still works in source system:

Take the same Promotion Job as before and refresh dependencies – no Sub Folder in Business View Branch is shown anymore:

Promote again:

^

Now it shows Success:

It looks like a success in BV Manager too:

and also in Crystal Reports:

BUT: If you open report in BI Launchpad, you don’t see any List of Values:

Tested Workarounds

Promote BusinessViews separately (not working)

I tried to promote BusinessViews and LOV objects separately from the report. I have the same issue regarding storing repository objects in subfolders. Besides this I found the following:

  • Just promoting the BusinessView and underlying objects works fine according to Promotion Management. But if you look into BV-Manager you’ll get errors like this
  • Promotion Management doesn’t allow to select LOV objects separately.
  • If you then promote the same Crystal Reports containing the DCP but do NOT select the dependencies, all the BusinessView objects (and LOV objects) are promoted anyway and break the functioning of the BusinessView and LOV in the target system. Currently we couldn’t find a way to promote a Crystal Reports with DCP without automatically promoting all dependencies and therefore break the target system.

Remove DCP, export / import LOV using BV-Manager (not working)

In order to escape the circumstance that Promotion Management automatically promotes DCP objects etc. (see point above) I tried the following:

  • In the source system, set the Crystal Reports parameter to a Static list of value and save the report.
  • Promote it – no repository objects are promoted.
  • In order to “promote” LOV objects independent from report we used the option to export BV and LOV definitions in the BusinessView Manager.
  • We imported the LOV object into the target system using the import option of the BusinessView Manager.
  • Reset the static prompt to the imported LOV.
  • Result:

    The LOV of the second level doesn’t work.

Remove DCP, export / import BV, recreate LOV (working)

  • In the source system, set the Crystal Reports parameter to a Static list of value and save the report.
  • Promote it – no repository objects are promoted.
  • Promote BusinessView only using export / import in BusinessView Manager
    Using Promotion Management doesn’t work properly! (see errors in BV-Manager above)
  • Recreate LOV objects manually in target system
  • Reset the static prompt to the newly created LOV.

Although this is NOT what I expect from SAP in terms of a properly working software – at least these final steps lead to a working solution without too much of manual recreation of repository and report objects!

For all SAP internal guys if you want to track (and support me ;-): The message number with the same case description as above is 971741 / 2012. I will open up some more cases as the things shown above is just the top of the iceberg of what doesn’t work properly in Promotion Management.

Desktop Intelligence to connect again to BO 4.1

Let me share an interesting finding with you, especially those who were not attending the recent SAP BO User Conference in Orlando. When I first saw the following pic I thought this must be another joke about Deski:

(Source: pic.twitter.com/Wckjr4HX)

During the recent BO user conference (the BusinessObjects Arbeitskreis / BOAK) hosted by IT-Logix in Zurich / Switzerland I mentioned this and got numerous requests to look for more details. Obviously many of Swiss BOBJ customers still use Deski and it is quite a show stopper to them regarding any upcoming migration to BO 4. Yet this morning Blair Wheadon from SAP confirmed the slide above was no joke but serious:

As you can see the Desktop Intelligence Compatibility Pack (DCP) should be available in BO 4.1, the next minor release (don’t confuse this with patch 4.1 which equals to BO 4.0.4.1). So far I couldn’t find any rumours when BO 4.1 will be available. Feel free to write your estimation by adding a comment!

Update: I’ve just found more information here:

scn.sap.com/docs/DOC-31798

Great thoughts by Eric Vallo here: bit.ly/NEi3b9

Backup & Recovery in BO 4.x

This post is dedicated to the available means of backup & recovery in SAP BusinessObjects BI 4.x. There are several changes compared to the previous version XI 3.1 including some literally missing functionality.

The recovery scenario: Partial restore of report and universe objects

In my eyes the typical recovery scneario is a partial restore. It happens quite quickly that you either delete a folder with a whole bunch of reports or that you want to revert a change in a report or universe development. Especially if we consider the ad-hoc reporting capabilities of Web Intelligence you probably don’t have a local copy of the corresponding report. In addition people which do any mistake leading to a recovery procedure tend to notice that they did such a mistake only with a certain gap in time, this means they request the recovery e.g. of a given folder not immediately after its deletion but perhaps two weeks later when they realize they deleted some reports too much. In the meanwhile the system might have been used heavily, that’s why a full recovery of the system itself is not really an option. What you need in such a situation is the possibility to recover only selected objects from a backup set to the original system. In this blog I will concentrate on this scenario. I use “original” system as a term to identify the system on which I take the backup and to which I want to recover something back.

The available possibilities in BO 4.x

There are three major approaches in taking a backup of BO 4.x and recover partial content:

  1. Create some kind of BIAR file (multiple options available, see below) and try to recover selected elements back to the original system.
  2. Do a full backup, restore the full backup to a separate BO “recovery” system and finally use LCM to “promote” selected objects back to the original system.
  3. Use a professional backup & recovery solution like 360Plus from GB and Smith

Let me evaluate the above approaches in the next few sections.

The BIAR approach

The BO Admin Guide states in section 12.1.1.3 (page 466):

It is recommended that you use the Lifecycle management console for SAP BusinessObjects Business Intelligence platform to regularly back up your Business Intelligence content, such as reports, users and groups, and universes. Having current backups of your content makes it possible to restore your Business Intelligence without having to restore your entire system or your server settings.

Whoever wrote this sentence at SAP doesn’t seem to have either any concrete experience with LCM or not a clear idea what a backup & recovery tool should fullfifl in practice. Respectively let’s have a look at just any given freeware to backup your Windows files. Therefore to point this out right at the beginning: Keep your hands off in trusting LCM as your one and only backup solution for BO. LCM is a tool to promote (or in the SAP jargon ‘transport’) objects from one environment to another. LCM was never made to be a backup solution. Let me explain in some more details:

The preferred way to take a backup using LCM is exporting a LCM job into a LCMBIAR file. Finally with FP3 / SP04 you can now schedule the export to such a file. But there are some critical short comings with this (as of SP04 Patch 1; anyone having differing experience with a higher patch level please comment below!):

  • reimporting the LCMBIAR file to the original system on which you created the file will fail as soon as you delete the original LCM job. What real backup solution makes itself depending on the job object creating the backup set?
  • whenever you import a BIAR file you don’t have an option to select / unselect objects to restore. There is only black or white: Either you import all the contents from your (LCM)BIAR file or nothing.
  • LCMBIAR files do not save your successful instances. Only recurring instances are backed up. But by the way you cannot decide whether to restore recurring instances or not, as mentioned before you have to restore everything belonging to the BIAR file.

A next approach in using BIAR files is to use the new Upgrade Management Tool or the “legacy” biarengine.jar. The good news here are that LCM finally is capable to import regular BIAR files which were created by these two means. The following things should be considered:

  • In contrast to LCMBIAR files, regular BIAR files can be imported without any dependancy to any LCM job.
  • The Upgrade Management Tool as well as the biarengine.jar takes a backup of both, recurring as well as successful report instances.
  • Unfortunately SAP was so stupid – sorry to say it like this, but I couldn’t find any other term to express my feelings about this situation – to remove (or just not allow…) the possibility to import a BIAR file of the same software version using the Upgrade Management Tool. In XI 3.1 this became quite standard during a recovery procedure to load the BIAR file using Import Wizard and then select only the objects you need to recover. In combination with the short coming of LCM not to be able to select individual objects this is a real sad thing (#factoryofsadness …). Dear SAP: Just give us back basic functionalities like restoring selectively either using Upgrade Management Tool or LCM!

For those interested in the biarengine.jar – I couldn’t find any hints on it in the BI4 documentation, so I took the admin guide from XI 3.1 and it seems that everything still works as before (for more detailed infos see this blog):

First of all you need a properties file to specify what you want to be backed up:

exportBiarLocation=C:/temp/BiarEngineBackup.biar
action=exportXML
userName=Administrator
password=<your password>
CMS=cloudsrv012:6400
authentication=secEnterprise
exportDependencies=true
exportQuery=select * from ci_infoobjects where si_parent_folder = <your own id or query> OR SI_ID = <your own id or query>

Save these lines of text in a file, e.g. mybackup.properties. After all you can execute the following commands on the command line or in a batch file (replace C:\BOE4 etc. with your own BO install path):

cd “C:\BOE40\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\win32_x86\jre\bin”
java -jar “C:\BOE40\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\java\lib\biarengine.jar” C:\Temp\mybackup.properties

You can use either the biarengine or LCM to restore content to the original system. As you can only restore the full BIAR file, I recommend to have a dedicated recovery or sandbox system in place where you can import the BIAR file as such and then use LCM to restore only what you need back to the orginal system. Such a dedicated system you need anyway for the second major approach, restoring objects from a full backup.

The full backup / restore approach

As long as you have a dedicated system available to “mount” the full backup into a running BO system this appraoch is quite straight forward and nothing to be afraid of (as long as you know what you do ;-)). The following high-level steps will guide you through the recovery process:

  1. Take a full backup of your original system on a regular basis. This includes at least a backup of your CMS system / repository database, the FileStore folder(s). As of FP3/SP04 SAP added an official “hot backup” option (see the “Settings” area in CMC), therefore you don’t need to shutdown your BO system to take the backup. Just define a time window in which you create both, first the backup of your system database and then the backup of the FileStore. In addition to system database and FileStore, please note your Cluster Key and Administrator password from the original system!
  2. Prepare the Recovery-System: I assume you have an already installed “recovery” system. This can be a sandbox or as well e.g. a QA system you want to temporarily use as your recovery system. Stop all existing SIA and Tomcat services on the «Recovery» system. Have a look into Task Manager and make sure that all CMS.exe and sia.exe processes have been stopped.
  3. Restore the System-DB: Restore the backup of your «original» system database to a new, empty database / schema. After restore, execute the following SQL statement on this restored DB to remove all server entries: Delete from CMS_INFOOBJECTS7 where ParentID=16
  4. Restore the FileStore: On the «Recovery» system rename the existing FileStore folder to «FileStore_orig». Restore the FileStore from «Orginal» to the «Recovery» system into its original location.
  5. Create ODBC source: In case your recovered system DB is hosted on a SQL server, create a 64bit ODBC source to it on the «Recovery» system.
  6. Create Recovery SIA (1/2): On the «Recovery» system, create a new SIA with a new CMS. Point the CMS to the recovered system database (probably using the ODBC source created in the previous step). Select the «Use a temporary CMS» option.
  7. Create Recovery SIA (2/2): Once the new SIA is added, change the Cluster name from the orginal name to a new name, e.g. «Recovery». Start the newly created SIA and check in Task Manager if CMS starts up and keeps up running. Then stop the SIA again.
    (if you want you can combine step 6 and 7 and add only one additional SIA)
  8. Create second SIA to add regular servers: Add a second SIA including regular servers, you can even add a second CMS. Start this SIA and Tomcat. Login to CMC on the «Recovery» system and check in the Servers area if all expected servers are up and running.
  9. Verify File Repository Servers: Check if the file path indicated in the Properties of the Input and Output File Repository Servers correspond to the location where the FileStore has been recovered.
  10. Run the Repository Diagnostic Tool: Run the Repository Diagnostic Tool in order to remove any inconsistencies between File Repository Servers and (recovered) system database.
    (replace C:\BOE4\ with your own BO install path; more info about the command line parameters you’ll find in the BO admin guide):
    cd “C:\BOE4\SAP BusinessObjects Enterprise XI 4.0\win64_x64”
    reposcan.exe -dbdriver sqlserverdatabasesubsystem -connect “UID=sa;PWD=<password>;DSN=<ODBC_Name>” -dbkey <cluster key> -inputfrsdir “C:\BOE4\SAP BusinessObjects Enterprise XI 4.0\FileStore\Input” -outputfrsdir “C:\BOE4\SAP BusinessObjects Enterprise XI 4.0\FileStore\Output”
  11. Do a «selective restore» from the recovery to the original system using LCM (or one of the other ways explained above, mostly depending whether you need to recover report instances or not)
  12. Recreate original settings on Recovery system: If you don’t need the «Recovery» system anymore, you can reset everything to match the original settings. For this simply stop the created SIAs and either set their startup mode to disabled or delete the SIAs from the system entierly (a practical how-to you’ll find here). Rename your FileStore on the Recovery system from “FileStore_orig” back to FileStore. This means you need to either delete the recovered FileStore folder or give it another name before. In addition you can remove the recovered database (schema).

Once you excerised this process a few times it will serve you as a reliable way to recover (partial) elements in a reasonable amount of time. But still it is not the “elegant” way to go. And therefore I would like to introduce you to my third and favored major approach. What SAP fails to deliver is usually deliverd by one of the add-on providers.

The professional approach

As a professional BO administrator I like professional tools. 360Plus is one of my favorite tools, not only regarding backup & recovery. But this is one of the major reasons why I recommend this solution. 360Plus doesn’t keep any separate information outside the regular BO system database, it’s just an alternative view to its content in addition to the CMC.

Let the pictures speak for themselves:

First of all you need to create a backup job in the web based interface of 360Plus, you can choose from various object types. In addition you can choose whether to include subfolders, report instances or Favorites folders in case you choose groups and users:

You can schedule this job to run “now” or at a later point in time. By the way: All the jobs scheduled with 360Plus can be triggered by an external scheduler like $Universe etc.

Once having executed the backup job you’ll find a new entry in the context menu of any given folder or document:

And for folders which do not exist anymore completely you’ll find the Trash Bin icon:

After all you can choose from available recovery options as you are used to from any other professional backup & recovery solution:

That’s it. The only thing you need to do in addition is to save the 360Plus file folder on your BO server by a regular file backup tool.

A more detailed overview of the 360-approach you can find here

Are you dissatisfied with the existing backup & recovery capabilities in BO 4.x too? Or do you see different ways of improving this process? Let me and other knows and write a comment! Thanks for your participation!

BO 4.0 FP3: get eFashion and other MS Access datasources working

I’ve just noticed a problem with our IT-Logix Migration Assessment Environment. The problem is with eFashion and other MS Access based demo databases, namely that you get the following error in (online) Webi, both on BO 4.0 SP02 as well as with FP3 – due to 64bit connectivity problems:

You don’t get the error in Webi Rich Client usually. In this post I will quickly outline the reasons for this and how to solve it:

First of all: Others got these errors too:

http://scn.sap.com/thread/2118132

http://scn.sap.com/thread/2043784

The answers from SAP (namely http://scn.sap.com/people/henry.banks) are not really satisfying. Of course it is not very clever to use Access as a demo datasource – but why SAP then provides these (access based) samples in BO 4.0 and not e.g. within the database they include within the setup? Anyway, there are three options you can choose:

  1. Move your efashion and other MS Access databases to a “real” database like SQL Server (Express), MySQL etc. It must be just accessible by 32 AND 64 bit drivers.
  2. Migrate to BO4 FP3 – and read the rest of the blog of how you can get Access databases running…
  3. If you are on BO4 SP2 – sorry, I don’t know a way how to get Access running on a 64bit driver – if you are interested in the reason, read on… (If you know another solution, please post a comment!)

In BO 4.0 still all the client tools (like the Webi Rich Client) use 32bit drivers. Regarding eFashion this is not a problem as any default Windows XP / 7 / Server will provide preinstalled drivers. The BOE setup will automatically create the corresponding 32bit-ODBC datasources. Therefore you’re all fine.

On server side it is important to note that e.g. Webi Processing Server always uses 64bit drivers. As far as I can overlook it as well for MS Access. But these 64bit drivers seem not to be installed by default, at least they weren’t on my cloudshare.com environments. In addition there is a strange thing that the BOE setup creates both, 32bit as well as 64bit ODBC connections for eFashion and club.The below screenshot shows the 64bit ODBC Admin (trust me 🙂

But be careful: Whereas the 32bit ODBC connections work fine at least on my side I got the following errors when I wanted to modify e.g. the efashion connection:

If you want to create a new ODBC connection you will notify there are no 64bit drivers installed for MS Access:

My suggestion to solve this is to go here and download the Microsoft Access Database Engine 2010 Redistributable – because there is a 64bit setup / drivers for this:

http://www.microsoft.com/en-us/download/details.aspx?id=13255

Download the 64bit setup… and run it:

Finally your 64bit ODBC Admin “Add connection” dialog should look like this:

Now you can create the efashion, efashion-webi etc. data sources. Make sure you write it absolutely identical as it is written in the 32bit ODBC connection!

So far everything works fine for both, BO 4.0 SP02 as well as FP3. As usual there is a big BUT: You will still get the same errors shown right at the beginning of this post. Remember, you just installed the Access 2010 redistributable. This means you have to change your universe connection to use the appropriate driver (for this log in to Universe Design Tool and choose Tools – Connections). And here is, where at least I had to say there is no (obvious) way of how to solve it with SP02:

Sorry guys, no Access 2010 support in BO 4.0 SP02. But at least FP3 provides something for us:

And finally it should work. To sum up:

  1. On a BO 4.0 FP3 server install MS Access 2010 Redistributable 64bit
  2. Create necessary 64bit ODBC connection
  3. Modify your universe connections to point to the Access2010 driver
  4. have fun with efashion 😉

PS: I don’t have any issues with our BO 4.0 SP02 environment which has SP02 installed only as a Patch. We installed this environment during ramp-up for SP02 (in these times Webi was still labeled Interactive Analysis, that’s why I noticed the difference…) and only later applied SP02. I didn’t investigate, but it seems like Webi Proc servers uses 32bit drivers here… (no 64bit drivers for access installed on this system…)

PPS: Don’t have FP3 available but you ‘d like to test yourself? I can get you easily access to copy on cloudshare.com – see the corresponding blog post.

Do you have similar experiences? Any other hint I missed? Please post your comment.