We’re Heading To IP Expo – Europe’s No.1 Enterprise Event

It’s official, Ometis are heading to this year’s IP Expo, Europe’s leading enterprise event, at Excel London from 4 – 5 October.

We’ll be joining more than 300 exhibitors as the must-attend IT event of the year for CIOs, heads of IT and insight leaders. IP Expo is a platform for exclusive content and senior-level discussion topics, and we’re happy to announce we’ll be joined by Qlik to host a seminar session.

The annual event is an important part of the Ometis calendar and incorporates expert opinion and knowledge around analytics and IoT, key factors in the understanding and capturing of business data. We’ll see you inside the Analytics area on stand KK17, make sure you pop by.

We’ve also organised a special appearance by Qlik Director of Retail & Services Market Development, Paul Winsor, who’ll be hosting a seminar session on discovering the value in IoT Data.

It is estimated there will be over 20x connected devices per person on the planet by 2020, and anything from 50-200 billion IoT devices. That’s a lot of data being generated from IoT ecosystems. The challenge will be making all that data accessible and understandable to start extracting value from it all. Paul will explain how Qlik are making this happen at 2.40pm on Thursday 4th October inside the Analytics Theatre.

While we’re there, we’ll be taking the opportunity to discuss Qlik’s various products and complementary platforms that ensure you get the most from your software. These include Narratives for Qlik and NodeGraph and we’ll have a number of pre-set demos that showcase the power of Qlik, with true data visualisation at the touch of a button.

If you’re heading to this year’s IP Expo in London, come and say hello – we’ll have a few goodies to give away and plenty of charm and wit to make your visit worth it.

Don’t despair if you’re not attending this year’s event either, we’ll keep you up-to-date over the official Ometis social channels, so get following! We’ll also have a complete IP Expo review to follow so you won’t feel too left out.

 

By Ross Greig

Planning Your ETL in Qlik

Qlik has a powerful script engine which allows you to bring information in from a variety of sources. It’s included within Qlik and because you load all the data into memory, once saved it is effectively a portable business intelligence (BI) solution. Of course in the real world the file would remain on a server where all the decision makers can access the information held within.

Naturally, one would assume each dashboard would bring in the data needed from its sources and this can be the case. However, when you’re planning a larger scale strategic implementation this becomes less efficient.

Qlik has a simple yet effective option to allow you to develop these large suites of BI dashboards. As mentioned, Qlik stores the information into memory and also has the ability to export data tables in a special format which is already optimised in the standard Qlik file format (QVD). This optimisation reduces the data size without the need for compression. A Qlik dashboard file can now be seen as something different; it becomes a script which moves a set of data from one location to another and perhaps transforming it as well.

For large deployments we would recommend creating three layers (three Qlik dashboard files):

  1. Extract
  2. Transform
  3. Dashboard Load

The Extract layer would take information from the source systems and place it straight into a QVD file. Here, the latest information from the source system can be added to the data retrieved previously – also known as incremental loading. This method is far more efficient than taking the whole history each time you re-load. It’s important at the extract stage that no data is changed, combined or additional fields added. This way you can easily test that your saved data matches the source file.

The next step is Transformation.  Here we read the previously saved data and get it ready for consumption for a dashboard. As before we would do this incrementally for large tables that are added to over time. Transformation would include; joining tables together, adding flags and calculations.

Finally the Dashboard Loads the transformed data it requires. As the data is fully prepared in the transformation stage, it can be loaded as ‘optimised’ which means it happens very quickly.

When storing QVDs in the Extract and Transform layers it’s good practice to split large tables into manageable chunks. These tables typically contain your transactional data which grows over time (commonly referred to as FACT tables). These bite sized chucks depend on the data volume although we would typically recommend monthly files – this makes archiving data in the future easier. Also, if your dashboard only requires information from the last twelve months, you don’t need to load a QVD file which potentially could contain many years of data, which of course would slow down your process.

The benefits of this type of data load structure include:

  • Your source systems are only accessed once, reducing the demand on their resources
  • You can validate the accuracy of each stage of the process
  • QVD files are built up over time and are optimised
  • Several dashboards can re-use the same data which is both efficient and consistent

If you have any questions or you’d like more information on how Qlik business intelligence can help your organisation, please don’t hesitate to contact us on +44 (0)330 363 9900 today.

Thanks for reading!

Richard Pearce, Senior Consultant.