Business Intelligence in the Cloud: Sorting Out the Terminology

Over the past few months, both mainstream and startup software vendors have been announcing a steady stream of hosted business intelligence (BI) solutions. The various terms used to describe these offerings are often confusing. Examples of terms here include on-demand BI, BI software-as-a-service (SaaS), and BI in the cloud. The objective of this article is to help reduce this confusion by explaining the technologies behind these terms and providing an overview of the capabilities they provide.
Let’s start by reviewing six different business intelligence product offerings announced in this area this year. [For simplicity, I will use the term hosted solution to describe all of the products involved in these announcements.]

• Panorama Software’s PowerApps provides a hosted and scalable analytical engine for managing OLAP cubes built from relational data. The first application to use this engine will be Panorama Analytics for Google Spreadsheets. PowerApps is currently in beta testing and will be made available by the end of 2008.
• Vertica Analytic Database for the Cloud is a hosted and scalable version of Vertica’s columnar relational database system. The underlying grid-based computing system supporting this solution is provided by Amazon’s Elastic Computer Cloud offering.
• Kognitio’s Data Warehousing as a Service (DaaS) is a hosted and scalable version of Kognitio’s relational database system.
• LucidEra offers several hosted BI analytical applications that work with Salesforce.com and other CRM offerings. LucidEra Lead Insight is a new solution that provides pre-built metrics focused on the quality and performance of marketing leads, and how effectively they are converting to opportunities and to closed business.
• Business Objects Business Intelligence OnDemand is a solution that provides a hosted version of the BusinessObjects XI BI tools product set. Other hosted solutions from Business Objects include: Crystalreports.com for sharing reports over the web, Information OnDemand for third-party market and financial data, and Business Applications OnDemand, a set of hosted applications (acquired from Nsite) for quote and proposal management.
• PivotLink (formerly SeaTab Software) provides a set of hosted analytical and reporting tools, together with pre-built industry specific metrics. PivotLink for Retail, for example, gives retailers metrics on highest-margin SKUs, inventory turns and stock levels, order status and store comparisons.

An examination of the websites for each of these six products shows the confusion caused by the many different terms used to describe hosted solutions. Here are some examples of the terms used by these vendors:
Panorama Software
… SaaS BI Solution
… analytics as a service platform
… cloud-based data warehouse and OLAP cubes
Vertica
… performance on demand
… analytic database for the cloud
… grid-enabled
Kognitio
… data warehousing as a service (DaaS)
… on-demand data warehousing
… hosted BI applications
LucidEra
… business visibility on demand
… on-demand business intelligence
… LucidEra SaaS
Business Objects
… hosted platform
… on-demand business intelligence
… software-as-a-service model
PivotLink
… intelligence on demand
… SaaS model
… subscription-based model

Business Intelligence in Cloud

Cloud computing based on pay-as-you-go hardware infrastructure like Azure, Amazon or Google offers business intelligence users new data warehouse options that bring unlimited scalability without traditional data center overhead and budget constraints. It can provide end to end platform for BI applications for storing and analyzing data. High scalable storage in cloud can be used to create large data warehouses enabling storage of data in Terabytes simultaneously utilizing the on-demand compute power to analyze data.

In my view, there are three main constraints to BI adoption in this new era of analytic data management for business intelligence:

• Effort required in consolidating and cleaning enterprise data: Prior to integrating data within the system it is required to detect and correct (delete) corrupt or inaccurate data from the store. This process will involve removing any typos, validating data definitions against the destination datawarehouse and correcting values against a known list of entities.
• Cost of BI technology: This requires well developed infrastructure in terms of high end data center servers and software for ETL (Extraction, transformation and loading) process. Developing BI applications means additional overheads on IT infrastructure leading to more financial commitment for driving business.
• Performance impact on existing infrastructure / inadequate IT infrastructure: Based on the business processes, rules and attributes defined by the business, data required for business intelligence is extracted from legacy and transaction systems, cleansed, validated and loaded into a dedicated database, known as a data warehouse. Developing BI applications using existing IT infrastructure can have a performance impact on the existing applications as it requires heavy processing during ETL processes and may be sometimes inadequate for BI processes.

Cloud computing is potentially an answer to two of these problems- second and third one. Its compute power enables organizations to analyze terabytes of data faster using BI applications more economically than ever before and is delivered in an on-demand basis. Cloud customers “rent” dedicated servers and the enterprise need to house, secure, and manage them.

Advantages of Transforming BI in Cloud

Cloud computing is transforming the economics of BI and opens up the opportunity for smaller enterprises to compete using the insight that BI provides. Cloud-based analytics will impact BI by:

• Easier evaluation of Technology: Cloud enables software companies to make new technology available to evaluators on a self-service basis, avoiding the need to download and set up free software downloads or acquire hardwares fitting to the technology.
• Increased short-term ad-hoc analysis: Where short term needs (weeks or months) for BI is required, cloud services are ideal. A data mart can be created in a few hours or days, used for the necessary period, and then cancel the cloud cluster, leaving behind no redundant hardware or software licenses. The cloud makes short term projects very economical.
• Increased flexibility: Due to the avoidance of long term financial commitments, individual business units will have the flexibility to fund more data mart projects. This is ideal for proof of concept, and ad-hoc analytic data projects on-demand. This agility enables isolated business units to respond to BI needs faster than their competitors and increase the quality of their strategy setting and execution.
• Drive data warehousing in MB markets: Medium-size businesses often have very large volumes of data for analysis, yet only a few IT resource at their disposal to analyze tons of terabytes of historical data to fine tune market strategies. Cloud-based analytics can enable such businesses to warehouse and analyze terabytes of data in spite of these resource constraints.
• Drive the analytic SaaS market: Companies that collect economic, market, advertising, scientific, and other data and then offer customers the ability to analyze it online will be able to bring their solutions to market with much less risk and cost by utilizing cloud infrastructures during the early stages of growth.

Growth Considerations

As data volumes grow, for analytic cloud projects to succeed they will require an architecture that is designed to function efficiently in elastic, hosted computing environments. At a minimum, such application must include the following architectural features:

• “Scale-out” shared-nothing architecture: To handle changing analytic workloads as elastically as the cloud. Auto-scaling of Virtual Machine (VM) can be used to proved necessary compute power required during heavy workload and an efficient  algorithm need’s to be worked out in order to auto-scale in VM’s when not required.
• Aggressive data storage: Cloud provides an appropriate infrastructure for storing large amount of data at low cost. No further additional overheads are required to store data on cloud thus helping achieve manpower savings for operations like data backup and server maintenance. For example in case of Windows Azure, Table Storage is designed to be massively scalable and a typical Azure Table can contain billions of records amassing to Terabytes of data. Blob Storage provides a means to store unstructured data much in the same way that would store a bunch of images on the File System of a server. Blobs can be mounted as XDrives on the Virtual Machine instance where a particular service is running and accessed exactly like a file system would.
• Automatic replication and failover: This will provide high availability in the cloud. In case of Windows Azure, data is stored on 3 nodes to enhance both access speeds and reduce data redundancy.

Challenges

While the current environment hints at several of the aforementioned advantages of the technology, they also bring into focus the challenges that need to be overcome in order to make BI on Cloud model work. Moving data to the cloud is an expensive proposition due to high network cost invloved in the movement of existing data from on-premise to cloud. Storing data in the cloud can be troublesome as data is core and proprietary to enterprises also BI components as services are not availabe from BI vendors.