Trident Capital Blog

New Analytic Applications for the New Data Sets

On a daily basis Internet publishers (e.g., Yahoo, MTV) and Internet applications such as e-commerce sites (e.g., Amazon, eBay), social networks (e.g., Facebook, MySpace, Tweeter), and ad networks (e.g., VideoEgg, Valueclick) generate very large data sets with new types of data.  For example, a site like MTV.com may generate 90TB of raw data per year which, after being augmented with demographic and geotagging data, can easily balloon to 700TB.  A recent post on the management of large data states that eBay has a 6.5PB data warehouse and Facebook a 2.5PB data warehouse.  Facebook is capturing 15TB of data daily.  This new, Internet-based data consists of various types of logs, user generated content, etc.  The size of these data sets dwarfs the corporate data, e.g., sales transactions, collected and stored in more “traditional” data warehouses used by the non-internet members of the Global 2000.  These “traditional” data warehouses typically store 600GB-1TB of data.  Most mid-size companies, i.e., companies that do about $400M in annual sales, operate even smaller data warehouses that rarely cross the 300-400GB level.  The existing analytic tools and applications, e.g., Business Objects, or Cognos, were not developed with the intent of operating on anything that resembles the king-size, Internet-based data sets.

During the last couple of years we’ve seen significant innovation in the area of the area of data management, first with the introduction of data warehouse appliances by companies such as Netezza, Datallegro, Greenplum (that were based on relational database technology) and more recently with the introduction of appliances such as Aster’s and Vertica’s that are using column-based databases.  The latter two are starting to be adopted for the management of the Internet-based data.  We have also seen the development of systems such as Hadoop that provides a framework which applications can use to work on very large data sets.  These products are maturing quickly and their use is significantly reducing the cost of managing very large data sets.

While companies are making good progress on managing these very large data sets, their ability to effectively and efficiently analyze these sets is lagging.  Companies like Google, eBay and Yahoo are using internally-developed frameworks (e.g., Google’s MapReduce) and home-grown routines to analyze the data they generate because, in most cases, the existing analysis products they throw at them can’t scale to operate on these sets, or don’t have the necessary functionality to address the questions that must be answered through these analyses.  Sample questions that e-marketers (who represent only one of the constituencies that need to analyze this data) are trying to answer include:

1. What should my keyword-bidding strategy be (which keywords, what price) for each of Google, Yahoo, and MSN?  How should I allocate my budget between SEM and SEO?
2. Which ad networks are giving me the best performance?
3. Across which channels and at what percentages should I allocate my marketing budget to reduce lead acquisition costs and increase conversion rates?

Over the past few months I’ve been meeting with several startups that are developing new analytic applications to address such questions, and am particularly excited about the data analysis innovations they are working on.  Because more money is shifting online and the importance of the decisions made using this new data is rapidly increasing, this area will attract strong investor interest and has the potential of producing several winners.

Data Warehousing in the Cloud

Companies that provide on-premise BI solutions, like Business Objects, Microstrategy etc. connect to data warehouses that are built on top of third-party databases like Oracle, DB2, SQLServer, PostgressSQL and MySQL.  These database management systems, as well as data warehouse engines such as Teradata’s, provide the functionality and utilities for the creation, access, and management of a data warehouse or data mart.  The current generation of on-demand BI solutions like those offered by Pivotlink, Lucidera and Birst depart from this model by integrating a query application with a database management system both of which reside in the cloud.  More recently, data warehouse appliance vendors such as Aster Data, and Vertica started offering cloud-based options of their appliances.  Through these options they allow companies to develop data warehouses and data marts completely in the cloud and access them via on-premise or on-demand query applications. 

Cloud-based BI solutions that integrate a query application with a data mart are not entirely new.  Data marts containing web analytics data have been offered almost exclusively as cloud-based solutions by companies like Omniture and Google.  Corporations using these SaaS solutions are storing increasingly more complex and mission-critical data in these data marts, including customer data. Should on-demand BI vendors continue offering solutions that integrate the query application with their own data management system, or should they start partnering with the vendors that offer cloud-based data management systems which can be used to be standalone data warehouses in the cloud?  The answer to this question depends on your point of view regarding on-demand BI and data warehousing.  I’ll consider three perspectives: the on-demand BI solution vendor’s, the end user’s, and the IT user’s.

 The On-Demand BI Solution Vendor

For the on-demand BI solution vendor this is not an easy question to answer.  Having full control of both the application layer and the database layer offers its advantages but also has drawbacks.  The major advantage comes from the vendor’s ability to better control how data is loaded to the data warehouse, how it is organized and how it is queried. 

The ability to load data quickly allows the vendor to work with larger data sets during smaller time windows as dictated by the customer, as well as service more customers at any one time.  While smaller companies may only refresh the data in their data mart weekly, larger customers typically refresh data daily.  Moreover, the transactional database of such larger customers can only stay offline for short time periods during which data must be transferred to the data marts.  For example, a retailer may only provide a 3-4 hour window daily during which sales data can be uploaded to a data mart.  If such an operation has to be performed for hundreds of the SaaS BI vendor’s customers, then one can appreciate the importance of fast data loading. 

Data loaded in the data warehouse has to be organized in the best possible way to enable the optimal execution of queries.  The managers of on-premise data warehouses and marts constantly look at the queries executed against their databases to determine how to best organize the data to achieve best query execution times.  Some of these optimizations can be performed automatically by the database management system but most require manual intervention.  Having full control of the database management system enables the SaaS BI vendor to better optimize the organization of the stored data.

Finally, the expressiveness of the language used to query the data mart determines the range of analyses that can be performed and reports that can be created by the BI application, and also contributes to the speed with which queries are executed.  Even though SQL is a standard all database management vendors implement their own extensions, including several that are specific to data warehousing, in order to add to the language’s expressiveness.

Creating a strong database management system to support data warehouses and data marts for a variety of analytical applications is hard.  The areas mentioned above such as data loading, query optimization, query management, etc. are difficult to address and require specialized knowledge.  As many database management vendors have shown over the years (from Teradata to Red Brick, Netezza and Aster Data) good database implementations require significant R&D investments.  By decoupling the database from the application, the on-demand BI vendor will be able to focus on improving the application’s functionality without having to worry about the underlying database system.  However, despite the higher investment necessary, I think that in the short term (at least the next 2-3 years) it will be difficult for on-demand BI vendors to decouple their applications from the databases and go with a third-party offering.  In order to provide strong ROI to their customers, on-demand BI vendors will need full control of the stack as they learn about the application areas where on-demand BI will provide a strong alternative to its on-premise equivalent. 

The End User

An end user subscribes to on-demand BI applications either because his organization doesn’t have an in-house BI solution and developing such a solution internally is deemed as too expensive and requiring too much time, or it does but the solution is hard to access.  Such a user doesn’t care whether the data is stored in the cloud-based data warehouse of the vendor who provides the on-demand solution, or in a third-party’s cloud-based data warehouse.

The end user cares about convenience and analysis effectiveness.  “Convenience” means that the end user would prefer an on-demand vendor that provides a one-stop to BI.  “Effectiveness” means that the user is interested in quickly analyzing data to achieve a specific goal, e.g., determining which merchandize to discount in which regions and for what time period.  During the analysis task the user is concerned about securely loading the right data to the data warehouse, quickly and easily interacting with that data to produce the desired results and reports, and keeping the data stored safely for as long as it’s necessary. 

The on-demand BI solutions available to date have been developed with such a user in mind.

The IT User

The IT user has only recently started entering the conversation around on-demand BI solutions.  This is happening as more of the larger companies that actually have IT organizations have started using SaaS BI solutions in conjunction with (or some times instead of) their on-premise ones.  IT organizations worry about the security of the data that moves from their companies’ internal databases to the on-demand vendor’s data warehouse, the integrity of the data sent to the vendor, since they must guarantee that the data stored in a company’s internal databases is always synchronized with the corporate data stored in SaaS vendor’s data warehouse, and their ability to take back the data when the need arises.   

IT must feel confident that the on-demand BI vendor uses the strongest possible security measures to safeguard the data in the warehouse.  The SaaS vendor is expected to perform at a higher standard than the IT organization itself when it comes to security.  CIOs are starting to fear that as SaaS vendors become better established with growing client rosters that include prominent corporations, they will become prime targets for hackers (individuals or groups, independent or government-sponsored), who would want to exploit the existence of large databases with prized corporate data in a single place.

Today every on-demand BI vendor requires that they be provided either with behind the firewall access to corporate databases so that they can extract data as necessary, or with the appropriate data sets to load into their cloud-based data warehouse.  Most corporations provide the SaaS vendors with data extracts because it’s safer than giving them access to the corporate databases.  However, this approach, while safer, results in multiple copies of the corporate data.  Keeping this data updated and synchronized, so that the analyses resulting from the extracts can be impactful, can become a hard and expensive task that further taxes IT resources. 

Finally, IT must ensure that the corporate data will not become hostage of the on-demand BI vendor.  To this end, the vendor must make it easy for the customer to take back its entire data set either because of deciding to switch on-demand BI vendors, or because the vendor is going out of business.  An approach to addressing this problem may be provided by Cloudkick.  In addition, the on-demand BI vendor must make it easy for the customer to obtain subsets of the data stored in the cloud-based data mart to use it with other applications.

I wouldn’t be surprised if the majority of IT users would prefer a cloud-based BI application that accesses data stored behind the corporate firewall and doesn’t move the data permanently to the cloud.

The growing success of on-demand BI applications will lead to a lively dialog among these three constituencies on how such applications should be partitioned among the various types of cloud-based vendors and the clients themselves.

 

The Case for On-Demand BI Solutions (Part 2)

I have been a bullish investor of companies that provide on-demand BI solutions (to date I invested in Pivotlink and Host Analytics).  After being involved with BI for over two decades I see SaaS as a fundamentally transformational model whose benefits outweigh the adoption issues I described in my last posting.  Moreover, these issues are not “show stoppers.”

 The value and uniqueness of on-demand BI is coming from the following areas:

1. Cloud-enabled collaboration. Collaborative BI can effectively be enabled through cloud computing.  Companies want to collaborate through data.  Employees want to collaborate with other colleagues and partners by looking together at a single view of internal and external data that impacts daily their business “ecosystem.” Today the only decisions that are effective are those based on data blended from inside and outside the company.
2. More efficient analysis paradigm.  On-demand BI transforms the "traditional" ETLA (Extract Transform Load Analyze) process found in most data warehousing/BI environments, to an ELA process (Extract Load Analyze) process. This doesn't simply mean that you are removing a costly step (Transform) from the process, but that you are changing the entire analysis paradigm. Because on-demand BI applications make Extraction and Loading very cheap, the user of an "ELA environment" doesn't have to be extremely thoughtful on what data to include in the warehouse (or data mart) and how to transform it in order to improve the analysis results. Instead he can load a bunch of data and "experiment" with it. If the experimentation goes well then he continues. If it doesn't go well then he "throws the loaded data away" and load some more without delay or guilt. On-demand BI companies like Pivotlink have been developing extremely fast data extractors and loaders in order to better support the ELA paradigm for an expanding set of cases.
3. Social networking/Community.  Social networking and other social media applications have demonstrated the power of communities and are now moving from personal use to enterprise use.   Community-based tools built around on-demand BI solutions can increase a BI solution’s adoption, reduce service delivery costs and, through the established dialog between users and the solution provider, improve the application’s feature set and performance.  Imagine, for example, through a presence tool, such as Microsoft Messenger, a user determines that another user who is an expert on inventory analysis is online.  By looking up this user’s profile he finds out that the user has received good recommendations from co-workers and customers about his ability to create inventory analysis reports.  A real-time dialog can begin and lead to collaborative problem-solving.
4. Better maintenance of the application and the data. Because the application/reports and the database/data warehouse are located in the vendor’s data center, software updates, database maintenance for optimal query performance, report organization and other tasks associated with the support of a properly running BI solution are managed and performed centrally by the vendor.  This activity results in optimally running BI solutions enjoyed by all users.  The performance of on-premise BI solutions is dependent on the IT organization’s abilities to a) devote the right resources to address the above mentioned tasks along with software updates and patches coming from the vendor and b) develop local expertise to optimize the performance of each BI solution being used.  This may be possible in certain companies with large and sophisticated IT organizations but, unfortunately, it is not found in every company that uses on-premise BI solutions.  For example, a recent Oracle critical patch bundle included 16 updates for Oracle Database Server, 12 updates for Oracle Application Server, 3 updates for Oracle Applications, 4 updates for Oracle PeopleSoft and JDEdwards Suite, and 8 updates for BEA Products Suite.
5. Usage optimizations. By monitoring the on-demand BI solution’s usage, the vendor can a) dynamically make optimizations to improve its performance, and b) determine which features are being used effectively, which features are not being used at all, which features present difficulty to the user (even though they are perceived as useful) and also make improvements as appropriate.  There is a bigger point here relating to a vendor’s monitoring a SaaS application’s usage.  It should first be noted that this type of monitoring cannot be accomplished for on-premise applications.  The vendors of such applications don’t know how their software is being used.  On-demand application vendors have the opportunity to closely monitor usage to identify: a) if the application is used as expected, b) which parts of the application are being used the most (not unlike how a web analytics package reports on which web site pages are being visited), and c) even provide cross user statistics in the way Nielsen and comScore report various types of audience metrics.
6. Easier to avoid vendor lock-in particularly now that the major on-premise BI vendors (Cognos, Business Objects, Hyperion) have been consolidated. Because of how critical BI is to every corporation, customers should always try to select the BI vendor with the most-suitable solution rather than the vendor that offered them "a great deal" on BI by bundling it with a bigger IT contract, as the major IT vendors are likely to do.
7. Innovation.  The consolidation of the major on-premise BI companies can also result in slower innovation since they are now divisions of larger IT vendors that may have different overall objectives than to provide innovative BI solutions.

 Areas that are impacting the TCO of an on-demand BI solution:

1. Fewer personnel needed to support the application.  Moreover these personnel don’t have to possess specialized knowledge in the way that personnel that support on-premise BI solutions must have.  BI solutions have been particularly personnel-intensive (database designers and administrators for the data warehouse, programmers and analysts to create and maintain reports, data analysts to extract information from the collected data).
2. No upgrades to hardware.  The dissemination of on-premise BI solutions within the enterprise has been frequently stymied by the need for upgrades so that client machines can satisfy the processing needs of BI tools.  On-demand BI solutions are not only accessed through a web browser obviating the need for hardware upgrades but also resulting in fast deployments of these solutions.
3. Cheaper implementation of a data warehouse/data mart because using a standard data model is faster and can benefit from the improvements (functional, architectural, implementation) performed by the vendor’s dedicated staff for all customers.
4. No hidden costs in terms of solution upgrades, customizations, systems and data integration.
5. Dedicated vendor personnel keeping the data warehouse in top shape (through data model refinements, data utilization, additional aggregations, etc.) and disaster recovery.  When considering the cost benefits of SaaS, many companies do not include the value of transparent and immediate upgrades to the latest software version.  Companies deploying traditional on premise solutions must account for the ongoing cost of planning, implementing and supporting new software upgrades and bug fixes.  This can easily cost from tens to hundreds of thousands of dollars in professional services, IT resources and duplicate hardware and it usually involves running parallel applications during the implementation, testing and acceptance period. It is not surprising that the average ERP implementation is typically 2-3 versions behind the latest software update.

 BI continues to score high as an important corporate solution even during this time of shrinking IT budgets.  On-demand BI addresses many of the issues that had plagued on-premise BI solutions and offers some unique advantages making such solutions possible for more and larger segments of the market.  As such it warrants serious consideration by small and large companies alike.