Trident Capital Blog

Hadoop is Many Things Including the Ideal ETL Tool for Big Data Analytics

The success of the recent Strata and Structure conferences (conclusions from last year’s conference and resulting trends can be found here) reinforced the accelerating corporate interest in big data and the specific need for applications and techniques that take advantage of Hadoop.  Based on the presentations I attended or read about, it appears that more companies are getting comfortable collecting Petabyte scale data sets from sensor networks, social applications, IT log files, and online advertising applications.  However, many of the companies collecting these data sets appear to have been struggling with the analysis of the collected data, primarily because of its volume and its complexity (reflected in both the number of unique attributes but also in the fact that data relevant for a particular type of analysis may be spread among several data sets that need to be brought together).  Larger corporations that have already developed data warehousing and analytic infrastructures must understand how to use Hadoop in order to deal with the data volumes being collected in the context of their existing infrastructures.  Smaller private companies, on the other hand, are trying to establish such infrastructures for the first time and trying to determine what role Hadoop can ultimately play.

Companies analyze data either through interactive queries or by creating models.  Such models can predict the outcome of future actions or automatically describe characteristics of data sets.  Examples of emerging analytic big data applications include social media sentiment analysis, and analysis of various forms of data logs for online advertising, computer security, etc.  Hadoop’s MapReduce batch processing programming model has proven effective for model creation.  One may even say that for model creation Hadoop could further benefit by interfacing with open source modeling tools such as those offered by Revolution Analytics.  But for applications that require the generation of real time decisions using such models, e.g., deciding which online ad to display in a particular web page, or determining the likelihood that a credit card transaction is fraudulent before authorizing it, Hadoop’s batch model is inappropriate.  Similarly, Hadoop’s batch processing model is inappropriate for applications that are driven by interactive analytic queries, e.g., identifying the effectiveness of particular price discounts in increasing the unit sales of a product in a specific geographic region.  Hadoop’s strengths make it the ideal “preprocessor” for big data; essentially, the ETL tool for big data analytic infrastructures.

Let me elaborate.  Users of analytic applications that are driven by interactive queries can use Hadoop to select the right data to query (which often involves testing the information content of various data sources before deciding which ones to use), determine whether data from the selected sources needs to be merged into a new source, or transform the data in some other way, (for example, by creating new derived attributes).  In other words, a lot of what commercial ETL tools do today.  In a company with a pre-existing data warehousing infrastructure, analysts can import (load) the preprocessed data into the commercial analytic database, e.g., Teradata, Netezza, Greenplum, or the in-memory database, e.g., Qliktech, and start interacting with it through queries.  In a company without such infrastructure, as is the case with most startups that make heavy use of big data analytics, the preprocessed data can be subsequently loaded in a NoSQL database for further analysis.

Where does this leave big data analytic workbenches like those offered by companies such as Karmasphere and Datameer?  To reach the big data analytic mainstream such workbenches must be part of the Hadoop ecosystem as they have the potential to make it easy for users to set up analytic tasks and perform analyses.  However, their ultimate success will depend on whether Hadoop continues to be viewed as an important component for big data analytics, the approach big data stack vendors, e.g., Cloudera, decide to take regarding the end user/analyst, i.e., will they support such a user directly or through partners, and how well these workbenches become integrated with each big data stack.

Related articles

• DataStax Brisk: Marrying big data tools Hadroop and Apache Cassandra (zdnet.com)
• Hadoop Goes Mainstream for Big BI Tasks (pcworld.com)
• Deriving Intelligence from Large Data Using Hadoop and Applying Analytics (oreilly.com)

Impressions from Strata 2011

Last week I attended Strata, a conference organized by O’ Reilly and devoted to big data.  I was a large conference (790 attendees) whose content included both technical talks and tutorials about the new generation of big data tools, e.g., Hadoop, Cassandra, visualization, as well presentations on big data business applications.  The diversity and size of the audience and the reported business successes provided a strong indication of how important and popular the area of big data has become. 

Big data is pervasive in many of the companies Trident has funded the last few years.  We have invested in companies that generate and/or process big data, e.g., eXelate, Extole, HomeAway, Sojern, Turn, Xata, as well as companies that provide platforms for storing, managing and analyzing big data, .e.g., Acteea, Host Analytics, Pivotlink.  We recognize that many of the companies we invest in the future will need to have competence in big data.

There is a big difference between big data and data warehousing stemming primarily from the nature of the data.  Data warehousing was all about analyzing transactional data that was captured from enterprise applications such an ERP or POS system.  In addition to the actual transactions, big data is about capturing, storing, managing and analyzing data about the behavior of transactions, i.e., what happens before and after a transaction.  This has several implications.  First it means that the captured data is less structured.  It is easier to analyze a collection of purchasing transactions in order to try to identify a pattern, instead of analyzing a series of selections made across of set of web pages to establish a pattern of behavior.  Second it implies that meaning must be extracted from events, e.g., the browsing activity prior to buying an item.  To be effective in this more open-ended exploratory data analysis one has to break through the data silos that are typically found in enterprises and bring all available data to bear.  It also means that one must be collecting all available data rather than trying to decide a priori which data to collect and keep.

Data science is becoming a field.  Big data is eliminating the segregation between the people who manage the data, the people who analyze the data, and the people who present/visualize the data.  A good data scientist must be able to do all three, though, as I wrote last week, translating business requirements to a data problem and the resulting insights to business actions and value remain largely missing skills in data scientists.  Good data scientists are in high demand, as indicated by the jobs being advertised at the conference and as reported at the conference by LinkedIn.  They are expected to play a significant role on how their companies evolve.  That’s not something we were used to hearing about data analysts who were always considered fixtures of the back office.  I know because I started my career in data analysis.

Corporations have a lot to learn about big data from consumer-oriented companies that generate, manage and analyze big data, e.g., Amazon, eBay, Facebook, Twitter, and LinkedIn to name a few.  This is a reversal of sorts.  In the mid 90s when I was with IBM I was running an organization that was devoted to building data warehouses and providing analytical tools and services to Global 1000 companies.  At that time various companies, including many of the then nascent Internet companies, were trying to learn from the data warehousing and business intelligence practices of Walmart, Citibank, and First Data.  Today such companies will do well by trying to understand and apply the big data techniques being developed by many internet and social media companies.  One big difference is how such companies approach data stores.  Traditional businesses see the enterprise data warehouse as storing the “single version of truth” about the data.  Big data stores are viewed as containing multiple perspectives.  Their contents must be analyzed with the right set of tools in order to gain a perspective about the problem at hand.

Talking to the conference’s attendees I got the impression that more companies than ever before are starting to view data as an invaluable asset and a potential key to their success. They are no longer intimidated by data volumes and are using the new generation of big data management and analysis tools to bring more data under their control. 

Strata was a great conference that brought under one roof the leaders in big data thinking, and doing.  It also showed that, though increasingly important, this is still a small community and in many respects its overall size has not changed since the time I was one of the analysts.  We all need to find ways to accelerate the education and introduction to market of new data scientists.  The ability of many companies to continuously innovate, become leaders, and remain in this position could largely depend on their ability to recruit data scientists who can effectively exploit their big data assets.

Related articles

• Will data warehousing survive the advent of big data? (radar.oreilly.com)
• Executive Summit kicks of O'Reilly Strata Conference 2010 (cloudave.com)
• Strata Conference 2010: Real World Applications in the Enterprise and Industry (cloudave.com)

Business Analytics Obstacles

Last week my partners and I hosted a meeting of our IT Advisory Board.  This board consists of senior IT executives from Global 2000 companies including CIOs and CTOs.  I will write about the topics discussed in this meeting in a few days, once I had the opportunity to clean up the several pages of notes I took.  Today I wanted to relate one of the conversations I had with a couple of the board’s members during one of the meeting’s breaks.  We started talking about the effective utilization of business analytics by companies.  Both executives commented that their companies are increasing the utilization of analytics to understand their consumer customers.  In fact, one of them stated that during the last 3 years their analytics group grew from 4 analysts to 20.  Moreover, they stated that senior business management in their companies is more sensitized to the importance of analyzing corporate data to gain any type of competitive advantage.  As I tried to ask them about the analytic tools and solutions their companies currently employ and how these have been changing over time to deal with the increasing volumes of data, they both stopped and said that their biggest obstacle for the broader utilization of business analytics was not technology but the proper and effective application of the information they extract from the analyses they perform.

Both of these companies have always been early adopters of analytic and associated data management technologies.  Both executives indicated that they feel particularly good about the caliber of their corporate data analyst groups.  However, today, by their admission, both of their companies lack the people who can provide a “two-way translation,” i.e., first to properly translate a business problem to a data analysis problem (that can subsequently be tackled by the quants), and second to formulate (or re-translate) the analysis results in a way that business executives will understand, appreciate, and be able to act on.  Companies that provide business intelligence (BI), analytics and even data warehousing solutions talk about how they target “business analysts.”  The business analyst has become a mythical position in corporations.  The business analyst is generic description for an individual who works for a business unit, as opposed to an IT organization, and uses such analytic solutions in the course of business.  However, our advisory board members said that the business analysts in their companies use such solutions to address well-understood problems and activities, not novel situations that may call for an data-driven analytic approach.  These business analysts may use a query and reporting solution like Qliktech’s or a multidimensional analysis solution such as Microstrategy’s but they are not able to provide the two-way “translation” I described above.  In their opinion, the right translator/analyst must have the appropriate level of business understanding and experience to understand complex business problems in their entirety, be articulate enough to describe it appropriately to data analysts, the right amount of data knowledge to be able to broadly identify the types of data that will need to be utilized by the quants to provide insights and information, the ability to take these insights and relate them to the original problem providing actionable solutions and finally the executive gravitas to present these solutions to the business executive(s) who will act on them.

There exist independent consultants who play this two-way analytics translator role very effectively but their extensive and continuous use by corporations is not feasible, mostly for financial reasons; they are too expensive and in too high demand.  In discussions I’ve had with some members of IBM’s 7000-strong analytics and optimization consulting unit I heard that those of their consultants who can provide such services are in the highest demand.  In fact IBM can’t find enough of them to hire and deploy with corporate clients around the world.

So while we celebrate the development of new analytic tools or solutions that can deal with even larger and more complex quantities of data that must be processed faster than ever before, we must not lose sight of the fact that we must address our inability to identify enough “translators” who can help us analyze the right problems and effectively use the insights we discover.

Related articles

• IBM doubles down on business analytics (infoworld.com)
• IBM's Watson: The Future of Business Analytics? (enterpriseirregulars.com)

We Need a New Generation of eCommerce Analytics

In 1996, while I was running IBM’s BI solutions organization, one of my groups developed the Surfaid web analytics solution.  Surfaid, one of the first such solutions, was later acquired by Coremetrics (that was in turn recently acquired by IBM and made part of its marketing automation solution).  Later on Omniture dominated the high-end web analytics market by figuring out the right ingredients of a web analytics solution (quick set up, effective data collection and management, informative reports for the emarketer) and the right model for delivering it (SaaS).  Google also entered the market and dominated the low-end with a free offering.  The evolution of this market over the past 10 years has taught us that web analytics will remain a relatively small component of the overall analytics and BI market. 

Ecommerce marketers and merchandisers were some of the earliest adopters of web analytics.  However, the continued growth of ecommerce combined with the increasing complexity of the decisions these business users must make is causing retailers (both pureplay and multi-channel) to look for more sophisticated analytic solutions than those offered by the web analytics vendors, or the ecommerce platform vendors, e.g., Demandware and ATG (recently acquired by Oracle),  could offer.  While web analytics solutions can be used to determine which landing pages encourage customers to make a purchase, or which pay per click ad campaigns are most effective, ecommerce marketers and merchandisers want to now understand customer loyalty, the impact of their customer retention strategies (discounts, coupons, extra services), and the customer segments with the greatest Lifetime Value (LTV).  Today’s web analytics solutions cannot address these needs.

During the summer of 2009 Infopia, one of my portfolio companies and an ecommerce platform company at the time, asked the larger of its 300 etailers about their analytics needs.  The company found out that while all of its customers were using a web analytics solution, none were able to address their particular ecommerce decision needs through these solutions.  The market demand for a sophisticated ecommerce analytic application was so strong that the company’s board decided to direct significant resources towards the development of such an application.  A year later the company sold its ecommerce platform business to Versata, changed its name to Acteea, did a pivot and is now a SaaS ecommerce analytic application company.  Acteea’s SaaS analytic application enables merchandisers and marketers to track customer LTV and define winning product and customer segment strategies. 

Acteea’s analytic application integrates into a cloud-based data mart pricing data, marketing campaign effectiveness data, adword data (words bid and words purchased), customer order data, inventory data, web site activity data (what is typically fed to the web analytics solutions), customer activity data from other channels, e.g., catalog, promotions data, and competitor pricing data.  Data integration and cleaning has become a very complex process compared to the data integration that web analytics solutions must address.  Extole, another of my portfolio companies, has developed a SaaS platform for social marketing that is quickly being adopted by etailers.  The data produced by the platform’s applications (3 to date) will undoubtedly become another source to Acteea’s analytics as it can help etailers with decisions around social commerce.

In a recent board meeting we reviewed some of the early successes Acteea’s customers are having through the use of the company’s analytic application.  For example, one of the companies analyzed keyword, adword, web analytics, pricing and product catalog data, refined its adword bidding approach and identified “driver” products that drew existing target customers into making add-on sales.  Another customer began measuring the total return on marketing investment, and cart value by customer segment which led to revamping customer segmentation based on channel-loyalty, and cross-channel behavior leading to improved quarter-over-quarter sales.  Finally, a third customer analyzed product sales and gross margin return on inventory and quickly identified the lowest performing products eliminating them from the appropriate channels, including the web site.

It is still too early to tell whether Acteea’s pivot will be successful, though the initial results are encouraging.  Regardless, the company’s work is proving the market’s need for complex ecommerce analytic solutions that are distinct from the existing web analytic toolkits that have been available.

Mobile Data Analytics

Over the past couple of years I have met with several startups that offer analytic solutions for mobile data.  I have not invested in any of them.  I had felt that the data captured from feature phones and early generations of smartphones was not rich enough to lead to interesting and distinct analytics.  For example, while data captured from a mobile web browser such as sites visited, pageviews, time spent browsing could be analyzed, we didn’t need a new company to do that.  Omniture could do that just fine.  However, the new smartphones capture more interesting data.  These data sets could drive the creation of a new and interesting analytics.  As a result, I am becoming interested in mobile data analytics and have been actively looking for investment opportunities in this sector.

The new smartphones are becoming sensor platforms, as well as being computing platforms.  In addition to photo and video camera, touch screen, GPS and accelerometer, new types of sensors are being connected to smartphones.  For example, Bling Nation has introduced a sensor that adheres to a smartphone and is linked to the user’s PayPal account.  Our own portfolio company Zeo has announced that it will connect its sensor to the iPhone in order to capture sleep-related data.  Some of the data sets generated by all these sensors that I find interesting include:

1. The time-series of GPS and accelerometer data for each subscriber.  By analyzing these time-series one can predict where and when the subscriber will be next and offer relevant services at the predicted location, e.g., parking availability with offers from parking garages.
2. Data generated from the use of augmented reality (AR) application can create new advertising opportunities, as well the opportunities to serve up relevant content the user had not thought to ask for.
3. Configuration data on the complete software stack running on each phone (from firmware to operating system to application software).  This data can then be used, for example, by an app store to recommend new available applications that will be augment the user’s productivity.  Such configuration databases today exist only in corporate IT settings.
4. Mobile payments data combined with geolocation data.  Analyzing this data can lead to predictions about customer brand or product loyalty.

Entertainment-related applications, e.g., gaming, and health care applications, e.g., prescription dispensing, will also benefit from the analysis of this type of data.  I am not certain whether new data management systems will be necessary for such data sets, though I imagine that the data will be big and complex, particularly as various time-series are captured, and will be stored in the cloud.

The wireless carriers may not be in the best position of collecting this data, not only because of their lack of experience with diverse data types, but also because they are regulated businesses.  Google and Apple are in a much better position because they already collect much of this data through their Android and iOS platforms respectively.  While these companies may also be best able to mine the data, they won’t enter this business in the near term.  Instead it will be startups that will first experiment with creating interesting data sets out of the collected data and analyzing them.  My assumption, also driving my interest in the sector, is that companies like Google will wait to see how these “experiments” go and proceed to acquire the more interesting of the analytics startup companies.

Users will need to give their permission for this rich data to be collected and combined.  Vendors, including wireless carriers, will get the users’ permission by offering free services (something for which consumers have shown interest and affinity), better experience (optimized bandwidth, improved application performance, more accurate recommendations around applications, products, services, social connections, etc), and more accurate targeting of ads in ad-supported services.

The mobile space remains highly fragmented and the talent to create and analyze these data sets may be hard to find.  The new smartphone platforms present opportunities for collecting valuable data sets that will lead to the development of unique analytics which will in turn drive important and novel decisions.  Startups can lead the way to create these analytics and the enterprise platforms that manage them.

Related articles

• A sea of sensors (economist.com)

Insight as a Service

The survey data presented in last August’s Pacific Crest SaaS workshop pointed to the need for a variety of data analytic services.  These services can be offered under the term Insight-as-a-Service.  They can range from business benchmarking, e.g., compare one business to its peers’ that are also customers of the same SaaS vendor, to business process improvement recommendations based on a SaaS application’s usage, e.g., reduce the amount spent on search keywords by using the SEM application’s keyword optimization module, to improving business practices by integrating syndicated data with a client’s own data, e.g., reduce the response time to customer service requests by crowdsourcing responses.  Today I wanted to explore Insight-as-a-Service as I think it can be the next layer in the cloud stack and can prove the real differentiator between the existing and next-generation SaaS applications (see also here, and Salesforce’s acquisition of Jigsaw).

There are three broad types of data that can be used for the creation of insights:

1. Company data.  This is the data a company stores in a SaaS application’s database.  As SaaS applications add social computing components, e.g., Salesforce’s Chatter, or Yammer’s application, company data will become an even richer set.
2. Usage data.  This is the Web data captured in the process of using a SaaS application, e.g., the modules accessed, the fields used, the reports created, even the amount of time spent on each report.
3. Syndicated data.  This is third-party data, e.g., Bloomberg, LinkedIn, or open source, which can be integrated (mashed) with company data and/or usage data to create information-rich data sets. 

Some of the issues that will need to be addressed for such services to be possible include:

1. Permission to use the data.  For this to be possible, corporations must give permission for their company data to be used by the SaaS vendor for benchmarking.  For example, if Salesforce customers are willing to make their data available then their sales forces’ effectiveness can be benchmarked against that of peer companies.  It may be more likely for companies to give their permission if the data is abstracted or even aggregated in some way.
2. Data ownership.  The ownership of usage data has not been addressed thus far.  Before creating and offering insights, ownership will have to be addressed by the SaaS vendors and their customers.  Once ownership is established, as I had written before, this data can, at the very least, be used by the SaaS vendor to provide better customer service or even to identify upsell opportunities and customer churn situations.  While some vendors, e.g., Netsuite, are starting to utilize parts of usage data, utilization remains low and scarce. 
3. Data privacy.  Company and usage data will most definitely include details that may need to be protected and excluded from any analysis.  The SaaS vendors will have to understand the data privacy issues and provide corporate clients with the necessary guarantees.  Thus far SaaS vendors have only had to make data security guarantees.  Privacy concerns around this data will be similar to those that currently surround the internet data that is being used to improve online advertising.
4. Potential need for pure-play Insight-as-a-Service vendors.  The SaaS application companies may not prove capable of providing such insight services.  It may be necessary to create specialized vendors to offer such services.  Such pure-play vendors may have more appropriate and specialized know-how which will be reflected in their software applications (essentially analytic applications that can organize, manipulate and present insights).  In addition they will be able to offer a broader range benchmarking since they will be able to evaluate data across SaaS vendors.  However, having such vendors will also necessitate the move of company and usage data to yet another location/cloud thus increasing the security and privacy risks. 
5. Eligibility for accessing these insights and business models under which they can be offered.  One approach would be to only offer such insights to as a separate product by the SaaS application’s vendor to its customers.  Another approach, particularly if the insights are to be created by a pure-play insights vendor, would be for such vendors to create data coops.  Under this scheme corporations contribute company and usage data to the coop, the Insight-as-a-Service vendor analyzes all contributed data, and only offers the results to the companies that belong to the coop.  For this service the vendor can use an annual subscription fee not unlike what industry analysts like Forrester and Gartner charge.  Internet data companies such as Datalogix, that has created a coop with retail purchase data, can serve as good models to consider.  Another business model may be for the vendor, either the SaaS application vendor or the Insight-as-a-Service vendor, to share revenue with the companies providing the company and usage data.  Internet data exchanges like Blue Kai and eXelate would provide good business model examples to imitate. 
6. Geography.  As we’ve learned with consumer internet data, each country approaches data differently.  For example, European countries are more restrictive with the use of collected data.  SaaS companies must try to learn from the relevant experiences of internet data companies as they determine how to best offer such insight services.
7. Data normalization.  Usage data will need to be normalized and then aggregated since each customer, and maybe even each individual user, uses a SaaS application differently.  This could be tricky.

Hosted applications need not apply.  Not all vendors will be able to offer such services.  For these services to be successful, data from the entire customer base needs to be aggregated and organized.  This implies that vendors claiming to offer SaaS solutions when they are only offering single-tenant hosted solutions deployed in, what amounts to, private clouds will not be able to provide such insight services.  In fact, multi-tenant architectures will be even more important for insight-generation because they make data aggregation easier.

Insight-as-a-service can become the next layer of the cloud stack (following Infrastructure-as-a-Service, Platform-as-a-Service and Software-as-a-Service).  In addition to SaaS application vendors that can start offering such services, there exists an opportunity to create a new class of pure-play Insight-as-a-Service vendors.  Regardless, vendors will need to start addressing the issues and many more that I can’t anticipate at present.  But since surveyed customers are already starting to ask for such services, it is time to start creating them.  It means that the time for Insight-as-a-Service has arrived.

IBM's Business Analytics Blogger Event

Last Tuesday I was invited to speak at an IBM event for technology bloggers. The event addressed two topics that represent key growth priorities for IBM: business analytics and smarter planet. IBM’s Fred Balboni, Global Business Lead for Analytics (responsible for all business analytics services at IBM), Rod Smith, VP of Emerging Technologies, and Mike Rhodin, SVP Software Solutions Group (responsible for all business analytics, BI, and collaboration at IBM) participated in the meeting, along with two IBM clients the Beacon Institute and the NY Tax Office. IBM invited seven bloggers to the event: Erick Schonfeld (Techcrunch), Derrick Harris (Gigaom), Alex Williams (ReadWriteWeb), Larry Dignan (ZDNet), Mike Vizard (IT BusinessEdge), Mike Loukides (O’ReillyMedia) and Mark Kobayashi-Hillary). I was asked to talk about why VCs are interested today in business analytics.

My presentation followed the talks of the three IBM executives who stated that:

1. Worldwide customer demand for business analytics services is so strong that IBM is now trying to hire an additional 3000 consultants in Fred Balboni’s group. Just a few months ago that group was supposed to have 4000 consultants.
2. IBM customers are starting to ask for Hadoop-based data warehousing and analytics solutions in order to determine if they can them to a) manage and analyze large data sets more cost-effectively and b) to quickly perform exploratory data analyses on multi-dimensional data. However, the case studies provided by Rod Smith of IBM customers using Hadoop today were not dealing with big data (i.e., TB- or PB-scale data sets).
3. As part of its smarter planet initiative IBM continues to expand its collaboration with cities around the world where it applies business analytics and uses business processes from the corporate world in order to streamline and optimize the operations of cities.
4. Even after spending several billion on BI-related acquisitions, IBM intends to continue building out its business analytics technology stack through additional acquisitions and internal product development.

In my presentation I claimed that VCs are investing in business analytics for the following reasons:

1. The impact of cloud computing. Cloud computing is enabling the creation of new analytic applications, such as Infopia's ecommerce analytics application, and others about which I had written here, where analytics are embedded in the application from the beginning rather than as an afterthought (as it used to be the case in the past). It also enables new takes on existing analytic applications, such as the SaaS corporate performance management application developed by Host Analytics. Finally cloud computing promotes new usage patterns in BI solutions such as collaborative decision-making across the extended enterprise, i.e., across customers and partners, and better economics, i.e., better price/performance, over equivalent on-premise solutions. As an example of collaborative decision-making, I mentioned that cloud-based BI solution vendors can offer their clients corporate benchmarking services (think of it as Insight-as-a-Service) by analyzing uploaded and collected data from across their entire customer base, identifying important patterns, and translating them into the appropriate business actions.
2. The emergence of big data. Ever increasing volumes of structured and semi-structured data, including web data, such as that created by ad networks, and sensor data, such as that created by he Internet of Things, are starting to test the scalability of existing data warehousing solutions, make such solutions prohibitively expensive, and inhibit the timely creation of actionable analytics. I had written about the data volumes created by the web applications and how a new generation of analytic applications and data warehousing infrastructures need to be created in order to effectively deal with it here and here. During the meeting the director of the Beacon Institute spoke about the institute’s collaboration with IBM to monitor the Hudson River using Complex Event Processing and streaming databases. This solution generates real-time analytics from the voluminous data (most of the data being of little interest but 2% being of extreme interest) produced by thousands of sensors that Beacon has deployed on the Hudson. VCs have already begun to make investments in companies (e.,g., Cloudera, Aster Data, Karmasphere) that are developing solutions for these problems.
3. The need to analyze mobile and social data. The continuous growth of the mobile internet and commerce, as well as the accelerating growth of social internet and commerce create new sources of data and the need for new types of analytics. 

One of the questions asked was whether IBM’s BI acquisitions are leading to an increased VC appetite for business analytics companies. Over the past couple of years business analytics, BI and data warehousing are experiencing a renaissance that is indeed leading to an increasing funding activity. However, this activity has nothing to do with IBM's acquisitions in this area. In fact, IBM's acquisitions only serve as additional validation of the importance of these areas. Forward-thinking VCs have been funding BI and DW companies for as long as BI existed, certainly before IBM started its BI acquisition spree. Over this period VCs have created many successful analytics, BI, and data warehousing (think of companies like Netezza, Datallegro, Omniture and more recently Greenplum). However, it is also true that certain subsectors are getting overfunded.

This was indeed a very interesting event and I look forward to reading the other blogs that will be written about it.

Thoughts on Big Data

Earlier in the year I wrote about areas of investment interest for 2010 and included Big Data aggregation, management, and data mining/processing.  A couple of weeks ago I had attended Gigaom’s Structure 2010, a conference devoted to big data.  As is also noted by Derrick Harris here there is a growing market of vendors that are working around open source tools to support big data initiatives.  I wanted to summarize my thoughts from the conference but travel got in the way.  When EMC’s acquisition of Greenplum was announced big data came to fore again since Greenplum had created a good implementation of MapReduce into its platform (even though I think EMC has been thinking along more traditional data warehousing lines with this acquisition).

1. Only 10-20 companies have a good grasp of big data issues and are innovating in this area.  Though many companies in the Fortune 1000 are starting to experiment with Hadoop, today only 10-20% of enterprises need big data solutions.  This number could grow as high as 40-50% in 5 years. 
2. NoSQL databases are emerging as the preferred systems for storing and managing big data sets.  The data in these sets is at the terabyte or petabyte scale, it is semi-structured, highly distributed, and much of it is of unknown value so it must be processed quickly to identify the interesting parts to keep.  NoSQL databases provide efficient and effective storage, management and processing of such data sets at low cost.  However, it is unlikely that these databases will evolve into general purpose platforms, like relational databases did.  It will therefore be important to match the big data problem being solved to the right database.  Data analysis and business intelligence are emerging as the best applications for taking full advantage of NoSQL databases.  Almost every company that presented at the conference discussed such applications.
3. Too many NoSQL database companies have already been created (Cloudera, 10gen, MongoDB, VoltDB, CouchDB, etc).  While the user interest in such databases is increasing (many Fortune 1000 companies have started Hadoop evaluation projects), the market won’t be able to sustain them.  I expect to see significant consolidation in the next 3-5 years.
4. Today there is no “LAMP stack” equivalent for big data processing and analytics and I think that there is an opportunity and need to create one.  As Jim Kobielus at Forrester wrote in his blog, there is not going to be a standalone market for Hadoop.  The pioneers of the big data movement, e.g., internet portals, social networks, ad networks, etc., are creating ad hoc “stacks” using open source tools for data management, data aggregation, in-memory storage.  In most cases they extend/modify these tools heavily.  In addition to base Hadoop, tools such as Hive, Cassandra, Scribe, memcached, MapReduce, Google App Engine are part of such stacks and, consequently can be the components of a pre-integrated, pre-certified Big Data Platform.  Using these tools still requires a programmer’s talents (PIG, and QL are efforts to provide a higher abstraction programming layer to Hadoop and Hive respectively so that database administrators can interact with the databases).  The Big Data Platform must be usable by business analysts, as well.  Hadoop client-side applications such as those being developed by Karmasphere and Datameer provide steps in the right direction but they remain standalone. 
5. For no reason apparent to me, NoSQL database companies are trying to reinvent the data warehousing and business intelligence infrastructures that have been created over the years. Some of the “reinventions” may be absolutely necessary and will lead to important innovations.  However, these companies appear to be also ignoring important aspects of the data management and data analysis technology that has been developed over the years around data warehouses built using relational technology.

Real-Time Analytics

Last Friday I attended SDForum’s The Analytics Revolution Conference.  The presenters were startup public and private company executives and investors.  The presentations were mostly around real-time analytics.  For the past few years Gartner analysts have been writing and talking about the Real Time Enterprise.  Gartner sees the analysis of the data that is generated by various enterprise applications such as an ERP and residing behind the corporate firewall and an important ingredient to achieving the Real Time Enterprise.  However, Friday’s presentations brought to focus that the impetus for real-time analytics is not the faster analysis of enterprise data, but of the Big Data captured on the Web (structured and unstructured social data, activity logs, data coming from mobile applications, geolocation data, data mashups, the interactions during the lifecycle of the captured data, etc).  This data is orders of magnitude larger and more complex than the data currently stored in the enterprise data warehouses.  I had written that Internet SaaS applications will drive the SaaS innovation agenda.  Friday’s presentations provided more examples and additional justification for this conviction.

Companies like Facebook, eBay, LinkedIn and Zynga that presented at the conference are interested in making analytics-driven decisions in Internet speed and cheaper than it is currently possible with the existing data warehousing and analytics technologies. They want to use their data to continuously optimize their businesses, simulate the effect of decisions on business processes, and forecast the impact of actions.  Making decisions based on reports that summarize and find trends in data that is a week or a month old simply won’t do.

To take advantage of emerging opportunity of real-time analytics of big data, we are considering investments in the following three areas:

1. Infrastructures of next-generation data management systems, including open source systems like Hadoop and derivatives, and systems for creating OLAP cubes in real time.
2. Horizontal and vertical analytic applications, examples of which are listed here.  Additionally, these applications will need to find a way to monetize on the data they capture, not only on the data they process.  See, for example, how internet data exchanges like BlueKai are able to monetize the cookie data they capture.
3. Analytic services offered around the captured big data, e.g., the services offered around big mobile data by companies like Ground Truth, and Flurry.

My underlying hypothesis (driving one of my investment theses, and shared by several of the investors present at the conference) is that cloud computing will make big data analysis faster and cheaper.

I am convinced that over the next few years, investments in each of these three areas will result in several strong exits.  I don’t believe that the established on-premise data warehousing, BI and analytics vendors will be able to create adequate technological and business model innovations around real-time analytics.  The results of a survey conducted by InformationWeek and published on March 27, 2010 make it clear that more enterprises are looking to cloud computing to address the perceived shortcomings of on-premise applications, with SaaS BI being one of the top areas of interest.  Finally, as stated here, incumbent on-premise software vendors will continue to face the innovator’s dilemma that won’t allow them to move fast to offer the novel solutions for real-time analytics that the market needs and is starting to demand. 

Presentation at Fordham University for IBM’s Launch of Business Analytics Curriculum

Last Wednesday I presented at an event organized by IBM and FordhamUniversity ’s business school, around the launch of a new business analytics curriculum sponsored by IBM.  Earlier in the fall, IBM announced the establishment of a NY-based analytics center with 400 consultants.  Through its collaboration with Fordham’s business school, as well as through a similar collaboration with North Carolina State University ’s computer science department, IBM is hoping to gain access to qualified and well-educated candidates to staff its analytics services organization that will grow to 4000 consultants. 

Through surveys that were recently completed by business and IT leaders around the world, IBM has determined that the corporate interest in business analytics and information-based (rather than gut-based) decisions is increasing, while the number of qualified candidates who can help such organizations remains small.  Similar conclusions have been reached by researchers at Villanova University's School of Business in a study found here.  The Fordham program will educate individuals on how to blend business with quantitative analysis skills.  Today universities graduate “pure quants” that find their way to many corporations from Wall Street investment banks, to internet social networks like Facebook.  Pure quants have obviously strong quantitative analysis skills but relatively weak business skills.

I was asked to talk about the drivers for business analytics and their impact on investments in new analytics companies.  For several years Trident realized the importance of analytics in business decisions and has invested in several software companies that either develop applications and platforms to support the creation of business intelligence and analytics or their business is mainly driven by analytics.  Today almost 20% of our active portfolio companies belong to these two categories.  Moreover, we continue to look for additional investment opportunities in these areas here and abroad.

I see three drivers for today’s growing investor and corporate interest around business analytics:

1. Big data.  Data is becoming strategically important to enterprises of any size and type.  It is also being generated in unprecedented volumes. This is particularly the case with Internet companies.  Yahoo, Fox, AOL and some of the larger ad networks routinely generate upwards of 100 TB of data per year each.  Facebook is generating an order of magnitude more data than that.  Once prepared for analysis, the size of each such data set can triple in size.  By comparison, in the 90s, when I was running IBM’s BI Solutions organization and later when I was the CEO of Customer Analytics, we were dealing with data warehouses that contained, at most, a few terabytes of data and most frequently only a several hundreds of gigabytes.  To analyze big data in an impactful and timely manner, we need new data management paradigms (e.g., we are starting to see the broad use of column-based databases, and the emerging use of Hadoop by a variety of organizations across several industries) and new analysis paradigms that efficiently combine solid analytic techniques with business/industry knowledge, practices, etc.
2. The quest for performance-driven decisions.  Corporations are moving from report-driven decisions (where business intelligence was used only to passively present historical data so that a business executive can make decisions) to performance-driven decisions where analytics are used to, oftentimes automatically make decisions that impact corporate performance.  For example, ad networks combine sophisticated analytics with novel data management to determine within a few milliseconds which ad to show an Internet user.  Sophisticated optimization techniques are used by ecommerce companies to determine the price of keywords used in search advertising or the price they will pay for a new customer lead.
3. The need for broadening the corporate use of analytics.  In their quest to achieve analytics-driven decision-making, corporations are accelerating the use SaaS BI and cloud-based analytic solutions because of their lower total cost of ownership, speedier implementation compared to their on-premise equivalent solutions, and support of community-based and collaborative problem-solving.

Realizing the impact of analytics-driven decision-making, corporations are now thinking about analytics while designing new business processes and the associated applications, rather than after the fact, as was happening to date.  By comparison, we used to build data warehouses and BI applications by extracting and reporting on data from systems that had been around and were never designed for analytics-driven decision-making, e.g., ERP systems, or supply chain management systems.  Today, most state of the art applications are designed with an analytics component instead of being fitted with one. 

Over the next couple of years we will be able to assess whether Fordham’s business analytics curriculum will have the desired impact and provide IBM with a rich pipeline of analysts that combine the right mix of the business and quantitative skills needed to satisfy the corporate demand for analytics-driven decision-making.

Interview: Venture Capital Investor Sees Solid Future for Cloud BI, DW

Original content by Linda L. Briggs

Source: http://www.tdwi.org/News/display.aspx?id=9608

The appeal of the cloud is catching on in the business intelligence and data warehousing world, where cloud-based data warehouses and software-as-a-service (SaaS) BI can offer clear benefits in both data warehousing and BI. Those benefits include lower initial capital expenditures -- especially attractive in a weak economy -- a shorter time to deployment, and options for collaborations between divisions and even across company borders.

As a managing director investing in information technology for Palo-Alto-based venture capital firm Trident Capital, Evangelos Simoudis is admittedly bullish about SaaS BI and cloud-based data warehousing. In this interview, he explains why, as well as where he sees SaaS BI headed despite the recent demise of one of the first SaaS BI companies. He also discusses how companies can mitigate the risk of moving data warehousing to the cloud.

Simoudis, who has more than 25 years of experience in information technology, holds a Ph.D. in computer science from Brandeis University and a B.S. in electrical engineering from Caltech. He joined Trident in 2005 and invests in Internet and software businesses. He is a director on the boards of two BI companies, Pivotlink and Host Analytics.

BI This Week: As a venture capitalist watching technology markets, what do you see as the immediate and long-term future for BI offered as an off-premise service?

Evangelos Simoudis: Clearly, there's a renaissance of sorts with BI and analytics in general. Businesses, regardless of industry sector or size, are becoming analytics driven. Online businesses in particular have led the way in demonstrating the effective use of data to make meaningful decisions.

Seeing how online businesses (for example, Amazon's use of data about customer activity to make its product recommendation engine more effective) are analyzing their data with good results, other corporations now want a quick path to analytic information. Unfortunately, though progress has been made in the 25 years that BI solutions have been available, the reach of BI within corporations is still limited. Even within the Global 2000, estimates are that on average, only 20 percent of corporate users that might benefit from BI and analytics, actually use any type of formal BI solution.

We can attribute this low adoption rate to either high cost or the perceived complexity of most solutions. Smaller companies, in particular, often lack the financial resources to purchase a costly solution or the IT resources to properly deploy and support it.

Even with the slower adoption rate for formal BI solutions, corporations usually have dozens or even hundreds of small ad hoc data marts in house, built using inexpensive databases such as MySQL and Microsoft Access, often accessed through Microsoft Excel. These applications are typically maintained by their business owners rather than by IT personnel, because while their content is valuable, they aren't viewed as strategic enough to warrant IT resources.

SaaS BI solutions can effectively address all of these less-than-ideal situations. Such solutions are simpler to deploy, require few corporate resources to manage and maintain, and have a lower total cost of ownership (TCO) than on-premise BI solutions. As a result, SaaS BI solutions are emerging as a strong alternative for companies of any size that aim at analytics-driven decision-making.

Here are just a few of the unique advantages that SaaS BI solutions offer over their on-premise counterparts:

The wisdom of the crowds: By monitoring the use of a SaaS BI solution, the vendor can (a) dynamically optimize it to improve performance and (b) determine which features are being used effectively, which features present difficulty to the user although users find them important (thus enabling the vendor to offer training), and which features aren't being used at all, enabling the vendor to remove them from the application and avoid maintaining them over time.

Customer metrics: This information allows the vendor to make immediate, incremental improvements to the product. That is difficult to do when the vendor's solution is on-premise with each customer. The bigger point here is that vendors of on-premise solutions don't know how their software is being used. SaaS vendors, in contrast, can closely monitor usage and even provide user metrics to customers in the way Nielsen and comScore do.

Leveraging collective knowledge across divisions and companies: Companies whose data is in cloud-based warehouses can easily choose to share or combine certain data sets with other companies using the same vendor, thus enabling collaborative problem-solving. Employees within a company but separated geographically or by division can more easily collaborate with colleagues and partners by looking together at a single view of internal and external data, all housed in the cloud.

Decisions that are most effective in today's business world are often based on data blended from inside and outside the company. I'm already seeing early examples of this type of collaboration, and I expect collaborative BI will accelerate in the next five years; SaaS solutions will be the catalyst. Specific analysis results and associated data can also be tagged (using systems like the ones found today in consumer Internet products such as Yahoo's Delicious). Tags can be searched and shared, enabling yet another level of information use and collaboration among users.

Given those advantages, the prospects for SaaS BI seem very good. However, what does the recent demise of SaaS BI company LucidEra indicate?

I continue to consider the near- and long-term prospects of SaaS BI solutions to be extremely bright. LucidEra's recent failure is no different than what we investors see in every other early market opportunity, because even though BI has been around for many years, SaaS BI is relatively new. In new markets, some companies succeed and some fail.

I'm sure LucidEra's failure is causing companies that are already using SaaS BI solutions, or are planning to use them, to once again go over the soundness of their decision. However, my prediction is that this will only serve to improve how companies interact with their SaaS application vendors, including the ones that provide BI solutions. That is, companies should become more proactive about asking for detailed specifications on the vendor's data policies, disaster recovery policies and plans, and protection of customer data should the vendor fail.

How should organizations think about data warehouses in the cloud?

Cloud-based data warehouses offer smaller companies their first true opportunity to take full advantage of data warehousing in a cost-effective manner. By the same token, large companies are interested in the potential of cloud-based data warehousing solutions to speed up the creation of data marts from enterprise data warehouses. They can do this while reducing development time and maintenance costs, thus improving the "time to value" of data by making data available to analysts faster.

Finally, cloud-based data warehouses can provide companies with elastic capacity to help address the usage spikes that are common with mission-critical data warehouses and marts.

What sorts of concerns do you hear from IT executives about cloud-based data warehouses?

I'm hearing a number of legitimate concerns, which vendors will need to address in order to move cloud-based data warehouses forward.

Security: As prominent corporations create cloud-based data warehouses and draw visibility to the technology, CIOs fear that the cloud will become a prime target for hackers who would want to exploit the existence of large databases with prized corporate data, all in a single place.

Data Integrity: IT must guarantee that the data stored in a company's operational databases is synchronized with reasonable frequency with the data stored in the cloud-based data warehouse to ensure the cloud-based DW has the correct data to drive analyses.

Data Ingestion Throughput: This is not an issue in the short term, but will become more important as ever-larger data warehouses and marts are implemented. The size of the data sets and the time it takes to ingest them will become issues around which IT organizations will need to obtain guarantees. This issue will also need to be considered if a cloud-based warehouse is used to provide elastic capacity to address usage spikes.

Service-level agreements (SLAa): The cloud data warehouse vendor must be able to address SLAs such as uptime, query response time, and disaster recovery plans. For backup and recovery in particular, any cloud-based software vendor, including those that provide data warehousing services, should have automatic failover capability to ensure the customer won't experience any service interruptions. In addition, and depending on the critical nature of the analytic applications supported by the cloud-based DW, the customer may want to insist that the vendor use an alternate cloud vendor for backup and disaster recovery services.

The Ability to Take Back the Warehoused Data: IT must ensure that the corporate data will not become hostage to the cloud-based DW vendor. To this end, the vendor needs to make it easy for the customer to take back its entire data set, either because of a switch in vendors or because the vendor is going out of business. The cloud-based DW vendor also must make it easy for the customer to obtain subsets of stored data to use with other analytic applications offered by other on-premise or SaaS vendors.

Warehouse Auditing: IT must be able to audit the warehouse in the cloud in the same way that they audit any on-premise data warehouse, for issues including data integrity, query efficiency, and information delivery reliability.

How should companies integrate data from cloud-based transactional applications into their SaaS BI systems?

The first generation of cloud-based data warehouses are actually data marts that provided reporting either on data stored in a single application (for example, Salesforce and Netsuite are already offering reporting functionality with their applications; SaaS BI vendors such as Pivotlink also provide solutions that analyze the data stored in SaaS applications) or on data that is extracted from existing on-premise data warehouses. The next wave of cloud-based data warehouses will integrate data from several SaaS applications.

SaaS application vendors are just now starting to develop APIs that allow the type of bulk data extraction necessary for the creation and updating of data warehouses. We are also starting to see the emergence of SaaS data extraction and integration vendors (some with proprietary software and others with open source software) whose tools are used in conjunction with these cloud-based data warehouses.

Today I can think of three types of risk related to data integration into cloud-based DW:

• Immature APIs: The APIs provided by SaaS applications are not yet mature, since their first use is transactional operations rather than analytical/warehousing operations. Since the operational databases are under the control of the vendor rather than the customer, it's up to each SaaS vendor to place the correct priority and develop APIs for warehousing operations.
• Immature ETL Tools: ETL tools for SaaS are improving in their ability to deal with data from a single data source, but are just now being tested on multiple data sources distributed across several SaaS application vendors.
• Distributed Data: Source data is becoming distributed across SaaS application vendors with different capabilities and priorities. That means that coordinating, synchronizing, and moving the source data across the cloud and into the cloud-based data warehouse needs to be taken into account when planning such a cloud warehouse, because the warehouse's owner has less control over the entire operation than with an on-premise equivalent.

Does that mean that having federated source data across multiple SaaS application vendors makes it harder to create a cloud-based data warehouse?

As I mentioned, we are still in the very early stages of the thinking regarding cloud-based data warehouses. Companies are most commonly at the stage of warehousing data from a single source (a single SaaS application, a single on-premise application, or another on-premise warehouse).

Companies are not yet warehousing data from multiple SaaS applications -- that still represents state-of-the-art practice rather than a mainstream practice. For the progression to be successful, IT organizations in collaboration with business users will need to exhibit new and different thinking. Will organizations be willing to devote resources to create this type of multi-source database cloud-based DW? That will depend on their overall experiences with first-generation cloud-based data warehouses. In any event, the next five years will be an active and exciting period for SaaS BI and cloud-based data warehousing.

 

Lucidera's Demise

Much has been written over the last week about the recent demise of Lucidera, one of the on-demand BI pioneers.  Some writers saw it as a general sign of the current economic times during which venture funding has become harder to raise. Others saw it as a direct consequence of the unsustainability of Software as a Service models.  In my opinion, none of these are the reasons for Lucidera’s failure.

Lucidera started as a well-capitalized startup.  The company had raiased over $20M in 2 rounds of funding from very well-respected investors with long experience in funding successful IT startups.  In addition to its strong founding team it had also attracted top management talent, most recently Rob Reid who last year became the company’s CEO.  Lucidera benefited from the strong overall interest (by customers, investors, and executives) in on-demand software and its application to business intelligence. So, what went wrong?

My suspicion is that Lucidera made two wrong choices which led to its predicament.  The first wrong choice was around its platform.  To expedite its time to market, Lucidera started building its on-demand BI platform by modifying an existing platform that had been developed by Broadbase.  While at the time this might have appeared as a brilliant idea, my sense is that the company couldn’t make things work out the way it was expecting.  Perhaps the platform was not scaling to the expected levels.  Or modifying it to make it into a modern multi-tenant SaaS platform was taking longer and costing more than expected.  The wrong choice of platform could also have another financial consequence: operating the platform was more expensive than originally projected.  This means that to properly scale and process the data sets the customers wanted to analyze, more hardware than the team had originally planned was necessary, implying that the company was burning through more money than it had anticipated.  My suspicion is that Lucidera’s management team may have decided to focus on specific analytic applications in order to finesse some of the platform issues.  Unfortunately, the second mistake was due to the applications chosen to be built on top of the platform.  While I’m sure these applications were selected in order to address a genuine market need, I wouldn’t be surprised if their selection was also due to the underlying platform’s shortcomings.  However, I think that customers were not willing to pay as much for these applications as the company may have expected.  If this is true, then it would imply that the company was burning through cash faster than it had anticipated.

Its burn rate caused Lucidera to seek a new round of funding in the fall.  It is true that last fall was the absolute worse time for companies to be raising money.  However, the timing alone was not the reason Lucidera failed to raise money.  While last fall venture investors had turned mostly inward and were scrutinizing potential investment opportunities much more than usual, financings did get done, especially in companies that were performing well.  For example, Pivotlink, one of my portfolio companies and a direct competitor to Lucidera, closed its most recent round in January.  Good Data, another Lucidera competitor was funded about the same time.  I’m sure the investors that were considering Lucidera had difficulty seeing how the company’s business model around the analytic applications it had developed was going to succeed, particularly during these economic times. As a result, investors didn't want to invest additional money since they couldn't see their way to a return. 

Some also say that investors may be having second thoughts about funding SaaS companies today because on-demand software models are not capital-efficient.  They claim that the reason for this inefficiency is because SaaS vendors must invest upfront for the creation of the infrastructure (including the data center) on top of which the on-demand solution is running.  While it is true that SaaS companies require more upfront capital to set up their infrastructures than their on-premise counterparts, more recently, as a result of the lessons that have been learned from the SaaS pioneers, and the use of open source software and commodity hardware, the capital requirements of SaaS companies have decreased significantly making on-demand software companies as capital efficient (if not more) as on-premise ones.  Moreover, judging from the financing activity of the last 6 months one can safely conclude that SaaS may be the only software models funded by venture investors.

Will Lucidera’s demise have a long-term impact on on-demand BI?  I don’t think so.  Lucidera’s customers may spend some time thinking whether to sign up with another SaaS BI vendor, even though both Pivotlink and Good Data have announced programs for taking over Lucidera customers.  However, demand for SaaS BI solutions is growing.  I can state this not only because of Pivotlink’s and Host Analytics (another of my SaaS BI portfolio companies) success, but also because of discussions with large on-premise BI vendors all of which are readying SaaS offerings.  In fact, SAP/Business Objects will introduce a new offering this fall.  The demand is growing because corporations of any size and industry are starting to realize the attraction of on-demand BI’s value proposition (time to value, strong and quick ROI, small set of preconditions).  You don’t need to: (a) have your data already in the cloud, (b) work with only small data sets, or (c) try to address simple problems before you can effectively utilize on-demand BI, as some people claimed.  As with on-premise BI approaches, you need to start with the right vendor that is adequately capitalized, has the right platform and the right business model that allow it to grow profitably and quickly become a self-sustaining company.

 

New Analytic Applications for the New Data Sets

On a daily basis Internet publishers (e.g., Yahoo, MTV) and Internet applications such as e-commerce sites (e.g., Amazon, eBay), social networks (e.g., Facebook, MySpace, Tweeter), and ad networks (e.g., VideoEgg, Valueclick) generate very large data sets with new types of data.  For example, a site like MTV.com may generate 90TB of raw data per year which, after being augmented with demographic and geotagging data, can easily balloon to 700TB.  A recent post on the management of large data states that eBay has a 6.5PB data warehouse and Facebook a 2.5PB data warehouse.  Facebook is capturing 15TB of data daily.  This new, Internet-based data consists of various types of logs, user generated content, etc.  The size of these data sets dwarfs the corporate data, e.g., sales transactions, collected and stored in more “traditional” data warehouses used by the non-internet members of the Global 2000.  These “traditional” data warehouses typically store 600GB-1TB of data.  Most mid-size companies, i.e., companies that do about $400M in annual sales, operate even smaller data warehouses that rarely cross the 300-400GB level.  The existing analytic tools and applications, e.g., Business Objects, or Cognos, were not developed with the intent of operating on anything that resembles the king-size, Internet-based data sets.

During the last couple of years we’ve seen significant innovation in the area of the area of data management, first with the introduction of data warehouse appliances by companies such as Netezza, Datallegro, Greenplum (that were based on relational database technology) and more recently with the introduction of appliances such as Aster’s and Vertica’s that are using column-based databases.  The latter two are starting to be adopted for the management of the Internet-based data.  We have also seen the development of systems such as Hadoop that provides a framework which applications can use to work on very large data sets.  These products are maturing quickly and their use is significantly reducing the cost of managing very large data sets.

While companies are making good progress on managing these very large data sets, their ability to effectively and efficiently analyze these sets is lagging.  Companies like Google, eBay and Yahoo are using internally-developed frameworks (e.g., Google’s MapReduce) and home-grown routines to analyze the data they generate because, in most cases, the existing analysis products they throw at them can’t scale to operate on these sets, or don’t have the necessary functionality to address the questions that must be answered through these analyses.  Sample questions that e-marketers (who represent only one of the constituencies that need to analyze this data) are trying to answer include:

1. What should my keyword-bidding strategy be (which keywords, what price) for each of Google, Yahoo, and MSN?  How should I allocate my budget between SEM and SEO?
2. Which ad networks are giving me the best performance?
3. Across which channels and at what percentages should I allocate my marketing budget to reduce lead acquisition costs and increase conversion rates?

Over the past few months I’ve been meeting with several startups that are developing new analytic applications to address such questions, and am particularly excited about the data analysis innovations they are working on.  Because more money is shifting online and the importance of the decisions made using this new data is rapidly increasing, this area will attract strong investor interest and has the potential of producing several winners.

Data Warehousing in the Cloud

Companies that provide on-premise BI solutions, like Business Objects, Microstrategy etc. connect to data warehouses that are built on top of third-party databases like Oracle, DB2, SQLServer, PostgressSQL and MySQL.  These database management systems, as well as data warehouse engines such as Teradata’s, provide the functionality and utilities for the creation, access, and management of a data warehouse or data mart.  The current generation of on-demand BI solutions like those offered by Pivotlink, Lucidera and Birst depart from this model by integrating a query application with a database management system both of which reside in the cloud.  More recently, data warehouse appliance vendors such as Aster Data, and Vertica started offering cloud-based options of their appliances.  Through these options they allow companies to develop data warehouses and data marts completely in the cloud and access them via on-premise or on-demand query applications. 

Cloud-based BI solutions that integrate a query application with a data mart are not entirely new.  Data marts containing web analytics data have been offered almost exclusively as cloud-based solutions by companies like Omniture and Google.  Corporations using these SaaS solutions are storing increasingly more complex and mission-critical data in these data marts, including customer data. Should on-demand BI vendors continue offering solutions that integrate the query application with their own data management system, or should they start partnering with the vendors that offer cloud-based data management systems which can be used to be standalone data warehouses in the cloud?  The answer to this question depends on your point of view regarding on-demand BI and data warehousing.  I’ll consider three perspectives: the on-demand BI solution vendor’s, the end user’s, and the IT user’s.

 The On-Demand BI Solution Vendor

For the on-demand BI solution vendor this is not an easy question to answer.  Having full control of both the application layer and the database layer offers its advantages but also has drawbacks.  The major advantage comes from the vendor’s ability to better control how data is loaded to the data warehouse, how it is organized and how it is queried. 

The ability to load data quickly allows the vendor to work with larger data sets during smaller time windows as dictated by the customer, as well as service more customers at any one time.  While smaller companies may only refresh the data in their data mart weekly, larger customers typically refresh data daily.  Moreover, the transactional database of such larger customers can only stay offline for short time periods during which data must be transferred to the data marts.  For example, a retailer may only provide a 3-4 hour window daily during which sales data can be uploaded to a data mart.  If such an operation has to be performed for hundreds of the SaaS BI vendor’s customers, then one can appreciate the importance of fast data loading. 

Data loaded in the data warehouse has to be organized in the best possible way to enable the optimal execution of queries.  The managers of on-premise data warehouses and marts constantly look at the queries executed against their databases to determine how to best organize the data to achieve best query execution times.  Some of these optimizations can be performed automatically by the database management system but most require manual intervention.  Having full control of the database management system enables the SaaS BI vendor to better optimize the organization of the stored data.

Finally, the expressiveness of the language used to query the data mart determines the range of analyses that can be performed and reports that can be created by the BI application, and also contributes to the speed with which queries are executed.  Even though SQL is a standard all database management vendors implement their own extensions, including several that are specific to data warehousing, in order to add to the language’s expressiveness.

Creating a strong database management system to support data warehouses and data marts for a variety of analytical applications is hard.  The areas mentioned above such as data loading, query optimization, query management, etc. are difficult to address and require specialized knowledge.  As many database management vendors have shown over the years (from Teradata to Red Brick, Netezza and Aster Data) good database implementations require significant R&D investments.  By decoupling the database from the application, the on-demand BI vendor will be able to focus on improving the application’s functionality without having to worry about the underlying database system.  However, despite the higher investment necessary, I think that in the short term (at least the next 2-3 years) it will be difficult for on-demand BI vendors to decouple their applications from the databases and go with a third-party offering.  In order to provide strong ROI to their customers, on-demand BI vendors will need full control of the stack as they learn about the application areas where on-demand BI will provide a strong alternative to its on-premise equivalent. 

The End User

An end user subscribes to on-demand BI applications either because his organization doesn’t have an in-house BI solution and developing such a solution internally is deemed as too expensive and requiring too much time, or it does but the solution is hard to access.  Such a user doesn’t care whether the data is stored in the cloud-based data warehouse of the vendor who provides the on-demand solution, or in a third-party’s cloud-based data warehouse.

The end user cares about convenience and analysis effectiveness.  “Convenience” means that the end user would prefer an on-demand vendor that provides a one-stop to BI.  “Effectiveness” means that the user is interested in quickly analyzing data to achieve a specific goal, e.g., determining which merchandize to discount in which regions and for what time period.  During the analysis task the user is concerned about securely loading the right data to the data warehouse, quickly and easily interacting with that data to produce the desired results and reports, and keeping the data stored safely for as long as it’s necessary. 

The on-demand BI solutions available to date have been developed with such a user in mind.

The IT User

The IT user has only recently started entering the conversation around on-demand BI solutions.  This is happening as more of the larger companies that actually have IT organizations have started using SaaS BI solutions in conjunction with (or some times instead of) their on-premise ones.  IT organizations worry about the security of the data that moves from their companies’ internal databases to the on-demand vendor’s data warehouse, the integrity of the data sent to the vendor, since they must guarantee that the data stored in a company’s internal databases is always synchronized with the corporate data stored in SaaS vendor’s data warehouse, and their ability to take back the data when the need arises.   

IT must feel confident that the on-demand BI vendor uses the strongest possible security measures to safeguard the data in the warehouse.  The SaaS vendor is expected to perform at a higher standard than the IT organization itself when it comes to security.  CIOs are starting to fear that as SaaS vendors become better established with growing client rosters that include prominent corporations, they will become prime targets for hackers (individuals or groups, independent or government-sponsored), who would want to exploit the existence of large databases with prized corporate data in a single place.

Today every on-demand BI vendor requires that they be provided either with behind the firewall access to corporate databases so that they can extract data as necessary, or with the appropriate data sets to load into their cloud-based data warehouse.  Most corporations provide the SaaS vendors with data extracts because it’s safer than giving them access to the corporate databases.  However, this approach, while safer, results in multiple copies of the corporate data.  Keeping this data updated and synchronized, so that the analyses resulting from the extracts can be impactful, can become a hard and expensive task that further taxes IT resources. 

Finally, IT must ensure that the corporate data will not become hostage of the on-demand BI vendor.  To this end, the vendor must make it easy for the customer to take back its entire data set either because of deciding to switch on-demand BI vendors, or because the vendor is going out of business.  An approach to addressing this problem may be provided by Cloudkick.  In addition, the on-demand BI vendor must make it easy for the customer to obtain subsets of the data stored in the cloud-based data mart to use it with other applications.

I wouldn’t be surprised if the majority of IT users would prefer a cloud-based BI application that accesses data stored behind the corporate firewall and doesn’t move the data permanently to the cloud.

The growing success of on-demand BI applications will lead to a lively dialog among these three constituencies on how such applications should be partitioned among the various types of cloud-based vendors and the clients themselves.

 

The Case for On-Demand BI Solutions (Part 2)

I have been a bullish investor of companies that provide on-demand BI solutions (to date I invested in Pivotlink and Host Analytics).  After being involved with BI for over two decades I see SaaS as a fundamentally transformational model whose benefits outweigh the adoption issues I described in my last posting.  Moreover, these issues are not “show stoppers.”

 The value and uniqueness of on-demand BI is coming from the following areas:

1. Cloud-enabled collaboration. Collaborative BI can effectively be enabled through cloud computing.  Companies want to collaborate through data.  Employees want to collaborate with other colleagues and partners by looking together at a single view of internal and external data that impacts daily their business “ecosystem.” Today the only decisions that are effective are those based on data blended from inside and outside the company.
2. More efficient analysis paradigm.  On-demand BI transforms the "traditional" ETLA (Extract Transform Load Analyze) process found in most data warehousing/BI environments, to an ELA process (Extract Load Analyze) process. This doesn't simply mean that you are removing a costly step (Transform) from the process, but that you are changing the entire analysis paradigm. Because on-demand BI applications make Extraction and Loading very cheap, the user of an "ELA environment" doesn't have to be extremely thoughtful on what data to include in the warehouse (or data mart) and how to transform it in order to improve the analysis results. Instead he can load a bunch of data and "experiment" with it. If the experimentation goes well then he continues. If it doesn't go well then he "throws the loaded data away" and load some more without delay or guilt. On-demand BI companies like Pivotlink have been developing extremely fast data extractors and loaders in order to better support the ELA paradigm for an expanding set of cases.
3. Social networking/Community.  Social networking and other social media applications have demonstrated the power of communities and are now moving from personal use to enterprise use.   Community-based tools built around on-demand BI solutions can increase a BI solution’s adoption, reduce service delivery costs and, through the established dialog between users and the solution provider, improve the application’s feature set and performance.  Imagine, for example, through a presence tool, such as Microsoft Messenger, a user determines that another user who is an expert on inventory analysis is online.  By looking up this user’s profile he finds out that the user has received good recommendations from co-workers and customers about his ability to create inventory analysis reports.  A real-time dialog can begin and lead to collaborative problem-solving.
4. Better maintenance of the application and the data. Because the application/reports and the database/data warehouse are located in the vendor’s data center, software updates, database maintenance for optimal query performance, report organization and other tasks associated with the support of a properly running BI solution are managed and performed centrally by the vendor.  This activity results in optimally running BI solutions enjoyed by all users.  The performance of on-premise BI solutions is dependent on the IT organization’s abilities to a) devote the right resources to address the above mentioned tasks along with software updates and patches coming from the vendor and b) develop local expertise to optimize the performance of each BI solution being used.  This may be possible in certain companies with large and sophisticated IT organizations but, unfortunately, it is not found in every company that uses on-premise BI solutions.  For example, a recent Oracle critical patch bundle included 16 updates for Oracle Database Server, 12 updates for Oracle Application Server, 3 updates for Oracle Applications, 4 updates for Oracle PeopleSoft and JDEdwards Suite, and 8 updates for BEA Products Suite.
5. Usage optimizations. By monitoring the on-demand BI solution’s usage, the vendor can a) dynamically make optimizations to improve its performance, and b) determine which features are being used effectively, which features are not being used at all, which features present difficulty to the user (even though they are perceived as useful) and also make improvements as appropriate.  There is a bigger point here relating to a vendor’s monitoring a SaaS application’s usage.  It should first be noted that this type of monitoring cannot be accomplished for on-premise applications.  The vendors of such applications don’t know how their software is being used.  On-demand application vendors have the opportunity to closely monitor usage to identify: a) if the application is used as expected, b) which parts of the application are being used the most (not unlike how a web analytics package reports on which web site pages are being visited), and c) even provide cross user statistics in the way Nielsen and comScore report various types of audience metrics.
6. Easier to avoid vendor lock-in particularly now that the major on-premise BI vendors (Cognos, Business Objects, Hyperion) have been consolidated. Because of how critical BI is to every corporation, customers should always try to select the BI vendor with the most-suitable solution rather than the vendor that offered them "a great deal" on BI by bundling it with a bigger IT contract, as the major IT vendors are likely to do.
7. Innovation.  The consolidation of the major on-premise BI companies can also result in slower innovation since they are now divisions of larger IT vendors that may have different overall objectives than to provide innovative BI solutions.

 Areas that are impacting the TCO of an on-demand BI solution:

1. Fewer personnel needed to support the application.  Moreover these personnel don’t have to possess specialized knowledge in the way that personnel that support on-premise BI solutions must have.  BI solutions have been particularly personnel-intensive (database designers and administrators for the data warehouse, programmers and analysts to create and maintain reports, data analysts to extract information from the collected data).
2. No upgrades to hardware.  The dissemination of on-premise BI solutions within the enterprise has been frequently stymied by the need for upgrades so that client machines can satisfy the processing needs of BI tools.  On-demand BI solutions are not only accessed through a web browser obviating the need for hardware upgrades but also resulting in fast deployments of these solutions.
3. Cheaper implementation of a data warehouse/data mart because using a standard data model is faster and can benefit from the improvements (functional, architectural, implementation) performed by the vendor’s dedicated staff for all customers.
4. No hidden costs in terms of solution upgrades, customizations, systems and data integration.
5. Dedicated vendor personnel keeping the data warehouse in top shape (through data model refinements, data utilization, additional aggregations, etc.) and disaster recovery.  When considering the cost benefits of SaaS, many companies do not include the value of transparent and immediate upgrades to the latest software version.  Companies deploying traditional on premise solutions must account for the ongoing cost of planning, implementing and supporting new software upgrades and bug fixes.  This can easily cost from tens to hundreds of thousands of dollars in professional services, IT resources and duplicate hardware and it usually involves running parallel applications during the implementation, testing and acceptance period. It is not surprising that the average ERP implementation is typically 2-3 versions behind the latest software update.

 BI continues to score high as an important corporate solution even during this time of shrinking IT budgets.  On-demand BI addresses many of the issues that had plagued on-premise BI solutions and offers some unique advantages making such solutions possible for more and larger segments of the market.  As such it warrants serious consideration by small and large companies alike.

 

The Case for On-Demand BI Solutions (Part 1)

In recently published CIO surveys conducted by Morgan Stanley (see below), Gartner, and Forrester, BI consistently scores high as a must-have solution and likely to experience little or no cuts in the current environment.

 

After participating in discussions on new BI solutions (here and here) I walked away thinking that while the interest in on-demand BI solutions is increasing, the SaaS delivery model for BI is not yet broadly accepted.  Five issues are perceived as obstacles to the adoption of on-demand BI:

• Data security.  Because BI can involve the analysis of sensitive data, e.g., personnel data, companies hesitate moving this data outside their firewall.
• Total Cost of Ownership (TCO).  The TCO of on-demand BI solutions is not significantly lower than that of corresponding on-premise solutions.
• Application uptime.  On-demand BI applications are not as reliable as their on-premise counterparts to support operational decision-making.
• Data uploading speed.  State of practice communication infrastructures don’t allow large quantities of data to be uploaded to the cloud fast enough to enable, through the use of BI tools, the creation of necessary insights in a timely manner.
• Application sophistication. The on-premise reporting tools, e.g., Cognos, Business Objects, provide richer functionality than their on-demand counterparts.

 

As the data from a recent Goldman Sachs CIO survey indicates (see below), the first three of these issues (security, TCO, and application uptime) are customer concerns for every other on-demand application, and are not specific to on-demand BI.

Source: Goldman Sachs CIO Survey

 

Data uploading speed impacts BI solutions more specifically since such solutions are by nature data-intensive and, in order to be effective, applications such as price optimization, markdown analysis and others require quick turnaround of data.  Finally, functionality comparisons between on-premise and on-demand equivalents have been offered as arguments against the use of any type of on-demand application, from word processing to ERP.

 

Yet all these issues have not slowed down the rapid adoption of SaaS applications.  For example, the growing popularity of Human Capital Management SaaS applications, like Workday and Success Factors, indicate that the companies of every size are satisfied with the ways such application providers address data security.  Similarly, the growth of on-demand ERP systems, such as NetSuite’s, points to the customer acceptance of the uptime performance of these mission-critical applications.  Successful on-demand price optimization applications, like DemandTec’s, show that uploading large data sets is being addressed effectively by at least some on-demand BI vendors.  It is true that certain power users of on-premise reporting and analysis tools may find that the on-demand BI tools lack some functionality.  However, the majority of the users will find that the on-demand BI tools have all the functionality that is necessary to enable them to work effectively. 

 

More and larger companies are increasingly relying on SaaS applications to run critical parts of their businesses.  The adoption trends we initially observed with on-demand CRM and productivity applications and later with ERP applications we are now starting to see with on-demand BI applications.  In the next post I will discuss the value and uniqueness of on-demand BI and of the areas that are impacting the TCO of such solutions.