The challenge of "big data" represents a significant business opportunity for software companies today. Business customers are challenged by escalating amounts of heterogeneous data that they want to mine for business value. But traditional data management tools are not meeting that challenge. If you or your software firm have expertise in data analysis, data storage, search or realtime analytics then now is the time to really take advantage of this wave.
So what do we mean when we use the term big data? Basically, it refers to any data analysis problem that involves the three V's:
• Volume (large amounts of data that cannot be easily handled with traditional relational database tools)
• Variety (different types and sources of data, such as structured/unstructured, cloud/data center/local, etc)
• Velocity (rapid streaming of data, data analysis that demands near real-time results, low latency requirements)
Big data is not only about the total size of the dataset. There are plenty of very large datasets that can be effectively analyzed with traditional relational database and data warehousing tools. Big data is used to describe data which has volume/variety/velocity characteristics which make it very hard to analyze with traditional tools.
Established firms and startups alike are using the "big data" term to describe their offerings. Some of this is hype, some is smart marketing, and some is real innovation that is happening in this space.
For example, EMC, the traditional enterprise storage company, is focusing heavily on big data in their marketing initiatives because big data drives the need for storage. So rather than just doing a storage product pitch they wrap that in a big data solution message.
Other established tech companies who are playing in the “big data”/”cloud data” space include IBM, Oracle, Microsoft, SAP, and Teradata.
Many of the traditional IT companies have been busy acquiring big data capabilities. IBM acquired Netezza which offers a high-performance analytics data warehousing appliance. EMC acquired Greenplum and introduced their big data analytics platform in the form of the EMC Greenplum Modular Data Computing Appliance. Other acquisitions include Oracle’s purchase of Endeca, and enterprise search company, and Hewlett-Packard’s acquisition of data analytics vendor Vertica.
SAP also joined the big data bandwagon, with advanced support and integration with big data Hadoop environments.
But the true innovation is happening in the startups. Noteworthy big data startups include Splunk, Aster Data (acquired by Teradata), Tableau, Cloudera, Cataphora, and Opera Solutions.
A report by Wikibon evaluated the big data market. They estimate a current big data market size of about $5B, and project it to grow an order of magnitude by 2017 to $50B worldwide, with an annual growth rate of a very strong 58%. IDC provided a somewhat more conservative market size estimate, but still with healthy growth - $17B in 2015.
Pure big data suppliers only account for about 5% of total revenue but they deliver most of the new innovations and approaches to data management and analytics.
The industry sectors that will focus most heavily on the need to analyze big data are energy, retail, financial services, manufacturing and healthcare.
This big data challenge represents real opportunity for software companies and other tech firms alike. Dan Vesset, program vice president, Business Analytics Solutions at IDC, noted that there are "opportunities for both large IT vendors and startups. Major IT vendors are offering both database solutions and configurations supporting Big Data by evolving their own products as well as by acquisition. At the same time, more than half a billion dollars in venture capital has been invested in new Big Data technology."
Likewise, McKinsey found that "Big Data will also help to create new growth opportunities and entirely new categories of companies, such as those that aggregate and analyze industry data. Many of these will be companies that sit in the middle of large information flows where data about products and services, buyers and suppliers, and consumer preferences and intent can be captured and analyzed. Examples are likely to include companies that interface with large numbers of consumers buying a wide range of products and services, companies enabling global supply chains, companies that process millions of transactions, and those that provide platforms for consumer digital experiences."
To start planning your big data software business, take a look at our
software company business planning package which you can download here.