On our recent 2018 Predictions webinar, we shared insights on what we expect to see in the way of business transformation, technology advances, and market shifts in the coming year. Speakers included Rob Carlson, the CEO and Co-Founder of Unifi Software; Srinath Reddy B, Senior Director, IT Strategy and Big Data for AON; Megan Asarrane, Senior Product Marketing Manager for Microsoft; and Chad Packard, Partner at Pelion Ventures.

Here are our Big Data Predictions for 2018

Self-Service Data Comes of Age in 2018.


Historically, IT served the business with tools like Cognos, Business Objects, and Informatica. Then BI Visualization tools came along and showed there was a way for business users to do self-service BI. That only worked if all your data was in a single table, otherwise, you were back at IT doing ETL. Then along came self-service prep that alleviated that issue but then the business user could not find the data they needed, so they required a catalog and discovery tool. IT was concerned about the developing “Wild West” data environment and needed to deploy governance, security, and data quality tools. Recently, collaboration and community tools have emerged as a critical element of shared learning and insight to reduce repetitive tasks and shorten learning curves.

Self-service data coming of age is the realization that businesses will understand that they need all these pillars—discovery, catalog, preparation, workflow, collaboration and governance and not just individual pieces in a silo. We see countless examples of enterprises that have had to rethink the way they deliver and manage their data exploration needs and we’ll see much more of this in 2018.

Organizations will no longer need to deploy and maintain large Hadoop environments for their data and analytic needs.


Over the past four or five years, organizations were investing very heavily in setting up huge Hadoop clusters, which were both cost and maintenance intensive. Hadoop environments come with set up, licensing costs and maintenance costs along with the need to make investments in skills to support these environments. With so many open source components, companies had to hire people familiar with each one.

Now that a new breed of self-serve tools has emerged for data prep, data catalog and collaboration it’s much easier to process data and build analytical models and still have Hadoop processing power on the back end. We will start to do away with dedicated Hadoop clusters because all these tools can connect to Spark and other engines to provide the same processing power and are much easier to use. Companies won’t need the same IT skills to do this job as business users will be much more empowered.

Understanding data through BI and data visualization tools will be a major area of investment in 2018.


There are four factors that are aligning for this to happen. Over the past few years, we’ve been talking about the explosive growth of data and the collection of that big data—going from terabytes to petabytes of data per day. Now we have this vast amount of big data to mine and get insights from. At the same time that this trend is occurring, we have major advances in computing power. We can now comb through all of those petabytes of data, in rapid fashion to glean insights. Additionally, user’s time is extremely limited. They need to make decisions quickly and have trust in those decisions. Lastly, we have a shift into a data-driven culture. Without the BI tools and visualization layer, it’s extremely hard for a user to have confidence in those decisions. That’s why the continued investment in these tools will grow exponentially.

Governance and security will become critical to enterprise self-service data


Self-Service Data will come of age in 2018 at the same time as more rigid and expansive privacy and security regulation requirements (such as GDPR in the EU). Organizations will have a challenge on their hands in maintaining compliance and security of their data while still allowing business users to have access to the data they need to do their jobs efficiently. Enterprises will have to think differently about how to approach Governance and Security and different tools to be certain the organization’s data is available, yet protected.

Investments will heat up for startups that focus on using AI to drive insights or reduce the workforce.


Investments have and will continue to be strong for companies that take an AI or ML-first approach to solving business challenges. The promise of nearly infinite compute and storage from cloud providers, new data tools like Unifi’s self-service platform, new AI frameworks and trained data scientists will allow AI to help businesses deliver more revenue, reduce costs, and produce better products, with the same number of people not necessarily with reducing headcount.

And, although there is a lot of worry about the reduction of the workforce because of automation using AI, I believe most of the dollars from customers and real value will be around learning the machine-to-machine transactions. This includes security detection and automation, recommendation engines, credit card fraud and employee tech support systems.

AI companies that build a value prop purely around the reduction of headcount will continue to see a lot of funding but the adoption and traction of a customer and enterprise side will happen from the machine-to-machine transactions. I’d add, many of the legacy vendors have labeled themselves with these buzzwords [AI and ML] but offer very little around the promise that these models actually offer.

AI will be applied to every state of enterprise data analytics.


When we start thinking about AI and how that applies to every stage of the enterprise with regard to analytics and business intelligence and looking into information, the task of big data cleansing, enriching, normalization, transformation, filtering, and formatting are not for the faint of heart. These are tough things to do. Business users have traditionally not been involved in these steps. These are tasks that have been fulfilled by skilled programmers and IT who are very good at interacting with information—until now. We are seeing this trend is changing, especially with the emergence of AI. Tools that help obscure the complexity of these tasks can only go so far.

What is needed is AI that can be trained to predict and recommend how data should be cleaned and combined in order to achieve the desired outcomes and identify patterns that are hard to see manually. Through AI, these complex environments will begin to flow easier. We’ll find answers to complex data questions without having to be a mathematician or a PhD. AI will also extend to things like natural search or natural language description asking insights of your data in plain text and being able to find information faster with relevant context, or even talking to an AI assistant such as Alexa or Siri, which is an ideal input method.

Data scientists and analysts joining Line Of Business (LOB) teams as organizations rely more heavily on the value of unlocking big data.


Everyone inside an organization can use self-service tools to enable data marketplaces or collaborate across different teams to do much more than what they have traditionally been able to do. Previously, tasks were relegated to IT. We had dedicated BI teams to handle large data processes such as ETL jobs where they had to use a combination of products for these tasks, which was all very labor intensive. They struggled with capability and performance. Now, with self-service tools, they can solve the majority of problems which IT teams cannot solve for them today. These tools provide an extensive capability to search big data, access data and collaborate across data. With the help of new age self-service tools data analysts, business analysts and LOB users don’t have to depend on IT teams anymore.

IT departments will begin cross-charging business departments for the cost of cloud consumption.


Elegantly scalable environments like AWS EMR or Azure HD Insights deliver unprecedented options to a business on concurrency and throughput scalability. But this flexibility runs the risk of the old cell phone sticker-shock before all-you-can-eat mobile plans were introduced. Providing the line of business with the flexibility to select the speed in which they need the results of a data transformation will be essential. This will be the Amazon Prime and delivery options for big data.

If you have questions about our Big Data Predictions for 2018, please let us know. We’re happy to tell you more about what’s behind our predictions and expectations for success in 2018.