The 13 Big Data Trends Industry Pros Are Watching

While big data has long been harnessed by leaders across virtually every industry to make key business decisions, today, the field is a proven and established subset of tech. With an ever-growing list of professions and use-case examples surrounding big data, trends have emerged in how that data is collected, organized and used. 

Obviously, developments in big data mean different things to different companies. We spoke to data pros at 11 tech companies to learn more about the big data trends they’re keeping up with and how they’re impacting business operations.

Social Determinants & Intuition

Trusting one’s gut is a practice that can lead to success in many professions. But data science is not typically seen as one of them, since trusting numbers is generally considered safer than trusting instincts. However, Data Science Manager David Thompson at Pareto Intelligence said that sentiment is changing. The leader at the healthtech company said intuition plays an increasingly large role in how industry pros work with big data.

 

What are the top big data trends you’re watching that are significantly impacting the industry?

Lately, and rightfully so, there has been a significant focus on social determinants of health — the conditions in which people are born, grow up, live, work and age that impact health status. Healthcare has been slow to arrive here, but trends are showing that nearly everyone in the industry is agreeing that there are benefits to considering everything about a person when assessing overall health and developing strategies for improving health. There is already an overwhelming volume of health-related data about each person, and SDOH adds even more layers to that.

"As we continue to nearly drown in messy and erroneous data, I believe intuition will begin to play an even larger role in analytics."

 

What under-the-radar big data trends are you watching that the industry isn’t talking about?

I have been watching the re-emergence of intuition in developing analytic strategies. While originally an enemy of data science, intuition seems to be acting more like an ally. Data scientists and analysts that have been working with big data for a while seem to have developed a recalibrated intuition that is rooted in deep experience with the data rather than general experience within the industry only. And as we continue to nearly drown in messy and erroneous data, I believe intuition will begin to play an even larger role in analytics.

 

How are these trends affecting the future of your company?

I believe these trends mean two things for Pareto Intelligence, and both are related to talent and human capital.

Weaving in SDOH data means we need more hands on deck. Additionally, we need to develop deep experience. If we are going to acknowledge the quiet undercurrent of intuition and leverage the benefits it can bring, we have to foster a culture that makes a long-term commitment to the company and the industry an easy and attractive decision for our data scientists and analysts. We are doing that every day through the benefits we offer, the community we cultivate, values we commit to, and so on.

 

Kubernetes & Simplifying Workflows

When you’re working with datasets so complex they can predict and analyze consumer behaviors and even emotions while they’re online, simplifying workflows can go a long way for increasing productivity. Machine Learning Engineer Drew Boshardy told us about some tech tools that were put in place at marketing tech company Networked Insights to do just that.

 

What are the top big data trends you’re watching that are significantly impacting the industry?

I’m closely following the ongoing development of Kubernetes and its related technologies such as Skaffold, Kubeflow and Helm. It may seem like overkill for some use cases, and it likely is for many basic applications, but for a moderately to very complex system, it really helps keep everything running and easy to deploy to. Similarly, orchestration technologies such as Airflow are going to be very big going forward to help simplify more complicated and essential workflows, making them easier to maintain and diagnose.

"Orchestration technologies such as Airflow are going to be very big going forward to help simplify more complicated and essential workflows."

 

What under-the-radar big data trends are you watching that the industry isn’t talking about? 

I think it’s being talked about a lot in the sphere of Google Cloud Platform-based companies, but Apache Beam is a real lifesaver for big data jobs. Both persistent and ephemeral self-managed Hadoop clusters still seem to be very common and, in many cases, unwieldy.

 

How are these trends affecting the future of your company?

We are currently using Google’s hosted Kubernetes solution Cloud Container Engine for our core infrastructure and other products. Our data infrastructure for our new DaaS product is running on Airflow and Apache Beam in the Google Cloud Dataflow to keep data up to date and reliable. We have the flexibility necessary to work with our many data partners, as well as the confidence that we can handle any irregularities or downtime easily and quickly thanks to Airflow. 

 

Digital Assistants

What good is having all the data in the world if you can’t make good use of it when you need it? GreenKey Technologies provides entities in the OTC capital markets field data based on their audio and text communications. Chief Data Scientist Tejas Shastry said front-office staff at client companies could use quick, on-demand access to their comms data without tapping a data scientist for help. So, Shastry sees digital assistants as the next big wave in big data.

 

What are the top big data trends you’re watching that are significantly impacting the industry?

We use natural language processing to help financial firms structure and search their millions of voice and chat conversations for insights. Over the last year, the biggest shift we’ve seen is a rise in the need to query big data in the front office. Instead of using data scientists and business intelligence analysts to scour their data, firms want chat-bot interfaces so their front-office workers can get insights from their data directly as they’re doing their jobs. We think this trend will continue, with digital assistant use growing across many industries to help surface specific insights from internal data lakes.

"Instead of using data scientists and business intelligence analysts to scour their data, firms want chat-bot interfaces."

 

What under-the-radar big data trends are you watching that the industry isn’t talking about?

The implementation of the data protection and privacy regulations under the General Data Protection Regulation in Europe have changed the way many of our clients think about leveraging their data. With tight regulations, we see firms increasingly requiring solutions that can be deployed on-premise or in a private cloud and co-located with privacy-sensitive data. This means technology companies will be increasingly offering solutions that don’t rely on a public cloud, and similarly, companies with GDPR-related data will need to embrace cloud-standard deployment models like Kubernetes in their own environments.

 

How are these trends affecting the future of your company?

Being cognizant of these trends has helped us deliver better products to our clients faster. We’ve adapted our NLP that initially focused on voice conversations to process chat as well, and doubled-down on our on-premise and private-cloud deployment models. As we move forward, we will continue to think of ways we can better expose big data directly to front-office workers without firms needing to build their own machine learning models or hire their own data scientists. We think both of these trends point to companies wanting out-of-the-box, on-premise solutions to make sense of their data.

 

Scaleable Open Source Data Warehouse Modeling

As big data tools like AI and machine learning grow more complex, Chewy’s VP of Software Engineering Karthikeyan Janakiraman wants processes to stay simple for his team. The leader at the pet supply e-commerce company said he hopes evolutions in big data tech will allow his engineers to work more efficiently with fewer hassles. 

 

What are the top big data trends you’re watching that are significantly impacting the industry?

We’re seeing impactful changes when it comes to data management and use cases emerging from relating disparate systems. Machine learning and AI are now the primary drivers of data consumption across users. We’re seeing the availability of tools that account for relational and dimensional modeling, transactional integrity and data governance in a way that’s easy to use. We’re also seeing companies in this space start to consider the user experience and help companies tap into the benefits of the cloud.

But the biggest shift has been moving to a scalable, open-source data warehouse model using public cloud. We appreciate when our tools offer solutions to the friction points of elasticity, scalability, cost efficiency and agility.

"We want to work with tools and software that make life easier, not more complicated."

 

What under-the-radar big data trends are you watching that the industry isn’t talking about?

The tensions we feel are probably similar to the ones felt throughout the industry. What we look for, and what we appreciate, is when our service providers adopt the same attention to customer service that we would take in our own business. We’ve seen some of our vendors and service providers take that approach and we hope to see more of it.

 

How are these trends affecting the future of your company?

We’re really excited about the ability of big data to help us keep delivering on our mission as a company. The best tools and services emerging in this space are the ones that create a more seamless work environment for our employees. We hope that the industry continues to evolve in a way that makes the work of our engineers less time-consuming. We want to work with tools and software that make life easier, not more complicated. We want to make sure that we’re organizing our teams in a way that allows every member to work on projects that directly impact the customer experience. In order to do that, we have to find simpler ways to explore the data and build on our findings faster.

 

Reinforcement Learning

Good decisions are at the heart of any business' success. And SessionM’s Head of Data Science and Engineering Amelio Vázquez-Reina said companies can make even better decisions in the future with techniques like reinforced learning. As the next wave of big data trends continues to unfold, the company plans to take advantage of evolving practices by applying them to their marketing services.

 

What are the top big data trends you’re watching that are significantly impacting the industry?

Delta data-lakes for extract, transform and load operations; automated machine learning; data governance and the advent of graph databases.

"Many important industry areas like robotics could benefit from AIs that can be efficiently trained in simulated environments."

 

What under-the-radar big data trends are you watching that the industry isn’t talking about?

One is off-line and off-policy reinforcement learning. Most companies are sitting on vast volumes of user data that they didn’t gather through active experimentation or by executing a well-defined policy. This data has enormous potential for optimization, but doing this well is hard.

High-dimensional Bayesian optimization is another trend. Most companies and systems need to optimize choices and decisions in the face of uncertainty. Doing this well and efficiently is still a challenging problem.

A third trend is transferring the learning of agents trained in efficient physics environments and engines to other spaces. Many important industry areas like robotics could benefit from AIs that can be efficiently trained in simulated environments, and later transferred to and calibrated on real-world scenarios.

 

How are these trends affecting the future of your company?

Reinforced learning and the Markov decision processes have the potential to power decisions most companies and software systems face. However, leveraging them adequately and safely in a complex system can be hard. At SessionM, we have identified many product opportunities for next-best-action marketing that would immensely benefit from thoughtful reinforced learning, and we are working hard to bring these products to market.

 

Data Privacy & Need for New Skillsets

With the introduction of new industry trends comes the need for skilled professionals well-versed in the latest technologies. Data Applications Engineer Olivia Ondeck at video advertising platform SpotX said companies are placing a greater emphasis on cybersecurity and data engineering skills as data privacy becomes top-of-mind across industries.

 

What are the top big data trends you’re watching that are significantly impacting the industry?

With the introduction of the General Data Protection Regulation in Europe and big data breaches, data privacy concerns are extremely widespread and awareness is only going to increase. Companies have to be more careful now and therefore, the demand for cybersecurity skills is higher than ever. There is also a rising demand for data engineering skills, not just data science and data analysis. As data becomes more of a currency, physical space and hardware requirements are increasing and the need to manage and optimize is getting more important.

"Our priorities are shifting to building and maintaining infrastructure that scales with our data growth."

 

What under-the-radar big data trends are you watching that the industry isn’t talking about?

When it comes to managing data and applications that use big data, better containerization is becoming more important. Container orchestration systems for automating application deployment — like Kubernetes and OpenShift — are rising in popularity because of their ability to run large-scale applications more seamlessly. 

Another trend we are noticing is that it’s not just millennials cutting the cord with cable, but older generations as well. People are ditching expensive cable services for both live TV subscriptions and regular streaming services. With those services come more opportunities for targeted video advertising. 

 

How are these trends affecting the future of your company?

Given the volume of our company’s data and how much that volume is growing, our priorities are shifting to building and maintaining infrastructure that scales with our data growth. An example of this is our transition to using OpenShift for managing our growing number of applications. The importance of having a large and strong data engineering team is higher than ever before because we are using more cutting edge technology in keeping our data secure and space-optimized.

 

Hybrid Cloud Systems

Sports enthusiasts have deep connections to their favorite sports, teams and players. Fanatics, a sports merchandise distributor, knows this well. Senior Engineering Manager Sunlight Yang said the company is digging deeper into hybrid cloud systems to evolve their data management in hopes of getting fans even closer to the games they love.

 

What are the top big data trends you’re watching that are significantly impacting the industry?

Over the past few decades, the physical world has become blended with the digital world. Thanks to the emergence of IoT, our lives have become more connected than ever and this brings us a wealth of data that we have never seen before. However, the traditional way of computing has become insufficient when dealing with data of such magnitude. That’s where machine learning and AI come into play. They are changing how we look at the world and how we do business.

"Hybrid cloud strategies have existed for years and we continue looking for advances in this space."

 

What under-the-radar big data trends are you watching that the industry isn’t talking about?

When building a large system that would cross the boundaries between cloud and on-premise systems, efficiency and data security requirements will always be concerns. Hybrid cloud strategies have existed for years and we continue looking for advances in this space.

 

How are these trends affecting the future of your company?

Fanatics is a company for fans, whether they be casual sports lovers or avid esports enthusiasts. The more we know about the fans, the better we can serve them with the products they want. Being intimate with those big data trends has helped us build a robust platform that allows us to offer them top-notch products and services and connect with them in both the real and virtual world.

 

Changing Access to User Data

Adswerve's Director of Technical Services Jacob Shafer said there’s a shift happening in the way companies access and utilize user data. He explains how privacy-based regulations and the evolving nature of tech are limiting the breadth of user information companies have access to, making it more challenging for companies to get their brands in front of users. Shafer said Adswerve is tapping into its own data to help agencies, marketers and analysts navigate the changing landscape.

 

What are the top big data trends you’re watching that are significantly impacting the industry?

Simply put, privacy. Technology, its users and legislation are all increasingly focused on a user’s right to control their data. In opposition, the industry has increased our reliance on free access to users’ data. This conflict is resulting in industry-wide disruption. Companies will no longer be able to make decisions based on broad sets of granular user data, but will instead need to rely on aggregated contextual data about its users.

"Technological advancement continues at such a rapid pace that lawmakers have had a hard time keeping up."

 

What under-the-radar big data trends are you watching that the industry isn’t talking about?

Two things come to mind. The first is analysis instead of machine learning. Many companies are excited about the opportunity to utilize greater computing power to discover new insights about their users. But these companies can typically garner a great number of powerful insights by analyzing the data they already have. 

There’s also a lack of informed legislation. Technological advancement continues at such a rapid pace that lawmakers have had a hard time keeping up. The legislation is often difficult or impossible to enforce and shows a misunderstanding of the technology itself.

 

How are these trends affecting the future of your company?

These changes in the industry require companies to craft a more advanced strategy to make sure their target audiences know about them. Adswerve is well-positioned to partner with its clients as they craft and implement a more advanced strategy. These industry trends — like privacy and new legislation — will continue to challenge us and marketers alike, but we’re excited for the opportunity to leverage our media and data insights to help our clients successfully navigate through it.

 

data democratization and cloud data analytics

Matillion works for large companies such as Amazon and Google BigQuery to transform data via cloud-based software into a usable state. Vice President of Alliances Andreas Schurch said he predicts that more companies will primarily work with cloud computing.

 

What are the top big data trends you're watching that are significantly impacting the industry?

The big data trends that we are watching are data democratization and cloud data analytics. As data needs to grow and shift, IT and business users are working through governance and control within their organizations so that employees can easily access the data they need. Once they have access to that data, they’ll need to transform it for analytics. Increasingly, we’re seeing enterprises use the power of cloud computing to transform data into a usable state.

"Matillion sits at the intersection of two big trends: cloud and data analytics."

 

What under-the-radar big data trends are you watching that the industry isn't talking about?

With changes in regulation and compliance, organizations are trying to figure out how to keep consistent business logic and rules for all of their data. I believe there will be a shift to a distributed data integration system where companies will want to apply regulatory, privacy, and/or financial rules to any data on any platform for complete management of the data journey. 

 

How are these trends affecting the future of your company?

Matillion sits at the intersection of two big trends: cloud and data analytics. We are committed to enhancing our extract, transform, load product line and data transformation software to support our customers as they navigate new data trends. Our focus is to continue delivering features required by complex IT environments in areas like scalability, development lifecycle integration and re-use. We will continue to provide our customers with choice through the cloud adoption journey and to simplify the procurement process for new users and use cases.

 

Evolving Data Pipelines

Quartet Health helps doctors and people with mental health conditions connect through technology. Vice President of Data Engineering Mamta Prakh explained how responsibly managing large data systems allows for patients to get the services they need quickly.

 

What are the top big data trends you're watching that are significantly impacting the industry?

Infrastructure for processing large-scale data has evolved in recent years. At Quartet, we rely on cloud solutions, distributed computing and open source tools to build robust, scalable, custom pipelines to efficiently and affordably manage large volumes of health data. 

Widespread adoption of this infrastructure has helped data science move from ad hoc data analysis to reproducible long-term research and deployment of statistical models into production at scale. Collaboration between data scientists, engineers, designers and product managers is on the rise, facilitating movement from traditional software systems to learning systems that adapt automatically to change and new data.

Modern analytics platforms such as Looker are seeing increasing adoption at organizations like ours. These platforms democratize data analytics and empower users to develop a deep and broad view of the business quickly.

 

What under-the-radar big data trends are you watching that the industry isn't talking about?

Data engineering is quickly evolving as a specialized field focused on preparing large-scale datasets for analytics. Data engineers are architecting distributed systems and data stores, creating reliable pipelines, and combining data sources for advanced analytics. Our data scientists and engineers are enthusiastic about applying anomaly detection techniques to automate some quality assurance and processing tasks.

We’re exploring frameworks to improve explainability of machine learning models. As we aspire to augment existing physician workflows for mental healthcare, these tools are key to making interpretable recommendations. We’re closely reviewing novel academic AI research areas that are yet to reach production at scale. 

"With reliable data pipelines, we can iteratively study patient outcomes and improve care options."

 

How are these trends affecting the future of your company?

At Quartet, we’re on a mission to improve the lives of people with mental health conditions through technology and services. We’re developing a scalable approach to make it easier for patients to get mental healthcare at the right time and in the right setting, based on their needs and preferences. 

Large and complex data is needed to understand the interplay between physical and mental health as well as connect patient needs to high-quality personalized care. Managing this data reliably with built-in subject matter expertise enables talented data scientists from many other fields to work toward improving the identification and servicing of mental health needs.

Building real-time, feedback-driven machine learning with explainable insights into our products will incrementally drive user trust and engagement, which in turn will reduce stigma and help more people get the mental health care they need. With reliable data pipelines, we can iteratively study patient outcomes and improve care options.

 

Applying AI & Machine Learning to Real-World Healthcare Data

Crossix analyzes healthcare data to create innovative solutions that advance healthcare marketing. Director of Advanced Analytics Adam Dubrow said applying trends like machine learning and AI to real-world patient data can lead to improvements in care and better outcomes for patients.

 

What are the top big data trends you're watching that are significantly impacting the industry? 

Some of the most important trends for the healthcare industry include the rapid growth of data available for new applications in improving patient health outcomes, including real-world data from the healthcare system as well as consumer and patient-generated data. From there, we receive transformative insights and solutions made possible by connecting these data sets and applying machine learning and AI.  There are many opportunities for using real-world data for regulatory and clinical applications such as supporting FDA applications. 

 

What under-the-radar big data trends are you watching that the industry isn't talking about? 

The trends I pay attention to are the convergence of the data, research and analytical efforts of commercial and clinical teams in the healthcare industry. The shift toward value-based care, among other factors, means that clinical and commercial teams are more frequently asking the same questions. For example, a model that predicts with high accuracy when patients will not take a treatment as prescribed by their doctors has use for both types of teams. 

"Crossix technology was engineered to be privacy safe."

 

How are these trends affecting the future of your company? 

Crossix, a Veeva company, is leading the way with innovations that will shape the future of the industry. Crossix technology was engineered to be privacy safe, and we’re uniquely equipped to connect massive and disparate data sets in a safe way to power solutions and insights that improve patient outcomes. Our technology and expertise across healthcare and data science have allowed us to deliver the best solutions and insights to challenging problems and we’re focused on continuing to innovate. 

The trends above highlight some of the dynamics that make the healthcare industry such an exciting and rewarding area for innovation. Healthcare is experiencing more growth in data than other fields (one study estimated that genomics alone is more challenging from a data perspective than astronomy). The importance of privacy is absolute and at the forefront, and the potential impact data innovations could have is monumental in terms of improving the lives and experiences of patients.