Seven years after the New York Times heralded the arrival of "big data," what was once little more than a buzzy concept significantly impacts how we live and work. At the end of 2018, in fact, more than 90 percent of businesses planned to harness big data's growing power even as privacy advocates decry its potential pitfalls.
As analyst and author Doug Laney puts it, big data is defined by three Vs: volume, velocity and variety. There's lots of it flowing in at great speeds from numerous sources. And its impact is immense, regardless of industry.
Big data has made once-holistic concepts, such as “what consumers want,” more measurable. It has facilitated inductive reasoning, a controversial data-first inversion of the scientific method. At many companies, it has ushered in a “culture of analytics” in which even non-tech employees both input data and have access to data-driven insights.
Real World Big Data Examples
- Discovering consumer shopping habits
- Personalized marketing
- Fuel optimization tools for the transportation industry
- Monitoring health conditions through data from wearables
- Live road mapping for autonomous vehicles
- Streamlined media streaming
- Predictive inventory ordering
- Personalized health plans for cancer patients
- Real-time data monitoring and cybersecurity protocols
Perhaps most significantly, nearly every industry uses big data for future planning by predicting how people will live and what they’ll buy. Still, it's not a crystal ball. Certain types of data sets, such as those that span decades or centuries (a.k.a. “long data”), have far more predictive power than a similar volume of data from only one year. And when it comes to foreseeing sudden cultural shifts, like the rise of smartphones, even the most sterling data has limitations.
At its best, though, big data grounds and enhances human intuition.
We’ve rounded up 17 examples of how big data is shaping industries from marketing to government.
Marketers have targeted ads since well before the internet—they just did it with minimal data, guessing at what consumers might like based on their TV and radio consumption, their responses to mail-in surveys and insights from unfocused one-on-one "depth" interviews.
As the internet and big data have evolved, so has marketing. Today it's possible to collect or buy massive troves of data that indicates what large numbers of consumers search for, click on and "like." There's also a huge influx of performance data that measures the effectiveness of marketing campaigns via impressions, click-through rates and other modern metrics that are far more nuanced than sales figures alone.
Here are a few examples of big data’s effects on modern marketing.
Amazon: Ads Based on What You Buy
How it’s using big data: Like Facebook and Google, adtech’s “duopoly,” Amazon got sucked into the advertising business by the sheer amount of consumer data at its disposal. Since its founding in 1994, the company has collected reams of info on what millions of people buy, where those purchases are delivered and which credit cards they use. In recent years, Amazon has begun offering more and more companies—including marketing companies—access to its self-service ad portal, where they can buy ad campaigns and target them to ultra-specific demographics, including past purchasers.
3Q Digital: Multi-Touch Attribution
Location: San Mateo, Calif.
How it’s using big data: At this independent digital marketing firm, big data underpins strategies that blend search engine, social, mobile and video marketing. The in-house Decision Sciences team perfects the mix of marketing channels by studying data on transactions, consumer behavior and more, using multi-touch attribution. This big data-informed technique allows analysts to distinguish between effective and ineffective ad impressions on a micro level.
Marketing Evolution: Hyper-Personalized Marketing
How it’s using big data: This marketing agency pulls data from hundreds of online and offline sources to create detailed consumer profiles that encompass beliefs, location and purchasing habits as well as environmental data like current local weather conditions. Analysts then use a software stack dubbed the “ROI Brain” to craft targeted campaigns where every element, from the messaging itself to the channel it arrives through, reflects individual users' preferences.
Maps to apps. That's the nutshell version of how navigation has been transformed by technology, with the vast majority of smartphone users relying on their devices for directions. And those directions are courtesy of big data — relevant information (on traffic patterns, for example) gleaned from government agencies, satellite images and other sources.
But big data doesn't just affect how people move, it affects how everything moves — including packages, planes and cars. Packages have tracking numbers (data!). Planes analyze data to (among many other things) increase fuel efficiency and predict maintenance issues. And cars, via onboard sensors and IoT connectivity, collect and transmit so much data that the autonomous driving revolution might be closer than we think.
Here are some examples of big data in motion.
General Electric: More Efficient, Eco-Friendly Airplanes
Location: Fairfield, Conn.
How it’s using big data: GE’s Flight Efficiency Services, recently adopted by Southwest Airlines and used by airlines worldwide, can optimize fuel use, safety and more by analyzing the massive volumes of data airplanes generate. How massive? One transatlantic flight generates an average of 1,000 gigabytes. GE’s scalable aviation analytics takes it all in, crunching numbers on fuel efficiency, weather conditions, and passenger and cargo weights.
HERE Technologies: Live Maps for Self-Driving Cars
How it’s using big data: The experts at HERE Technologies leverage location data in several ways, most notably in the HD Live Map, which feeds self-driving cars the layered, location-specific data they need. The map pinpoints lane boundaries and sense a car's surroundings. Thanks to data from intelligent sensors, the map can see around corners in a way the human eye can't. And a perpetual stream of intel from fleets of roaming vehicles helps the map warn drivers about lane closures miles away.
FourKites: Precise Delivery ETAs
How it’s using big data: FourKites’ platform uses GPS and a host of other location data sources to track packages in real time, whether they’re crossing oceans or traveling by rail. A predictive algorithm then factors in data on traffic, weather and other external factors to calculate package ETAs, so FourKites clients can give customers advance warning about delays and early deliveries — while also avoiding fees.
The U.S. government has a patchy relationship with technology. All of its agencies collect massive amounts of data. Especially at the local level, however, few let it shape priorities and workflows. Anthony Townsend, author of Smart Cities, told Government Technology that “most government agencies still operate on rote bureaucratic procedures.”
The agencies that actually do leverage the data they cull can find themselves in murky territory both morally and legally. For instance, when police departments that employ big data-powered “predictive policing” to foretell the locations and perpetrators of crime, they've been accused of privacy violations and relying on racially biased algorithms. In Chicago, where data-driven crime forecasting has coincided with a drop in gun violence, questions remain about how the city can most ethically leverage the information it collects.
Here are two examples of big data companies collaborating with governments.
Palantir Technologies: Predicting Roadside Bomb Coordinates
Location: Palo Alto, Calif.
How it’s using big data: Palantir helps American military agencies predict roadside bomb locations, foresee insurgencies and even (purportedly) catch the terrorist Osama bin Laden—all with insights drawn from big data. Using Palantir’s two platforms, government enterprises can store all of their data, from emails to spreadsheets, in one searchable place. Funded by the CIA and PayPal billionaire Peter Thiel, the company is expanding into the private sector while continuing to work with the defense industry and police departments.
RapidSOS: Data-Enriched 911 Calls
How it’s using big data: Rapid SOS Clearinghouse funnels emergency-relevant data to first responders out on 911 calls. Thanks to partnerships with Apple, Android providers and apps like Uber, the company can pull relevant data from patients’ phones and wearables in crisis situations. Free to public safety offices, Clearinghouse integrates into pre-existing call-taking and dispatch channels so the data—including GPS location data and real-time sensor data—reaches EMTs more reliably and securely.
There are several possible reasons that's the case, one being that many analytics tools analyze only small randomized samples of massive data sets. Doing so speeds up the discovery process but leaves a lot of data untapped. With other companies, it's just a matter of determining the value of voluminous data — what it can actually do for them.
Some companies, though, are ahead of the curve. These one actively and meaningfully integrate big data and big business.
Netflix: Data-Informed TV Streaming
Location: Los Gatos, Calif.
How it’s using big data: The premise of Netflix’s first original TV show—the David Fincher-directed political thriller House of Cards—had its roots in big data. Netflix invested $100 million in the first two seasons of the show, which premiered in 2013, because consumers who watched the British series House of Cards also watched movies directed by David Fincher and starring Kevin Spacey. Executives correctly predicted that a series combining all three would be a hit.
Now, six years later, big data impacts not only which series Netflix invests in, but how those series are presented to subscribers. Viewing histories, including the points at which users hit pause in any given show, reportedly influence everything from the thumbnails that appear on their homepages to the contents of the “Popular on Netflix” section.
Skupos: A Convenience for Convenience Stores
Location: San Francisco
How it’s using big data: The PC-based Skupos platform pulls transactions data from 7,000 convenience stores nationwide. Over the course of a year, that adds up to billions of transactions that can be dissected using the platform's business analytics tools. Store owners can use the insights to determine location-by-location bestsellers and set up predictive ordering. Distributors, meanwhile, can forecast demand, and brands can analyze a constant influx of product sales data.
Salesforce: Interdepartmental Analytics
Location: San Francisco
How it’s using big data: Companies often scatter their data across various platforms, but Salesforce is all about cohesion. Their customer relationship management (CRM) platform integrates data from various facets of a business, like marketing, sales, and services, into a comprehensive, single-screen overview. The platform’s Einstein analytics provide automatic AI-informed insights and predictions on metrics like sales and customer churn. Users can also connect Salesforce with outside data management tools rather than toggling between multiple windows.
Americans spend trillions of dollars a year on their healthcare, but the services often feel impersonal. Patients and doctors rarely have robust, long-term relationships. In fact, many patients enjoy a better connection (so to speak) with their devices.
That reality is slowly but surely changing the healthcare landscape. Diagnostics are migrating from clinics to wearables like the Apple Watch's built-in heart rate monitor and Alphabet's in-development skin temperature monitor. Unlike doctors, such devices can collect time biometric data over the long term instead of just during appointments.
But while personal healthcare devices have carved out an expanding niche, they won't replace human physicians anytime soon. Healthcare providers, too, are leveraging and acting on medical data in innovative ways.
Here are some examples of how big data facilitates new types of healthcare.
Tempus: Personalized Oncology
How it’s using big data: Tempus’ tablet-based tool has made file cabinets of medical records portable and accessible in real time. Designed to inform physicians’ decisions during appointments, Tempus trawls huge digital archives of clinical notes, genomic data, radiology scans and more to turn out data-driven treatment recommendations. These recommendations are personalized, too, though—based on data from past cases in which patients had similar demographic traits, genetic profiles and cancer types.
Propellor Health: Smart Asthma Inhalers
Location: Madison, Wisc.
How it’s using big data: Propeller Health reimagined the inhaler as an Internet of Things gadget. Widely used for treatment of asthma and other chronic obstructive pulmonary diseases (or COPDs), the sensor-equipped inhalers export data to a smartphone app that tracks inhaler use, as well as environmental factors like humidity and air quality. Over time, in-app analytics can help identify possible flare-up triggers and produce reports that patients can share with their doctors.
Innoplexus: Accelerating Pharmaceutical Development
Location: Eschborn, Hessen, Germany
How it’s using big data: Innoplexus's Ontosight life sciences data library, featuring search tools rooted in AI and blockchain technology, was compiled to help pharmaceutical researchers sift more quickly through relevant data and streamline drug development. A truly massive repository, it includes everything from unpublished PhD dissertations to gene profiles to a whopping 26 million pharmaceutical patents.
It's no surprise that the rise of big data has coincided with a rise in cyber attacks. The more data we store, the more hackers can steal by exploiting various vulnerabilities. Breaches, in fact, have become commonplace. In 2018 alone, according to one report, 446 million consumer records were exposed to security threats —more than double the previous year's tally. Cyberattacks are so ubiquitous that some companies protect their troves of digital info by hiring their own hackers to poke around for security gaps so proactive measures can be taken. Paradoxically, though, big data analysis also helps companies identify breaches by singling out anomalous activity that often signifies security issues.
Here are some examples of big data protecting big data.
Awake Security: Brain-Like Security Savvy
Location: Santa Clara, Calif.
How it’s using big data: Awake’s security system works a bit like the human brain. Sensors scan data where it’s stored, whether in the cloud or embedded in an IoT device. Much as our nerves relay information back to our brain, Awake's sensors port key findings back to the Awake Hub, a centralized deep-learning center that can detect threats and parse the intent behind unusual data.
Like a brain, the Hub also needs occasional outside input to determine the best approach. In certain cases, it's used in collaboration with a network of human cybersecurity experts who are up to date on the latest cyberattack techniques and industry-specific protocols.
Splunk: Real-Time Data Monitoring
Location: San Francisco
How it’s using big data: This company’s Security Operations Suite relies on big data to identify and respond to cybersecurity threats and fraud. Systemwide data flows through Splunk’s analytics tools in real time, allowing it to pinpoint anomalies with machine learning algorithms. Splunk’s data-driven insights also help it prioritize concurrent breaches, map out multi-part attacks and identify potential root causes of security issues.
AccessData: Speedy Emergency Searches
Location: Lindon, Utah
How it’s using big data: The company’s Forensic Toolkit, or FTK, stores enterprise-scale data in a straightforward database structure, processing and indexing it up front. In an emergency situation, that allows for quicker searches that are further accelerated through the use of distributed processing across an array of computers. FTK makes full use of its hardware resources, focusing all of its available processing power on extracting evidence that clients can leverage in civil and criminal cases.
Images via Shutterstock, social media and screenshots of company web pages.