What Is Dark Data?

Dark data is information collected by organizations that is not actively used, managed or analyzed. The relentless pace of data generation today is outstripping humans’ ability to harness its value, leaving terabytes of unused data buried in the recesses of company infrastructure. As a result, more than 50 percent of the data collected by enterprises qualifies as “dark.” 

What Is Dark Data?

Dark data is all the information collected by organizations that is not actively used or examined. Often likened to the physics concept of dark matter for its invisibility and ubiquity, dark data remains hidden or overlooked, constantly accumulating in the background. 

Common examples include: 

• Web server logs: Logs tracking website activity that are archived but not analyzed for trends.
• IoT data streams: Data from industrial sensors that is collected but rarely reviewed for optimization opportunities.
• Historical records: Legacy data in outdated formats, often inaccessible to modern analytical tools.
• Unprocessed feedback: Customer feedback left in raw text or audio form without categorization or analysis.

The existence of dark data is often the result of a lack of resources or infrastructure. 

“Many data [challenges] for organizations arise from difficulty managing data across many different parts of the org,” Parker Ziegler, a computer science PhD candidate at the University of California, Berkeley, previously told Built In. “With multiple data sources contributing small pieces of a full data picture, you often run into issues of data duplication or data drift over time.”

This hastily collected data differs from structured data sets, which are organized and readily analyzable. Meanwhile, dark data often exists in siloed, unstructured or inaccessible formats — acting as both an untapped opportunity and a potential liability. 

Types of Dark Data

Dark data comes in various formats across technological environments, including:

• Operational data: Metrics from IT systems, including server logs and monitoring tools.
• Customer interactions: User feedback, support tickets and behavioral analytics.
• Business communications: Meeting recordings, email archives and internal reports.
• Media: Unlabeled images, videos and documents that lack metadata or organizational context.
• Legacy information: Outdated files and databases stored without clear documentation.

Related ReadingThe Dangers of Too Much Data

How Is Dark Data Created?

There are several reasons data within an organization goes unused:

• Lack of awareness: Data generated during routine operations often goes unused because organizations are unaware of its existence or fail to recognize its value.
• Data silos: When departments independently store and manage data, it can lead to fragmentation, making valuable data sets inaccessible to other teams.
• Lack of data governance: Without effective data governance, organizations struggle to organize, track and manage data, leading to disorganization and unused data sets.
• Legacy systems: Data stored in outdated systems may become inaccessible if it cannot integrate with modern analytics tools.
• Incomplete integration: Gaps or inefficiencies in data integration processes can result in inaccessible or inconsistently linked datasets.
• Shifting priorities: As business goals change, previously valuable datasets may lose relevance and become neglected.
• Resource constraints and literacy: Limited resources often prioritize data storage over analysis, while a lack of data literacy can hinder one’s ability to identify and utilize valuable data.
• Data quality issues: Inaccurate or incomplete data is often dismissed as unreliable, rendering it unusable.
• Regulatory compliance: Organizations may store sensitive data longer than necessary due to poor tracking, despite regulatory requirements for destruction.
• Redundant, obsolete and trivial (ROT) data: Excessive copies, outdated information and irrelevant data clutter systems, making it harder to find and utilize valuable information.

Dark Data Risks

Dark data comes with various costs and risks, including:

Wasting Storage Space

Even unused data demands physical or digital storage infrastructure, such as servers, data centers, cloud solutions and backup systems. As dark data accumulates, it often consumes valuable storage resources that could be better utilized by active data. To keep up, organizations have to invest in more space, driving up operational costs.

Legal Liabilities

Over the years, governments worldwide have implemented stringent privacy laws that extend to all data, including unused information stored in analytics repositories. Even if data is unused or forgotten about, it still has to comply with these regulations, posing serious legal (and potentially financial) risks.

Operational Inefficiencies

Having to sift through vast amounts of irrelevant information hinders the data retrieval and analysis process, causing employees to spend excessive amounts of time searching for relevant data. This inefficiency reduces productivity and drives up labor costs.

Security Risks

The presence of dark data leaves organizations more vulnerable to data breaches, data loss and other cybersecurity risks. Without proper oversight, sensitive information hidden within dark data could be inadvertently exposed or mishandled, potentially leading to financial penalties and reputational harm.

Opportunity Costs

Companies often miss valuable opportunities by neglecting unused data. While eliminating this data can mitigate risks and costs, analyzing available data first is essential to uncover potential value.

Related ReadingWhat Is a Data Governance Framework?

How to Locate Dark Data

Locating dark data requires a systematic approach: 

• Take inventory: Map all existing data sources across systems and departments.
• Utilize metadata: Leverage metadata to organize and classify datasets.
• Automate: Implement AI-based discovery tools to identify and categorize dark data.
• Collaborate: Engage stakeholders from different teams to uncover overlooked or underutilized datasets.

How to Manage Dark Data

Managing dark data effectively involves integrating robust governance practices. Key strategies include:

• Data governance frameworks: Define clear policies for data retention, categorization and security.
• Artificial intelligence: Use AI to quickly analyze, process and derive insights from unstructured data.
• Archival solutions: Implement scalable storage solutions to archive data that has potential long-term value.
• Interdepartmental coordination: Establish channels for sharing data insights across teams to maximize efficiency.
• Periodic reviews: Conducting regular audits can help identify and eliminate redundant, obsolete or trivial data.

Frequently Asked Questions