In the last decade, security tools and programs across the globe have made immense strides in their ability to detect, prevent, and respond to advanced security threats. With these strides, though, has come another challenge: avoiding the fallouts of improved detection of threats.
A poll performed in 2020 asked 427 security professionals about the volume of alerts at their companies (ones with > 1,000 employees). In the survey, 70 percent of those individuals said their alerts have more than doubled in the last five years, and 93 percent of respondents claimed they could not address all alerts in the same day. The combination of these situations ultimately leads to the inevitable: alert fatigue.
What Is Alert Fatigue?
What Is Alert Fatigue?
Those who work in cybersecurity are likely familiar with the term since it’s an issue many organizations have dealt with in the last decade. Alert fatigue is the result of receiving an overwhelming amount of security alerts. In other words, it occurs when security staff becomes desensitized to the various alerts they receive in a given day.
As the statistics mentioned above suggest, most organizations are inundated with alerts on a daily basis as a result of monitoring various aspects of a company’s technological infrastructure. These warnings can come from device monitoring, email filtering, internet security, network firewalls, and more. And this list continues to grow as new threats emerge.
Why Is Alert Fatigue a Problem?
Although security programs across the globe have matured over the years, advanced detection capabilities are part of the reason daily alerts have doubled and sometimes tripled. Not too long ago, organizations were able to get away with having signature-based detection tools in place along with basic email and internet filtering capabilities to block spam and malicious websites.
In 2004, the global security market was valued at $3.5 billion. In 2021, that number ballooned to $262.4 billion, which emphasizes the level of growth organizations have had to commit to in order to protect their infrastructures. This growth results in increased levels of detection across the technology stack, leading to more alerts. Alerts are good, though, right?
On paper, they’re great. Alerts notify us of potential security issues, enabling security personnel to respond and hopefully prevent the success of whatever threat is on the network. The ramifications of too many alerts can be detrimental to an organization’s security program, however.
An unmanageable number of alerts leaves teams in a “boy who cried wolf” scenario, where analysts receive so many alerts, likely mostly low- or medium-severity, that they become more tolerant of them. A team can receive hundreds of notifications daily, many of which may be unactionable or false positives, leading them to be less likely to respond correctly and with proper urgency when a truly positive, critical alert happens.
Not only does this mean teams miss critical alerts, a SOC (Security Operations Center) in this mindset also has slower response times. Alert fatigue definitely lowers Mean Time to Respond, which is one of the primary metrics companies use to measure the success of a security program This diminished capacity negatively impacts the effectiveness of the security program.
Lastly, alert fatigue has an adverse psychological impact on staff. Organizations dealing with this problem not only receive alerts during the work day but also overnight. Even though a security staff expects to work off hours sometimes to respond to potential incidents, organizations with poorer alert maturity are likely dealing with frequent false positives overnight. This means analysts can spend hours at a time investigating what turns out to be nothing but a scheduled job or system update. Situations like this, especially when they become a frequent occurrence, hurt staff morale, ultimately leading to increased turnover rates.
What Can We Do to Fix Alert Fatigue?
As with most phenomena like this, alert fatigue has become the latest marketing term security vendors are using to reel potential customers in. The reality is that no vendor can solve this problem, however. Some security tools can certainly help, but organizations must invest significant effort to optimize their security workflows. This investment, in turn, reduces the alerts a team receives on a daily basis. Doing this comes down to five things:
5 Ways to Combat Alert Fatigue in Your Security Program
- Ensure all alerts are actionable.
- Set alert priorities correctly.
- Use thresholds and ensure they’re set appropriately.
- Automate anything that can be automated.
- Continuously review and improve existing alerts.
1. Ensure All Alerts Are Actionable
The first step to reducing alert fatigue is ensuring that all alerts are actionable. Nothing is worse than finding out your SOC personnel are spending parts of their days closing out non-actionable alerts. Although this may seem like a minor problem, it can take upwards of 30 minutes for an analyst to review and investigate an alert, only to determine nothing needs to be done to resolve it. Sometimes, they may even need to contact other team members or IT staff, which results in additional wasted time.
This problem has an easy solution: Don’t put any alerts into production without first developing a corresponding procedure and training staff on it. Too often, alerts from a new capability or tool are enabled without testing and developing a process around it. Engineers and administrators deploying the new capability must identify whether the alert is necessary and will provide value, then work with respective members of the team to develop an appropriate response procedure.
2. Set Alert Priorities Correctly
A priority matrix, also called an incident priority matrix or incident scoring system, is an internal resource that results in significant reduction in priority alerts (critical and high severity) and fewer fire alarms both during and off hours. As such, even though it takes time to develop and must undergo regular review, it’s a valuable document.
Various methods are available for developing this matrix, but they all typically involve identifying the severity of the activity under review and the level of criticality based on the systems and users involved. As an example, a phishing alert for one user would be less critical than if the SOC received more than 20 alerts for phishing within a five-minute timeframe.
To develop a priority matrix, security teams must first identify their severity levels, which are typically on a scale from low to critical/emergency. Low is the least impactful, usually informational alerts, and critical corresponds to a major event. Next, they must determine the types of alerts they deem critical or high priority and define thresholds around those, which I discuss more below. Again, a single alert may be deemed medium severity, but if it occurs multiple times in fewer than five minutes, that may warrant an upgrade to high or critical.
Similarly, medium- and low-severity alerts must be defined and clear criteria should be outlined for when escalation is warranted. Very few scenarios should require a low- or medium-severity alert to be escalated to high or critical, requiring additional security and IT personnel to be contacted and looped in to the response activities.
3. Use Thresholds and Ensure They’re Set Appropriately
As mentioned, a single occurrence of an event may be considered low or medium criticality, but if that event occurs continuously or multiple times in a specified timeframe, the team may decide to configure an alert to trigger. Using thresholds to detect multiple occurrences of suspicious behavior can help significantly reduce the number of low-priority alerts and false positives being generated by your SIEM (security information and event management).
Oftentimes, organizations avoid setting thresholds for fear of missing important detections. If the team agrees one or two occurrences of an event is benign, but then collaborates to define the number of occurrences that warrants an investigation, they can implement the threshold and tune as necessary. This enables the team to collectively agree the alert is necessary and work to make it as valuable as possible while also ensuring the level of detection/prevention is appropriate.
4. Automate Anything That Can Be Automated
Automation, a huge buzzword in the industry, is one of the primary ways to reduce alert fatigue. A security program can implement automation in various ways. For starters, it can automatically close low-priority alerts as informational. Often, security teams require certain alerts for auditing purposes so the data is available for review on an ad-hoc or scheduled basis, but don’t want to look at each individual alert. In these situations, closing alerts automatically makes sense, saving the SOC team members a little bit of time.
More commonly, automation is necessary when response activities are repetitive. A typical implementation of this is automated password resets when the system detects a click on a phishing URL. When it comes to phishing or malware alerts, it’s best to err on the side of caution, or in security terms, assume compromise.
Most organizations’ first step when detecting a phishing attempt is to reset the user’s password, even if they claim to have not clicked on anything. For larger enterprises with thousands of employees, this procedure can take up hours of time in a given day. Implementing automation to reset passwords enables security staff to focus on the more complex incident response activities and reduce the amount of mundane, repeatable activities they must do.
5. Continuously Review and Improve Existing Alerts
Managing security alerts is an ongoing process that requires frequent review to ensure the alerts are actionable above all else. Security teams should constantly review the alerts, asking questions like these:
- Is this providing value?
- Is this a false positive and, if so, how do we improve the accuracy of the alert?
- Is this alert redundant?
The easy thing to do with benign alerts and false positives is to close them out, effectively ignoring that an inefficiency exists. Security staff should constantly question the value of alerts that seem to be nothing but information, understand why they’re receiving so many false positives, and so on. Asking these questions and challenging other team members and leadership to answer them enables healthy discussion about whether the alert in question is necessary or should be tuned to make it more valuable.
Reduce Alert Fatigue for Your Security Team
You may have noticed some common themes throughout this article: false positives, actionable alerts, prioritization, and so on.
Many of the activities outlined above tie into one another and require collaboration across the security team. Although management might be responsible for signing off on priority levels and engineers are the ones configuring the alerts, the SOC analysts are the ones living in the SIEM from day to day and should work closely with the rest of the security organization to ensure the alerts they receive are actionable and providing value.
As with most aspects of cybersecurity, developing effective alerting to avoid alert fatigue requires diverse approaches. It also works best when the team members work cross-functionally to ensure any enabled alerts are required, the prioritization aligns with the severity of the alert, and the alerts generated have corresponding procedures.
As the capabilities to detect suspicious and malicious indicators and behaviors advance, security teams must further develop and mature their alerting processes to reduce the likelihood of alert fatigue. Although avoiding false positives won’t always be possible, you can control the level of alerting that’s enabled and appropriately tag the alerts that are generated to avoid an inundated SOC.
