Stack Overflow has updated its initial security notice with additional details of the recent breach of its systems—unauthorized access to its production systems. The company’s prompt response to the breach and updates with additional details is a an example of how a company should communicate with users after security incidentst.
Attackers had access to the Stack Overflow’s production systems for nearly one week and some user data was exposed, Stack Overflow’s vice-president of engineering Mary Ferguson wrote in the update. The initial notification on May 16 had said there was no evidence that customer or user data was compromised.
The initial intrusion occurred on May 5 after attackers gained access to stackoverflow.com’s development tier through a bug that had been released in that day’s build and escalated their access level to get to the product environment, Ferguson said. They spent a few days reconnoitering, and then performed an action to gain privileged access to production systems. This allowed them to make privileged requests that allowed them to obtain names, email addresses, and IP addresses of some Stack Exchange users.
“This change was quickly identified and we revoked their access network-wide, began investigating the intrusion, and began taking steps to remediate the intrusion,” Ferguson wrote.
Stack Overflow has separate infrastructure and networks for its Teams, Business and Enterprise products and there is no evidence that any of these systems or their customers have been impacted. Stack Overflow’s Advertising and Talent units also don’t appear to be affected. Roughly 250 users appear to be impacted and have been notified.
The company is looking through its logs for other suspicious activity and taking other “precautionary measures such as cycling secrets, resetting company passwords, and evaluating systems and security levels” in response to the incident, Ferguson said.
Breach Notification Done Well
Some Stack Overflow users praised the company’s prompt announcement and subsequent update. “I think this is one of the best sets of responses to a security incident I've seen,” a user wrote on Hacker News. The user identified two things Stack Overflow did well: disclosing the incident as soon as possible and being straightforward that the company was still investigating and didn’t have all the information yet, and adding more details during the course of the investigation.
“The proactive communication and transparency could have downsides (causing undue panic), but I think these posts have presented a sense that they have it mostly under control...I expect the next (or perhaps the 4th) post will be a fuller post-mortem from after the incident. This series of disclosures has given me more confidence in Stackoverflow than I had before!” the user wrote.
Organizations have to balance speed and thoroughness in breach notifications. Waiting to have all the information before making an announcement opens the organization to accusations of trying to hide the bad news or leaving users in the dark. Notifying users promptly without any details runs the risk of causing panic, overplaying the severity of the incident, or underestimating user impact. It is a tricky line, but the general recommendation is to notify promptly.
Stack Overflow struck the right tone, explaining that the investigation was underway and didn’t try to speculate or guess on the impact. “After we conclude our investigation cycle, we will provide more information,” Ferguson wrote in the initial notification.
In the update, Ferguson outlined the steps taken, such as “conducting an extensive and detailed audit of all logs and databases that we maintain, allowing us to trace the steps and actions that were taken,” and “remediating the original issues that allowed the unauthorized access and escalation, as well as any other potential vectors that we have found during the investigation.”
Elements of a Good Response
An incident response plan is invaluable in the case of a breach because it clearly defines the stakeholders and establishes a course of action. A good incident response plan also defines clear communication channels—how information is shared internally among employees, investigators, and other stakeholders; and who coordinates the communications externally. The plan needs to be clear about what customers will be told, and how. It is extremely easy to botch response.
One of the reasons companies get excoriated after a breach is because of the perception that they were not honest about what happened.
This might sound like the most ridiculously obvious thing to say, but don't lie when disclosing an incident," wrote security researcher Troy Hunt, in a discussion of how organizations should respond to data breaches. "I know the truth may hurt, but the harsh reality of most data breaches is that there's been a failure at some point and now you need to own that.
It’s not enough to just be prompt. Organizations need to be clear about what happened, admit fault if there was a mistake, and accept responsibility. They should provide mitigation details if they know what happened, share information on what they are doing to prevent similar issues, and provide tips on what users can also do.
There are other examples of organizations handling breach response well. Last year, sports apparel maker Under Armour disclosed that an unauthorized party had acquired data associated with user accounts on the company's diet and fitness tracking app MyFitnessPal.
“Under Armour is showing it learned some lessons from companies breached in recent months by notifying its customers rapidly after discovering the intrusion,” Forrester security expert Jeff Pollard said at the time.
Hunt praised two companies, image site Imgur and comment moderation site Disqus, for how they handled their data breaches back in 2017, .
Hunt notified Imgur of stolen data covering 1.7 million user records just before the Thanksgiving holiday in 2017. Imgur made full public disclosure of the breach 25 hours and 10 minutes after Hunt’s initial communication, Hunt wrote on Twitter. Disqus took 23 hours and 42 minutes from when Hunt notified the company of a breach involving email addresses, usernames, sign-up dates, and last login dates for 17.5 million Disqus users to public notification and protectting the accounts, Hunt said on Twitter, calling the response "exemplary".
Disqus “applied urgency,” disclosed right away, provided details, acted quickly to protect impacted accounts by resetting passwords, and apologized to users, Hunt wrote at the time. "This was a dark moment for Disqus and there's no sugar-coating the fact that somehow, somewhere, someone on their end screwed up and they lost control of customer data. But look at the public sentiment after their disclosure; because of the way Disqus handled the situation, it's resoundingly positive."