Security Principles: Addressing underlying causes of risk in complex systems

On December 14th, 2022, in collaboration with technologists on team CTO and attorneys in BCP, I gave a presentation at the Federal Trade Commission’s December Open Commission meeting on the systemic approach to security found in the Commission’s orders.

The goal of this post is to first explain how the FTC has worked to strengthen its remedies to address the underlying causes of risk in complex systems. The post then highlights some of the Commission's recent order provisions from data security and privacy cases and explains how they seek to systemically address risk.

Addressing the underlying causes of risk in complex systems

One of the most important figures in the last decade among practitioners of computer system resiliency was an anesthesiologist. Dr. Richard Cook, a leading scholar of complex systems[1], posed questions like what do the power grid, the emergency room, air traffic control, and a server farm have in common? And, what can we learn about the people who operate these systems, and the things they do to make them work reliably? 

One of those lessons is that systems must be designed to be operated by real humans. It's remarkably common for organizations to look at an incident and declare the root cause to be "human error." Cook and his colleagues argue that human error is, in fact, only the beginning of the investigation: did the system make it easy for the human to make a mistake? Was the human warned about the risks of what they were doing? Did those warnings have such a high frequency that they fatigued humans into numbness to the messaging?

Secure systems take these things into account. They don't end their inquiry with blaming individuals—rather they focus on the system, which has always been a goal of the FTC’s data security orders. The FTC strives to craft remedies that effectuate the principles Cook and his colleagues identified about how to safely operate complex systems.

There are organizations that have adopted these lessons from complex systems and safety engineering, and they are the ones that are doing a better job living up to their responsibilities to protect users' data. But frustratingly, many organizations haven't yet recognized the value of these approaches. 

FTC orders play a role in addressing this gap. In addition to remedying bad conduct, FTC orders send a signal to market participants that they need to learn these lessons if they want to effectively protect user data. 

This post highlight three practices from recent FTC orders in data security and privacy cases which embed these principles:

1. Offering multi-factor authentication (MFA) for consumers and requiring it for employees
2. Requiring that connections within a company's systems must be both encrypted and authenticated 
3. Requiring companies to develop a data retention schedule, publish it, and then stick to it. 

While these are not the sum-total of everything the FTC expects from an effective security program, they are a sample of provisions we’ve seen recently that speak directly to the idea of attacking things at their root cause to produce uniquely effective results. 

(1) Offering multi-factor authentication (MFA) for consumers and requiring it for employees

Multi-factor authentication is widely regarded as a critical security practice because it means a compromised password alone is not enough to take over someone's account. There are many different forms of MFA available to consumers and companies, such as text messages, mobile phone apps that generate rotating codes, mobile apps that use push notifications, or security keys. Of these, only security keys are resistant to phishing and other social engineering attacks: If a user can be tricked into typing in their username and password, they can be tricked into typing in a code from their phone.

For consumers’ accounts, it makes sense for them to be able to choose from a variety of different MFA types, based on their preferences. For companies’ own IT systems, they can make the decision that their employees must use the strongest forms of MFA.

Therefore, recent FTC orders have included provisions that:

1. Require companies to offer consumers the ability to enable MFA for their accounts[2][3][4][5]
2. Require companies to use phishing-resistant MFA, such as security keys,[6] for their own employees.[7][8]

In addition, FTC orders do several things that go further in how MFA must be used in protecting consumer accounts: 

• The orders have required companies to replace their legacy authentication practices, such as the use of "security questions" with MFA. Security questions (e.g., what is your mother's maiden name?) have many weaknesses, such as giving companies even more personal information. In addition, lots of the data these questions request is publicly available, making life easy for attackers.[9]
• The orders also have required that companies offer forms of MFA that do not require providing telephone numbers, which both protects privacy and ensures that more secure forms of MFA are available. [10]
• Finally, the orders have required that companies not use information collected for MFA for any other purpose, preventing a bad practice seen elsewhere in industry and increasing confidence among consumers that security mechanisms aren’t pretexts for more data collection. [11][12]

(2)  Requiring that connections within a company's systems must be both encrypted and authenticated 

Another requirement the FTC has included in recent orders is that connections within a company's systems must be both encrypted and authenticated[13]. To understand the significance of this, one needs to compare with the prior state of the art. For many years, the approach most companies took toward security was to try to have a strong firewall outside the corporate network, but once you were inside, you could move freely. With ever-expanding systems, this meant attackers who found one vulnerability anywhere, no matter how peripheral, had the keys to the kingdom. 

"Zero Trust" is the simple idea that just because you're on the corporate network doesn't mean you should automatically be trusted to access everything. Instead “zero [implicit] trust” should be the baseline. You should have to be authenticated and authorized to access a system, and connections must be encrypted so the attacker cannot simply snoop on legitimate connections. 

This approach helps dramatically limit the blast radius of a vulnerability in a company's systems. Further, zero trust systems can build on the strong identity provided by safeguards like phishing-resistant MFA to ensure systems inside a corporate network have the same protections as those outside. 

(3) Requiring companies to develop a data retention schedule, publish it, and then stick to it

A final provision is a requirement to develop a data retention schedule, publish it, and then stick to it[14][15]. This embraces the premise that the most secure data is the data that's not stored at all. Further, implementing this requirement inevitably requires companies to have a strong internal catalogue of all the data they store. This provides other benefits, such as ensuring that they'll be able to comprehensively comply with requests from users to delete data and have the information needed to prioritize protections based on the types of data they’re storing. 

Conclusion

These principles reflect the Commission’s focus on keeping pace with technological developments, learning from past experiences, and strengthening remedies to address root causes, even as technologies evolve. Moving forward, technologists within the Commission are committed to continuing to engage with the security community and stay abreast of the latest developments. 

The expert staff in the Bureau of Consumer Protection – particularly the Division of Privacy and Identity Protection and the Division of Enforcement – have led much of this work. Our team of technologists will continue to collaborate with staff across the agency to ensure our remedies keep up with evolving technologies and threats.

Thank you to those who contributed to the production of this post: Genevieve Bonan, Mark Eichorn, Rebecca Unruh, Ben Wiseman, Kevin Moriarty, Sam Levine, Stephanie Nguyen, Zehra Khan, Josephine Liu, Daniel Zhao.

To listen to the full recording of the December Open Commission Meeting session, please visit: https://www.ftc.gov/media/open-commission-meeting-december-14-2022