On Open-Weights Foundation Models

The history of traditional open-source software provides a vision of the value that could result from the availability of open-weights AI models – including enabling greater innovation, driving competition, improving consumer choice, and reducing costs. Understanding these models, including their upsides and risks, can help realize potential competition benefits and avoid consumer harm.^[1]

Open-source software (OSS) has provided significant benefits to software development and innovation. OSS can be adopted and incorporated into other software projects on well-known license terms, which give companies and individuals the freedom to use and modify open-source projects, and even build businesses on top of them.

The open-license paradigm of open-source software has resulted in reduced costs. A 2017 study argued, for instance, that increased use of open-source software by firms led to increased rates of productivity.

OSS can also allow for increased competition in areas other than software. For example, the availability of open-source data storage and software deployment platforms allows developers to choose from multiple cloud hosting providers. This is in contrast to a world where any given data storage solution is usable on only a single hosting provider, limiting the portability of applications and thus competition between hosting providers.

While OSS is well-defined, there is still an active dialogue around what “open” and “open-source” should mean in the emerging context of AI models– and it is important to understand the range of definitions when assessing the potential impacts. A complete definition could include a variety of attributes: For example, it could require some set of components of a model, such as the training data, software systems used to train the model, or the model’s weights (the data that results from training and allows a model to generate content based on a prompt) to be made openly available. It could also require that model components be licensed with terms that allow for broad use and reuse. Or it could require freely available publications on the design and advances of the model.

Because this dialog remains active, this post eschews the term “open-source” when talking about AI models; instead, it uses the term “open-weights model” to refer to models for which weights are publicly available. This does not reflect a normative view of what “open” should mean, but rather it is availability of a model’s weights that can enable benefits and risks discussed below. This usage is also consistent with how other government agencies are approaching this issue.

Despite the nascent terminology, open-weights models, like OSS, have the potential to be a positive force for innovation and competition. Building on the traditions of open-source software, open-weights models can, in principle, be incorporated into a product without going through a procurement process and can be tweaked and fine-tuned to suit the needs of the user (e.g., a product developer researcher, or scientist), providing flexibility to those who want to use advanced models without forcing them to train a model from scratch. Open-weights models can further enable competition by offering portability between different hosting providers.

For example, imagine a firm wishing to create a tool for their employees to aid in drafting internal memos in their house style, trained on a corpus of past internal memos. An open-weights large-language model would allow them to take an existing off-the-shelf model and fine-tune it with their internal documents – effectively providing the model with additional training data. As compared with closed models, the firm would be able to run this model on servers they operate (as opposed to sharing their internal corporate documents with the LLM’s creator) and have a greater degree of control over how the fine-tuning process occurs. And the firm would accomplish this without incurring the costs of training a model entirely from scratch.

In addition to recognizing the potential benefits, it is also important to be attuned to restrictions associated with some models claimed as “open.” For example, a model developer may make the model’s weights available, but only under licensing terms that are so restrictive as to effectively prevent its use in the marketplace – such as by prohibiting commercial use. Models with such licenses may be useful for research but are unlikely to directly promote competition. Other license restrictions could have a similar dampening effect on open-weights models’ benefits to competition.^[2]

There is also the risk of firms deploying openness opportunistically in ways that may provide some benefits in the short term but end up consolidating power over the long term. For example, in previous generations of technology, companies have employed an “open first, closed later,” strategy – starting off making a product openly available to gain market share, only to later pivot to a closed approach. In some instances, this strategy enabled firms to gain dominance and lock out rivals.

Open-weights models also directly impact consumers. On the positive side, because some open-weights models can be run on consumers’ devices, they have the potential to improve privacy, security, and auditability. For instance, closed models generally involve sharing user inputs with the model creator or a hosting provider approved by them. But with an open-weights model, the model’s creator need not gain access to the user’s queries or their results. Additionally, auditing and research into an open-weights model’s behavior – critical to understanding and preventing consumer harms – faces fewer potential roadblocks.

But open-weights models also pose additional risks to consumers over centralized closed models. The lowered costs and barriers to retraining and redistributing models extends not only to well-meaning companies but also to malicious actors scaling up spam, scams, and other harmful uses.

Malicious uses vary by particular model and modality (text, image, audio, video, etc.), and the marginal added risk of open-weights models (over existing technologies or over AI models generally) is under active research. But certain open-weights models have already enabled concrete harms, particularly in the area of nonconsensual intimate imagery and child sexual abuse material. While model developers may add technical guardrails to their systems – training the model to avoid generating content unwanted by the developer such as inaccurate, illegal, or abusive content – these precautions are not currently robust against persistent bad actors, and they can be defeated by prompting or fine-tuning such that the model “un-learns” the guardrails. How to train more robust protections into open-weights models remains an area of ongoing research.

In summary, open-weights models have the potential to drive innovation, reduce costs, increase consumer choice, and generally benefit the public – as has been seen with open-source software. Those potential upsides, however, are not all guaranteed, and open-weights models present new challenges. Staff are paying attention to the impact these models have on the market, and how they affect competition and impact consumers.

***

Thank you to those who contributed to this post: Alex Gaynor, Simon Fondrie-Teitler, Mike Tigas, Amritha Jayanti, Stephanie T. Nguyen, Michael Atleson, Krisha Cerilli, Leah Frazier, Wells Harrell, Ted Rosenbaum, Daniel Salsburg, Kelly Signs