Who should control AI?

Nonprofits aren't our only option.

This essay is research for a book I’m writing called We Should Own The Economy, readers can invest in the project and earn a share of profits when it’s done 👇🏻

When OpenAI launched in 2015, they did so with an impressive mission: To develop AI that would benefit humanity by automating repetitive work or curing cancer, but without unintended consequences that might harm humanity, like rapidly displacing jobs or trying to kill us all.

At the time, Google was already working on AI technology and Sam Altman, Greg Brockman, and Elon Musk thought that wasn’t a good idea. Altman wrote: “[I’ve] been thinking a lot about whether it’s possible to stop humanity from developing AI. I think the answer is almost definitely not. If it’s going to happen anyway, it seems like it would be good for someone other than Google to do it first.”

The three launched a nonprofit organization and research lab with no equity shareholders, just donations from tech philanthropists. They hoped to raise $1 billion in donation commitments so they could develop the technology without taking on investors. “Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return,” they said at the time. “Since our research is free from financial obligations, we can better focus on a positive human impact.”

Three years later, competitors from Google, Amazon, and Meta had significantly ramped up and OpenAI had raised only $137 million. The company needed more money to fund R&D and computing costs, but they still wanted to do it safely. In 2019 they restructured, creating a for-profit company that could attract investment as well as give employees shares in its success, but with earnings capped at 100x their investment. Anything above that funded the nonprofit, which served as the company’s owner and controlling entity while remaining beholden to their ethical mission, not investors.

OpenAI ownership model circa 2019. Source

This new hybrid structure was an innovative approach to ethical technology ownership. But there was inherent friction between the two organizations from the start. The nonprofit wasn’t building the technology—the for-profit was—and there was limited oversight the nonprofit could provide without being involved in day-to-day operations.

The two organizations also had competing values. Until 2019, the nonprofit had been open source, publishing their research and sharing code freely. Now, technical details for GPT-3 and GPT-4 were withheld on competitive grounds. If Google copied OpenAI’s code and beat them to the market, they would create the very circumstances OpenAI was funded to avoid: Google rushing the technology to market to benefit shareholders, rather than OpenAI developing it safely for the benefit of humanity.

After OpenAI debuted their first product, ChatGPT, in 2022, the friction intensified. The nonprofit worried the company was growing too fast—that the model still incentivized risky, profit-driven deployment of AI until the cap kicked in. Everything came to a head in November of 2023 when the nonprofit board abruptly fired OpenAI CEO Sam Altman and asked co-founder and President Greg Brockman to step down as co-chair, citing lack of transparency and safety concerns.

Immediately, the board faced massive backlash from the company—employees universally backed Altman, as did investors. Within a matter of days, Microsoft announced they would hire Altman and Brockman to lead a new AI team at Microsoft. Shortly after, nearly all of OpenAI’s employees—over 700 staff members—signed an open letter to the board demanding the reinstatement of Altman and the resignation of the board members who had fired him. They accused the board of abuse, saying: “Your conduct has made it clear you did not have the competence to oversee OpenAI.”

If the board didn’t comply, employees threatened to quit en masse and follow Altman to Microsoft.

Less than a week after the firing, Altman and Brockman were reinstated and the entire board was replaced. A law firm investigating the issue found the nonprofit’s reasons for firing lacking, and found no ethical or safety breaches on the part of Altman. One of the board members responsible for the firing even acknowledged their error and publicly denounced their prior decision. The company returned to business as usual, but began exploring alternative governance structures that might work better than their current one.

By the end of 2024, OpenAI announced their new structure: The nonprofit would be removed as the owner and controller of the company and instead become a benefactor of it, owning shares in the company which would fund their social mission. Meanwhile, the for-profit company would become a Public Benefit Corporation, just like competitors Anthropic and xAI.

Critics panned the move as a bad one. Public Benefit Corporations do not enforce the better actions of companies, and fail to hold them accountable for social results. More critically: Without the nonprofit holding the company accountable, it could now answer to corporate interests instead—like SoftBank, now leading a $40 billion funding round. As a for-profit company, the board could comprise the very financial interests the organization once found dangerous.

In April of 2025, a group of former OpenAI and Google DeepMind employees, as well as nonprofit organizations and law firms, penned an open letter to the California and Delaware Attorneys General arguing against the move, and demanding the nonprofit remain in control of the company.

It worked.

On May 5th, OpenAI announced that, after discussions with both Attorneys General and other civic groups, they would leave the nonprofit in place as the company’s controlling entity. The for-profit business would still become a Public Benefit Company, and the nonprofit organization would still benefit from that company as a primary shareholder. The cap on investor returns was removed and OpenAI can now raise significant investment, but the nonprofit retains the power to fire the CEO if the organization acts unethically in the future.

I think this is an interesting compromise, and perhaps a good one. Several companies have nonprofit-controlled boards including Novo Nordisk, Bosch, and the Guardian; and many are doing interesting things with Public Benefit Corporations, like Patagonia. But dozens of AI competitors have since entered the market, none of them particularly ethically controlled. Anthropic is controlled by investors and founders, DeepMind by Google. Chinese competitors have entered the space, forcing every organization to act more quickly. Despite OpenAI’s ethical intentions, investment-backed AI is here, with shareholders racing products to market. It simply won’t matter if OpenAI develops “safe AI,” if the rest of the world develops “unsafe AI.”

It’s worth asking who should control these companies. A nonprofit might work in OpenAI’s case, but what about the rest of them?

We are demanding better ownership structures for risky technologies, and for companies at large, and OpenAI has brought that to the forefront. But let’s remember what specifically this debate is about: We are asking who should serve on the board of directors for a company. We are asking who should be part of that inner circle that has the power to fire the CEO when they act unethically. Investors will prioritize financial growth above all other objectives, but nonprofits don’t always prioritize the business and can risk its existence altogether.

Thankfully, our options are not split between a board controlled by investors and a board controlled by a nonprofit.

There is a third, and potentially better option available to us: