This is Part One of J.’s new series on AI and Judaism, “The Perils and Promise of AI.” Learn more about the series here.
Scott Wiener stood before the California Senate last May in a gray suit, blue tie and matching blue Star of David lapel pin to introduce the third reading of his sweeping artificial intelligence bill. The Safe and Secure Innovation for Frontier Artificial Intelligence Models Act had two explicit aims. One was to make sure that California stayed at the cutting edge of AI’s warp-speed development. But the true heart of the bill focused on erecting what the state senator from San Francisco called “basic safety guardrails” around the new technology.
That’s because AI promises both extraordinary potential benefits and possible doomsday risks.
“Our responsibility,” Wiener later told me, “is to try to get ahead of those risks, anticipate them and, to the extent we can, reduce those risks so that people can experience the benefit of this technology.”
Congress, Wiener said, was paralyzed on the issue. Complacency was not tenable. By placing himself at the heart of the AI debate, he was adding himself to the very short list of people vital to the future of the technology. Most of them live within 30 miles of his constituency. Several, like Wiener, happen to be Jewish.
Editor’s note: Introducing ‘The Perils & Promise of AI’ series
AI’s potential benefits are legion. Bill Gates thinks it could save the lives of millions of children who die of preventable diseases and mitigate the effects of climate change. Roy Bahat, head of Bloomberg’s venture-capital wing, Bloomberg Beta, has said that AI could be a bigger deal than the whole of the internet, and now thinks it could be even bigger than that. Andrew Ng, founder and CEO of DeepLearning.AI, said it will become as big as electricity.
But with the launch of OpenAI’s ChatGPT in December 2022, artificial intelligence went from a distant science-fiction dream to a newborn reality. By March 2023, Elon Musk and Steve Wozniak, among other tech leaders, were calling for a pause in development because of AI’s “potentially catastrophic effects on society.” Two months later, OpenAI’s Sam Altman, along with the chiefs of other major tech outfits, signed prominently onto a petition from the Center for AI Safety, worrying that “the risk of extinction from AI [ranks] alongside other societal-scale risks such as pandemics and nuclear war.”
Whichever side you come down on — AI as a savior or AI as a destroyer, and many believe both notions have merit — one thing seems clear: This emerging technology will soon be ubiquitous.
“AI will play a role in a lot more than we expect,” said Craigslist founder and internet entrepreneur Craig Newmark. “We’ll see it everywhere, especially if it’s deployed with a serious commitment to doing it well.”
The speed of AI’s development and adoption has been astonishing. ChatGPT itself reached its millionth user within five days, had over 100 million users by the end of its first full calendar quarter, and at the end of 2024 boasted over 300 million users every week. ChatGPT is just one of many massive, public large language models (LLMs), alongside Google’s Gemini, Anthropic’s Claude, Meta’s Llama and others.
Which is why legislation like Wiener’s SB 1047 seemed so urgent. In essence, what he was trying to convince his colleagues to do was impose legal oversight not only on the development of AI, but on how to shut it down if need be. The bill would obligate all major AI systems to include a kill switch allowing executives to turn off a system that, by accident or design, was threatening to cause a mass casualty event — developing a weapon of mass destruction, say, or engaging in a broad cyberattack. As Wiener noted that day in May, two of the world’s most celebrated machine-learning researchers, both of whom were supporting SB 1047, had repeatedly warned policymakers that failure to act could have “severe consequences.”
How it all works
Because of ChatGPT, many people associate AI with the interfaces that provide banks with infinite experts on their “contact us” chats or help students cheat on essays, but the term is much more general. These generative AIs are just the latest and most visible form of a class of machines that mimic human intelligence.
What has changed recently is the ability of those machine-learning systems to process massive amounts of data and, with the explosion of usable data produced by the internet, the ability of their programmers to provide the systems with vast datasets — whether shopping patterns, biometric readings from health apps or every single word that has ever been digitized.
Machine-learning algorithms identify patterns in data and make predictions about what word, number or consumer good should come next based on those patterns. As a result, artificial intelligence is an immensely powerful tool able, for example, to comb through sets of medical histories and find a pattern of symptoms even when they are described differently in varying locations and disparate eras. What users might do with that diagnosis, though, is a human decision.
Contemporary AIs are, essentially, superefficient autocomplete programs, taking as much of human achievement as they are provided and predicting, on the basis of what has been correct before, what comes next in the sequence. Depending on the prompt, the next item can be a response to a question, a continuation of a sentence, a new chemical compound, fake voices or even videos.
Instead of giving computers a task with a specific determined outcome, like calculating an escape trajectory for a rocket, today’s AI developers produce models that suggest probable results. Depending on the developers, the models could help accelerate searches to produce new vaccines, or to start new pandemics.
This increase in power and shift from “deterministic” to “probabilistic” outcomes is the key to AI magic, but it’s also why models can “hallucinate.” Sometimes there will be inadequate information or a problem in the statistical weighting, and the models will predict items wrongly, making public errors in ways that we recognize.
What AI systems lack in intention or what we call common sense — machines have no human intuition, understanding, desires or intentions — they make up for in computational power. Because they are simulating human language and the understanding encoded in the ways that we use it, they seem to know who we are and what we mean.
LLMs are, however, just a special case among many — predicting linguistic outcomes, where other AIs predict medical, military, political or other outcomes. And there are many in the tech community who believe that the growth and convergence of AIs will lead to artificial general intelligence (AGI), which could perform any intellectual task a human can. Unlike narrow AI, which excels in specific tasks like playing chess by predicting winning moves, AGI could adapt, learn and reason across diverse domains without human intervention.
The logical continuation of AGI is ASI (artificial superintelligence) in which AI surpasses human intelligence in all aspects, including creativity, problem-solving and emotional intelligence. If developed, ASI could revolutionize science and society, but it also poses obvious ethical risks.
California’s impact
Lawmakers in state legislatures across the United States propose new bills every day, though most receive little outside attention. But with no federal legislation on the cloudy horizon, California’s actions can have significant national impact on AI. The state sits at the epicenter of the AI revolution. On Forbes’ list of the 50 most promising privately held AI companies, 32 are based in California, nearly all of them in the Bay Area. Indeed, over 50 separate regulatory bills were proposed to the state Legislature in 2024 alone, and 17 of them were signed into law. Most, though, only regulated aspects of the technology, like transparency of use or consumer privacy. Wiener’s SB 1047 was more expansive.
In a terse note in 2023, Ben Horowitz, the co-founder of Andreessen Horowitz, one of the biggest tech venture-capital companies in the world, had outlined his company’s thinking about regulating AI. “[We] need clear regulation to eliminate bad actors exploiting the technology for nefarious purposes,” he said. Still, “misguided regulatory policy” can undermine American dominance in AI, he said, and, implicitly, American investments. Bad regulation must be opposed, Horowitz said, because AI “must not be choked off in its infancy.”
Horowitz’s salvo came before SB 1047 had been composed, but Wiener was aware of the sentiments. As he was submitting his bill, he committed to continue “soliciting and incorporating constructive feedback.” It made political sense to pay attention to the head of a local multibillion dollar company in the industry he was trying to regulate, but he was not solely talking to Horowitz. For all the tech execs and lobbyists listening, Wiener’s offer in that May Senate speech was an olive branch, a promise that as the bill went through the process of becoming law, their voices would be heard.
One of those execs was Sam Altman, the 37-year-old CEO of OpenAI. The baby-faced entrepreneur from St. Louis, Missouri, has become the human face of AI, and his company has become almost synonymous with the new magic chatbots. In an interview with Time magazine editor-in-chief Edward Felsenthal, Altman explained how his upbringing had inculcated both boundless enthusiasm and extreme caution. “I’m a Midwestern Jew,” Altman said. “I think that fully explains my exact mental model, very optimistic and prepared for things to go super wrong at any point.”
He found, though, that his new celebrity made him a target for hate even as it brought new responsibilities. Online he saw his social media tarred with antisemitic slurs, which he talked about in interviews and posted about on X (formerly Twitter): “i am jewish. i believe that antisemitism is a significant and growing problem in the world, and i see a lot of people in our industry sticking up for me, which i deeply appreciate.”
By the time Wiener presented his bill, OpenAI was no longer an impudent newcomer. It was valued at over $100 billion and had attracted a multibillion-dollar investment from tech behemoth Microsoft. OpenAI had begun to fill downtown office real estate in San Francisco, and Altman later took up a position on the transition team of Mayor Daniel Lurie.
Even so, Altman, who declined to participate in this article, had harbored deep concerns about the potential consequences of AI technology. Nearly a decade earlier, he had said that AI would “most likely lead to the end of the world.” As a poster child for the AI industry who was already on the record with concerns for its safety, how he came down on the debate would be crucial for the ultimate success of Wiener’s bill.
An existential threat
The first atmospheric river of the season was approaching San Francisco when I spoke to Wiener in January 2025. It was just over a year since he had begun the preliminary filings for what would become SB 1047 and, though he is still passionate about the topic, his tone suggested he had grown weary of talking about it.
I asked why a state senator who until now had concentrated on such issues as housing and net neutrality would want to insert himself into such a controversial debate. “This is an issue that affects everyone and will affect everyone,” he said.
AI elicits strong feelings — from those who believe it will save humankind to those certain it will destroy us. Which is why getting to the right legislation could be so important, and so tricky. Before writing his seminal bill, Wiener said, he convened “a whole bunch of meetings” with people on all sides of the AI debate.
Although Wiener would not tell me who was at the meetings, or exactly what was said, he hinted that significant players were involved. Altman, with his concerns for safety in tension with his institutional desire for freedom from regulation, would have been an obvious partner in trying to thread the needle. After all, in September 2023 Altman had testified at a U.S. Senate subcommittee hearing on artificial intelligence that “we think that regulatory intervention by governments will be critical to mitigate the risks of increasingly powerful models.”
That hearing was part of the federal government’s attempt to grapple with AI. Eighteen months later, the Senate’s Bipartisan Framework on AI Legislation has yet to become law, or even reach a meaningful vote. With President Joe Biden’s executive order 14110 from October 2023 outlining federal safety policy on AI vulnerable to repeal, Wiener seized the moment. Many other California bills had touched on the uses and effects of AI — including transparency, copyright issues and privacy — but none had centrally addressed the existential threat that AI posed to human civilization. The stakes could hardly have been higher, nor the time shorter.
“This is about catastrophic risks that are very tangible and real,” Wiener said. “Because people are trying to inflict harm today. Like shutting down the electric grid or melting down the banking system or creating nuclear, biological or chemical weapons. … There are people on the planet who are trying to inflict these harms. And the question for us is, do we want to make it easier for them to inflict these harms, or do we want to try to reduce the risk of these harms in creating these massively powerful models?”
Over the next three months, Wiener negotiated with fellow legislators in Sacramento and tech insiders in the Bay Area. The bill went through committees in both houses and through three sets of readings in each. Wiener let go of a number of early suggestions but managed to maintain the two main principles of encouraging innovation and instituting “guardrails.”
On Aug. 19, when the amended text finally emerged and was sent back from both houses for Wiener to present for a vote, the proposal also included three significant provisions beyond the kill switch: transparency, certification and public research. With these obligations, massive AI companies would be under the scrutiny of researchers, the government and the companies themselves.
The initial plan had mandated a new government entity, the Frontier Models Division, that would establish an accreditation process for big tech companies to show how they were minimizing catastrophic risk from advanced AI. In July, Wiener spoke to an audience at the preeminent, and extremely influential, tech incubator and investor Y Combinator. While still suggesting that the state set up this FMD, Wiener said he would consider moving away from that provision. And indeed, by August, the idea of the new FMD had been scrapped in favor of setting up an oversight committee within the existing Government Operations Agency.
In addition to the oversight body, however scaled back, the state was proposing to set up a publicly funded body called CalCompute to review massive AI companies. CalCompute would work hand in hand with protections for whistleblowers to keep huge AI developers accountable and would provide an ecosystem for researchers in the state to stay relevant and innovative, even when they were not directly affiliated with the AI giants.
With all of these changes in place, the bill was ready to move from its amendment phase to a final vote in the state Senate. Then, on Aug. 20, OpenAI came out against the legislation.
“SB 1047 would threaten … growth, slow the pace of innovation, and lead California’s world-class engineers and entrepreneurs to leave the state in search of greater opportunity elsewhere,” chief strategy officer Jason Kwon wrote in a letter.
Even so, the bill handily passed its third reading on Aug. 28 and, again overwhelmingly, won a vote the next day to be sent up to Gov. Gavin Newsom to be signed into law. Instead of being a victory for compromise and consensus, though, the OpenAI letter meant that Newsom would need to make a choice. Wiener was now pitted against Altman. The governor would have to choose between the two, between democratic governance and financial power, between regulation and no regulation.
Newsom chooses
On Sept. 29, Newsom vetoed the bill, damning it as “a solution that is not informed by an empirical trajectory analysis of AI systems and capabilities.” Although it’s dead, SB 1047 remains one of the most significant legislative efforts in the country to address the risks of a catastrophe caused by AI. What’s more, this one made it the furthest among other legislative proposals of similar ambition in the United States, passing both chambers before the veto.
Newsom critiqued SB 1047 for regulating models based only on their cost and size, rather than function, and failing to “take into account whether an Al system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data.” Still, he acknowledged that a bill like SB 1047 was necessary.
“Safety protocols must be adopted,” the governor said. “Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable. … [California] cannot afford to wait for a major catastrophe to occur before taking action to protect the public.”
Wiener, too, is determined to continue to talk to stakeholders, as are his allies. Immediately after the veto, he told me he was already “talking to both supporters and opponents of the bill,” preparing to “engage with the governor’s team” to see what he can do in 2025. He has just proposed a new bill, SB 53, that would establish a version of CalCompute and protect whistleblowers as SB 1047 had intended.
Already in 2025, President Donald Trump has revoked Biden’s executive order on AI, leaving national policy unstated and AI largely unregulated at a federal level. Again, a California senator has stepped up with a proposal. This time, state Sen. Steve Padilla from San Diego has introduced an AI Bill of Rights that might cover some of the same ground as the executive order or the Wiener proposal.
Threading the needle between oligarchical billionaires who want minimal anti-extinction regulation and politicians who want to safeguard their constituents might be an impossible task. But it’s a vital one, not only for the people of California, but for the 8 billion people who are about to be affected by AI.
It’s a task that requires not only superintelligence, but a degree of political adroitness that is well beyond even the most advanced AIs. No matter how much quantum computing we have at our disposal, the disagreements over how to safeguard AI are human problems that require human solutions — and fast. To paraphrase the Mishnaic sage Rabbi Tarfon: The time is short and the work is great.