· AI
New York and California Want an AI Kill Switch Nobody Can Build Yet
The headline is a switch that could shut down a frontier model. The law already on the books shows what these ideas look like after industry gets a turn at the wording.
Governor Kathy Hochul raised the idea of an AI kill switch with reporters on September 21, floating it as a possible safeguard the state could pursue as it gets ready to enforce its new AI safety law. amNewYork reported that she called it preliminary, with no commitments yet. Four days earlier, California's Gavin Newsom had ordered his own working group to spend two months studying the same idea. His executive order directs the state to advance a shutdown mechanism for frontier models whose effectiveness would be checked on an ongoing basis by an outside verification group. Read the order closely and there is no engineering behind that goal. No spec for how you force a model already running inside a customer's infrastructure to stop. Two governors, four days apart, floated the same headline with nothing built underneath it.
That gap between announcement and mechanism is the actual story, and New York already gave us a preview of how it plays out once a real bill goes through the legislature.
The RAISE Act, signed in December 2025 and taking effect January 1, 2027, only applies to companies with more than $500 million in prior-year revenue that train models on more than 10^26 floating point operations, roughly the compute scale of today's largest frontier systems. Narrow by design. But the version Hochul signed is not the version the legislature passed. Reporting on the negotiation found that the governor swapped in language closer to California's own SB-53: the binding safety "protocol" became a looser "framework" describing a company's general approach, the outright ban on releasing models with "unreasonable risk of critical harm" was dropped, and maximum penalties fell from the $10 million to $30 million range the legislature had written down to $1 million for a first violation and $3 million after that.
Run the math on that against the $500 million revenue floor that triggers the law in the first place. A company that just clears the threshold could eat the maximum penalty for under six tenths of one percent of a single year's revenue. That is not a deterrent. It is a line item.
The steelman here is fair and worth stating plainly. Frontier AI spending is running at a scale no state has regulated before, Washington has shown little appetite to act (the administration pushed AI deregulation at a G20 meeting this month), and a governor saying "we are studying this, nothing decided" is a reasonable way to signal intent before writing something irreversible. New York's own bill sponsor argued the technology is moving faster than the legislative calendar, which is a fair read of the last two years regardless of where you land on the remedy.
But a state cannot hold both positions at once. Either frontier models carry enough risk to justify an emergency shutdown capability, in which case the law on the books should have kept the teeth the legislature gave it, or the risk is manageable enough that a $1 million cap and a "framework" in place of a binding protocol are the right call, in which case the kill switch talk is mostly noise. Industry did not treat the weakened bill as a win and move on, either. Andreessen Horowitz's Anjney Midha called the RAISE Act, even in its softened form, "yet another stupid, stupid state level AI bill that will only hurt the US." Getting a bill's penalties cut by lobbying is not the same thing as the market solving a real problem. It just moves the cost from the company's balance sheet to the uncertainty every other business now has to price in.
None of this touches you directly. You are not clearing $500 million in revenue or training a model on ten to the twenty sixth flops. But you are exposed to the mechanism if either state ever builds one: if your product depends on a single frontier vendor's hosted API for something core to how you run, a real shutdown order aimed at that vendor becomes your outage, not just theirs. That is the actual argument for knowing, ahead of time, which parts of your stack you could run somewhere else and which model you could swap in on a bad week. If you have not answered that question, it is worth working through now rather than after the next press conference makes it urgent.
Sources
References used in this article. Links also appear alongside the relevant claims.
Tell us what's on your mind.
You don't need a polished brief to reach out. A two-line email about what's bugging you is plenty; we'll tell you straight if we're the right fit, and what we'd tackle first.
We'll scope the work around your workflow, goals, and timeline before quoting anything, so you know what's included before committing.