California’s SB 1047, a bill initially aimed at preventing AI disasters, has been significantly weakened by amendments that reduce the state’s regulatory power, addressing concerns from AI firms while still holding developers liable for catastrophic events.

California's SB 1047, a bill originally designed to prevent AI-related disasters, has undergone significant changes following opposition from key players in Silicon Valley, including AI firm Anthropic. The bill, now passed by the Appropriations Committee, has been softened to address concerns from the AI industry while still aiming to hold developers liable for AI models that cause catastrophic events.

Key Amendments and Their Impact

One of the most critical amendments removes the power of California’s attorney general to sue AI companies for negligent safety practices before a disaster occurs. This change, proposed by Anthropic, instead allows the attorney general to seek injunctive relief or sue after a catastrophe. This amendment significantly reduces the preemptive power of the state in regulating AI safety.

Another significant change is the elimination of the Frontier Model Division (FMD), a proposed new government agency, from the bill. Instead, the responsibilities of the FMD are transferred to a newly expanded Board of Frontier Models within the Government Operations Agency. This board will be responsible for setting compute thresholds, issuing safety guidance, and establishing regulations for auditors, but its role is now more integrated into existing governmental structures.

Further amendments include a relaxation of liability measures for AI developers. AI labs are no longer required to submit safety test results under penalty of perjury, reducing the legal risks for these companies. The bill now asks developers to provide "reasonable care" instead of "reasonable assurance" that their AI models do not pose significant risks.

Additionally, the bill offers protections for open-source AI development. Developers who spend less than $10 million fine-tuning a model are not considered liable under SB 1047, shifting the responsibility to the original developers.

Reactions and Future Prospects

The changes have sparked mixed reactions. While these amendments are seen as a way to ease industry concerns and make the bill more palatable for Governor Newsom's approval, they have not fully satisfied critics. Some argue that the revisions are superficial and fail to address deeper issues, with some U.S. Congress members even urging a veto.

Despite these criticisms, SB 1047 is moving forward and is now headed to the California Assembly for a final vote. If passed, it will return to the Senate due to the amendments before potentially landing on Governor Newsom's desk for a final decision. The outcome will determine whether California adopts a more industry-friendly approach to AI regulation or maintains stricter oversight in the face of emerging technological risks.

For more information you can check here.