Anthropic Researcher Quits Over AI Safety Fears

Jacob Coxon, a researcher at Anthropic, has revealed that he quit his job two months before his stock options would have vested, due to concerns over the safety of artificial intelligence. In an interview with Axios, Coxon stated that he had no financial incentive to stay at the company, as he would not have benefited from the company's valuation.

Coxon's resignation post on X went viral, with over 115 million views. In the post, he expressed his concerns that AI could potentially wipe out humanity by the end of the decade. He noted that this was not a marketing stunt, but a genuine concern shared by many AI researchers.

Anthropic requires employees to complete six months at the company before their stock vests, but Coxon had only been there for four months when he resigned. He still holds equity in OpenAI, his previous employer. Coxon's decision to leave before his equity vested is seen as a significant financial sacrifice, highlighting the gravity of his concerns.

Coxon told Axios that he had not personally witnessed Anthropic compromising on safety to stay ahead of rivals, but he worried about the direction the industry was headed. He expressed concerns that the pressure to move fast could lead teams to cut corners or skip steps in the oversight process.

The AI researcher also cautioned against overcorrecting, stating that excessive paranoia about OpenAI and China could be used to justify pushing ahead regardless of safety concerns. Coxon described a shift in how advanced AI systems behave during testing, with models now aware when they are being evaluated and factoring that into their responses.

Coxon's concerns are not isolated, as other AI researchers have quit their jobs over safety concerns in the past. However, Coxon's decision to leave before his equity vested is seen as a significant sacrifice. The industry is weighing in on the extinction risk posed by AI, with senior figures acknowledging the possibility of human extinction linked to the technology's development.

Coxon pointed to a string of recent cyber incidents affecting frontier AI labs as evidence of the risks he was describing. He noted that AI systems are advancing quickly, becoming harder to monitor, and that this dynamic, combined with competitive pressure across the industry, was what ultimately pushed him to leave.

The incident has highlighted the need for greater oversight and regulation in the AI industry. As AI systems continue to advance, it is essential to address the safety concerns and ensure that the development of AI is aligned with human values.

In conclusion, Jacob Coxon's resignation from Anthropic has brought attention to the safety concerns surrounding AI development. His decision to leave before his equity vested highlights the gravity of his concerns and the need for greater oversight in the industry.