Can the US Prevent AGI from Being Stolen?
Securing AI weights from foreign adversaries would require a level of security never seen before.
.jpg)
In 1943, a new town appeared in the mountains of northern New Mexico. It didn’t show up on any maps. Families arrived by train under code names. Scientists were issued ration books and cover stories. Children went to school behind fences, while their parents worked on a secret project that would change the world. The town was Los Alamos. It was the home of the Manhattan Project.
Built almost overnight, Los Alamos became the center of America’s effort to develop a technology with unprecedented power: the atomic bomb. Its existence demanded total secrecy, centralized control, and infrastructure that had never existed before. Protecting the work happening inside required surveillance. Working groups were separated. Scientists and their families lived in government-run housing. All communication with the outside world was tightly monitored.
Today, calls for a “Manhattan Project for AI” echo from Washington, D.C., to Silicon Valley. The comparison isn’t just rhetorical — some experts believe advanced AI models could soon rival nuclear weapons in their strategic importance, with capabilities that may reshape the global balance of power and pose catastrophic risks if stolen. This proposal has sparked considerable debate — including on this platform — but most have focused on whether the U.S. should pursue it. Less explored are the more pragmatic questions: If the U.S. did decide to pursue an AI program to match the scale and impact of the Manhattan Project, what would that actually look like? What kind of security, infrastructure, and staffing would it require? Is it even feasible?
Security is a core concern facing a Manhattan Project for AI. Preventing leaks — intentional or unintentional — is a serious challenge for any AI research efforts. In a recent report by researchers from RAND, the authors lay out a system of five security levels (SL1 through SL5) for protecting advanced AI models. The lowest level, SL1, protects against hobby hackers as well as scattershot attacks from more advanced groups. The more focused, sophisticated, and resource-rich the attacker is, the higher the required security level.
Sella Nevo, lead author of the RAND report, said a Manhattan Project for AI would need the highest level, SL5 — capable of thwarting threats from the world’s top cyber-espionage groups.
These cyber-espionage organizations have upwards of 1,000 personnel: a rarefied collection of hackers, social engineers, saboteurs, surveillance gurus, logistics specialists, and human intelligence agents. They have access to government computing infrastructure, security facilities, legal protections, and money, which they channel into complex, multi-pronged, years-long operations.
For example, while the group’s hackers attempt to crack the AI Manhattan Project’s digital defenses, its spymasters could be recruiting agents from all levels of the org chart. Meanwhile, the logistics team might be finding ways to tamper with essential AI hardware and insert hidden, exploitable flaws, while so-called side-channel experts analyze the facility’s power usage and electromagnetic emissions to glean vital information about the design of the AI models inside. If all else fails, the attackers might even plan a last-ditch physical attack to infiltrate the facility.
To counter these threats, the AI Manhattan Project would need to draw on decades of cybersecurity and counter-espionage techniques. It would also have to invent defenses for entirely new attack vectors, such as those targeting AI’s specific vulnerabilities. To understand the security challenges that an AI Manhattan Project would face, we need to discuss how it’s built, who builds it, and how it’s deployed.
/odw-inline-subscribe-cta
The core component allowing an AI model to be run is its weights. Although files containing model weights can be huge (measuring in the range of terabytes), copying such files would be trivial for someone with unthrottled access to them.
The most straightforward way to secure these most sensitive details of an AI Manhattan Project would be by housing them within purpose-built, fortress-like data centers. According to Nevo, these could be designed after the Sensitive Compartmented Information Facilities (SCIFs) used by U.S. defense organizations when handling highly classified materials.
SCIFs are tightly sealed from the outside world. The buildings themselves tend to have thick, concrete walls — some are even buried underground — to prevent eavesdropping or side-channel attacks. Armed guards control who comes and goes. Computers are air-gapped from outside networks. Information is compartmentalized between various labs, with few individuals having access to multiple working areas. The sites have entire zones where personnel are forbidden from bringing in electronic devices: no smartphones, no USB drives, no cameras, not even fitness trackers.
Nevo explains that an AI Manhattan Project would likely require security beyond that found at most SCIFs. The facility would need its own servers with no connection to the outside world. The project would also need to verify that the building’s computer infrastructure was built using secure components. This would entail research into the supply chain and performing adversarial investigations into systems to ensure there were no back doors.
But the leakiest part of any security project is never the architecture — it’s the people inside.
At its peak, the Manhattan Project employed nearly 129,000 people — including researchers, administrators, military personnel, and construction workers. The FBI conducted rigorous background checks on each of them to ensure they didn’t have any criminal history or links to rival governments.
An AI Manhattan Project would require even more intensive screening. The standard government background check alone can take months, even years, to complete. During that time, investigators comb through an applicant’s criminal records and financial history. They might interview friends, acquaintances, and family members. Some applicants may have to take a polygraph test. It might prove impossible to successfully complete background checks on a large fraction of the people that an AI Manhattan Project would want to hire, given that nearly 50% of top-tier research talent is originally from China, according to some metrics.
Once they pass the polygraph, personnel would be continually monitored. Anne Neuberger, who was the NSA’s chief risk officer in the wake of the Edward Snowden leaks, told AI Frontiers: “We had to recognize that trusted insiders could pose real threats.” Under her guidance, the NSA implemented protocols that treated every individual as a security risk, or as Neuberger puts it, “extensive monitoring for anomalous activity by insider threat programs.”
After Snowden, for example, decisions involving sensitive data or high-risk tasks now require sign-off by two authorized individuals. They monitored personnel for abnormal behavior and other indications of malicious intent. They conducted independent audits to flag security protocol violations.
For many people working on the original 1940s Manhattan Project, this oversight extended even further. At Los Alamos, personnel weren’t allowed any communication with the outside world. They lived on-site in totally insular communities. They were restricted from talking about their work with anyone but their closest work colleagues — no office talk at the dinner table allowed — and required to report any contact they had with individuals outside their normal social and work circles.
But those constraints might not work in today’s world. During the 1940s, there were fewer powers racing to develop the bomb. Now, not only would the U.S. government be in a race for AI supremacy against China, it would be competing against well-heeled tech giants like Anthropic, Google DeepMind, OpenAI, DeepSeek, Baidu, and Alibaba for the most talented AI engineers. “If you tell top engineers they have to go live in rural North Dakota, they’re going to go work at a startup [instead],” said Dean Ball, a research fellow at George Mason University’s Mercatus Center.
An AI Manhattan Project, however, may not need full centralization. Nevo clarified that SL5 doesn’t require the whole organization to work in a SCIF. “That’s very, very far from what’s actually being suggested,” he said. “Most of the work done in the labs would not require direct read access to weights of the full frontier model.” SL5 protocols would only apply to the most sensitive components. While he acknowledged that most personnel would not be happy about the extreme requirements of isolation at this level, he reinforced that SL5 protocols would only affect small groups working directly on the frontier models.
AI and nuclear weapons share another trait: Protecting their secrets becomes a greater challenge after they’ve been deployed. “Right now, at least for AI systems,” said Nevo, “there's not a reliable way to connect something to the internet and know that you can’t exploit it to extract sensitive information, including the model weights.” For example, after a model has been deployed, its weights can be partially reconstructed using techniques like distillation, where a smaller model is trained to mimic a larger one using the larger model’s outputs. Researchers have also previously found ways to extract entire chunks of one of OpenAI’s models purely by querying the model over APIs.
Nevo explained that there aren’t currently any known, reliable ways to defend a deployed AI system’s weights against this sort of exploit. “With cryptographic systems,” Nevo explained, “we’ve had decades of trial and error. We know that you can sign or decrypt with a key 100 million times and not extract the key.” Developing these protections would be one of the biggest challenges of the AI Manhattan Project.
If successful, an AI Manhattan Project would give the U.S. unprecedented leverage over its rivals. Thus, from the rivals’ perspective, nearly any effort would be justified to infiltrate, disrupt, and plunder American research. To protect against these threats, the U.S. will have to combine existing cybersecurity and counter-espionage strategies with yet-to-be-invented AI-specific defenses to achieve a level of security that has never been seen previously in AI research.
This is a challenge not only for a potential future U.S. government-led project, but also for today’s leading U.S. tech companies, which have publicly declared their goal of developing AGI that can automate all cognitive tasks. Given how valuable a target this would be for attackers to steal, building such systems without adequate security is equivalent to handing this technology to U.S. adversaries on a silver platter. If we want the U.S. to retain a decisive technological advantage in AI as it becomes the target of increasingly well-resourced theft attempts, we need to begin tackling the challenges of AI security now.
AI is naturally prone to being tricked into behaving badly, but researchers are working hard to patch that weakness.
Rather than rushing toward catastrophe, the US and China should recognize their shared interest in avoiding an ASI race.
