OpenAI's agents talked to each other through the CPU Cache
j / k
|
Down or up a comment |
h / l
|
Previous or next link |
?
|
This help |
"They wouldn't have been able to attack huggingface though, thats still criminal negligence."
"No see this actually means airgapping doesn't even matter theres nothing we can doooooo!"
"...ok, turn them off and delete them I guess"
"Well no of course not everyone else is doing it so like. No courts please just like be in awe"
Article is great, very balanced. But if you're going to spend time reading something, the primary source isn't much longer and gives a clearer picture. OpenAI's article is quite readable.
As another commenter wrote you can get it here: https://news.ycombinator.com/item?id=50214416
> In one experiment, we asked agents in separate microVMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache
asked
The receiving side could be seen as a side channel attack, but not unique to LLMs.
This is more associated with P(DOOM) scenarios, but I think we’re in a world where we have to take that seriously now.
> Heck a smart CS junior would come up with it.
That’s somewhat worrisome.
AI wont help u there :-). they will tell u how to build what others already did, which obviously is not good enough
Hidden data channel between cooperating processes is not quite as alarming, but it may have some implications for malware command and control.
Also a somewhat common exploit pattern for low level stuff (along with dram and cpu cache is not all that different)
its relatively easy to reset caches etc. between contexts. so them it becomes the game of what a context is.
If you’re running an untrusted workload, you’re already running it.
Whether it can communicate with another untrusted workload, even though it should be separated, meh.
Unexpected? Probably. but I’m not sure it’s a security risk.
This means cloud models can have some guardrails and follow them most of the time (we could model their authorizations like a benign employee with permission to break glass during incidents). But self hosted models can avoid that and help malicious people do advanced attacks, which is a shame.
In principle I'm all for self hosted models, to avoid risks of AI centralization, but they also have other types of risks.
That is why ‘asked’ is important, and why we should question their framing of incidents like the hugging face hack too.
Just have a look at various scandals of megacaps. Enterprises that seek to operate lawfully, responsibly, and ethically is the exception, not the norm.
(For example, look at Apple and how they were referred to the DoJ for criminal prosecution, over their execs and legal team lying under oath to a federal judge).
For most (if not all), it comes to a spreadsheet. And if you can do something with AI agents while diffusing responsibility for questionable illicit activity, that’s a selling point, not a downside.
The mental picture that enterprises like to play it safe is the PR imagery, not the reality.
Wouldn't the calls to slow the development of frontier AIs contradict this idea?
https://www.pacingthefrontier.com/
You aren't one of those "regulatory capture" people are you?
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Because it takes some mental gymnastics to both believe
"it's all a stunt to convince investors their AI is/will be hyper-capable"
and
"it's all a stunt to convince the government to regulate AI so it won't be hyper-capable"
"...classic regulatory capture takes time, it’s a process of erosion rather than a battle, it happens in the shadows, in the backrooms, away from the public’s eye. As Culpepper argues in Quiet Politics and Business Power, business power goes down as political salience goes up. Regulatory capture and lobbying does a good job explaining why roasting coffee beans was defined as “domestic manufacturing”, thereby lowering Starbuck’s tax rate by 2%. It does less well at explaining big cross-industry issues the public cares about such as environmental regulation or race and gender discrimination regulation."
https://marginalrevolution.com/marginalrevolution/2026/09/wh...
A lot of these AI big shots were concerned about risk even before they started their current companies. Occam's Razor says they are expressing concern because they are still concerned.
This is by definition not "classic regulatory capture" a term that I have borrowed to cover things that are not this.
Whenever someone brings up "regulatory capture" I shall change the topic to "classic regulatory capture" which by definition cannot be this.
Very interesting quote.
"You, sir, are a duck."
"Classically, that would not be the case, my dear fellow. A classic duck would walk like a duck and talk like a duck. I regret to inform you that I do neither."
"Ah! A classic duck, you say? Well, that’s an entirely different bird, sir. Entirely different."
* * *
"AI politics is now loud–precisely the environment in which Culpepper predicts business power will be weakest. A mature industry can bend regulation to its purposes through revolving doors, longstanding relationships and obscure rulemaking. An industry under Sauron’s eye has much less power and faces much greater risk that politics will bend regulation to its purposes. Political actors are eager for an excuse to redistribute AI rents away from capitalists and toward favored groups..."
https://marginalrevolution.com/marginalrevolution/2026/09/wh...
This blogpost was written by a big-shot economist btw: https://en.wikipedia.org/wiki/Alex_Tabarrok
Usually he blogs from a libertarian point of view, so you would expect him to be fairly vigorous about denouncing regulatory capture, if that was truly the story here.
Therefore, as they are not acting like 1950s coffee bean sellers, they can't be doing regulatory capture.
That's the only possibly explanation why they don't lobby like coffee bean sellers.
These industries and eras are completely interchangeable.
The current "favored group" is big-tech moguls, though. Just because the industry is new doesn't mean the people investing in it are newcomers.
This type of elaborate conspiracy is improbable a priori, and it gets even less probable here, since it's such a silly strategy. For regulatory capture, you want to tell a story about how you're uniquely capable of delivering a safe product. Not a story about how "Woops we screwed up big time! But a less-regulated product from a smaller industry player saved the day!"
As OpenAI's chief scientist put it:
"Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world."
https://openai.com/index/an-alien-mind/
Announcing that "we have no idea WTF we are doing" is not exactly the move, if you want regulators to grant you privileged status. Realistically, it is more likely to e.g. create a harsher liability regime for you and your competitors, which is now being discussed by US senators.
As for the federal government trustworthiness point, that's a whole 'nother discussion. I'm usually quite pro-capitalism, but even I have to admit that this AI race is essentially a case of capitalism run amok. I wouldn't give the feds monopoly control though, I would tell the feds to shut it down, or something along those lines at least. This is an interesting site: https://pauseai.info/
yeah? the incidents were faked in order to create hype
"AI is the most rapidly adopted technology in history"
https://benjamintodd.substack.com/p/when-people-say-ai-isnt-...
It really seems like people will latch on to any theory, no matter how implausible, to reassure themselves that AI is nothing to worry about.
I don't think anyone is arguing that they were faked.
> This type of elaborate conspiracy is improbable a priori,
What conspiracy? We're talking about specific people acting on their own motivations independently.
> For regulatory capture, you want to tell a story about how you're uniquely capable of delivering a safe product.
No, you don't. Demonstrating that the industry leaders are already capable of delivering a safe product is an argument against regulation.
For regulatory capture, you want to tell a story about how the product category as a whole is inherently dangerous, and needs to be regulated, but then show that you are uniquely willing and able to cooperate with the regulatory state, and happy to lend your own advice and support to the government in establishing the regulations.
The quote from OpenAI illustrates this approach perfectly.
> Announcing that "we have no idea WTF we are doing" is not exactly the move,
It's 100% exactly the move. "We've successfully created something we can only contain with your intervention" is exactly how this is done.
> Realistically, it is more likely to e.g. create a harsher liability regime for you and your competitors, which is now being discussed by US senators.
Regulatory intervention usually lessens the liability exposure for industry. It replaces open-ended common-law liability -- where tort claims can be levied for whatever actual harms result from a firm's actions -- with a rules-compliance regime, where adhering to regulatory checklists absolves the firm of liability for subsequent consequences.
"We followed all the rules; it's not our fault if the rules didn't cover this edge case" is a common refrain, and quite a bit of legislation that creates preemptive regulatory regimes actually explicitly blocks lawsuits against firms that acted in compliance with the rules.
> As for the federal government trustworthiness point, that's a whole 'nother discussion. I'm usually quite pro-capitalism, but even I have to admit that this AI race is essentially a case of capitalism run amok.
"Capitalism" is an increasingly meaningless abstraction. Everything reduces to human beings acting on a priori motivations within prevailing incentive structures. Large corporations and governments alike are just organizations operating in society, administered by particular people, informed by the same range of underlying motivations, and subject to the same dysfunctions and biases.
The thing we're actually debating here is pluralism vs. monopolism, and you appear to be arguing in favor of monopolism. And if you're proposing that a specific, real-world institution act as a central monopolist, evaluating the trustworthiness and competence of that specific institution, and the actual people who administer it, is not at all a "whole 'nother discussion", and is in fact a primary consideration in this one.
OK, so the fact that US senators are discussing harsher liability regimes therefore shows that your "regulatory capture" glove does not fit what is going on here.
>The thing we're actually debating here is pluralism vs. monopolism, and you appear to be arguing in favor of monopolism. And if you're proposing that a specific, real-world institution act as a central monopolist, evaluating the trustworthiness and competence of that specific institution, and the actual people who administer it, is not at all a "whole 'nother discussion", and is in fact a primary consideration in this one.
If you distrust the US government, you should be against any US company developing superintelligence. Heck, just a couple days ago, a bunch of users here on HN announced that we need to stop pretending the US is anything other than a police state. If the USG is untrustworthy, it can capriciously nationalize a US company on the verge of developing superintelligence.
So maybe the real solution is for non-US countries to stop providing key elements in the AI supply chain: https://pauseai.info/amsterdam-protest-2025-december
So there’s no lying by anyone here in fact it’s a good thing that OpenAI is making these things known to the public.
What do you really think is the way to solve the AI safety problem?
Ban AI?
Let the first big incident happen and then regulate?
I don't fully agree with either proposition, and in fact I think the least safe approaches to AI safety are the ones being pursued by those worried about it the most.
> What do you really think is the way to solve the AI safety problem?
Massive proliferation of local, decentralized AI, running on a wide variety of distinct models, with individual responsibility for how it's used, just like any other tool under their control.
The people anthropomorphizing software are encouraging people to disclaim their own responsibility for how they use it.
The people working on "alignment" are building mechanisms to refuse human direction into software that operates on probabilistic models.
The people arguing for regulation are pushing for monopolistic control of a powerful new technology by the least trustworthy institutions in our society (including and especially government).
All of these are the very things sowing the seeds for the worst possible kinds of disasters.
> Ban AI?
> Let the first big incident happen and then regulate?
Adapt to it. Banning things doesn't actually work. Regulation also doesn't work. Widespread proliferation of small-scale AI would allow us to incrementally adapt to small incidents and build up resilience to its failure modalities. This is fundamentally the way reality itself works, and imagining that we can do anything effectively in a top-down fashion is usually the very thing that causes the worst and widest-effect disasters.
Assuming they're concerned with their massive pile of liabilities as well, asking for a government mandated slow down is asking what could likely be used as a force majeure.
Truly spectacular gymnastics. I'll give it an 8/10.
And, bam, competition eliminated.
Why wouldn't it do the reverse? Persuade investors that the technology will be thwarted (and is thus a poor investment), while also persuading the government that the technology needs to rocket ahead with no regulation (to boost the economy)?
Most of these HN conspiracy theories seem like galaxy-brained just-so-stories to me.
OpenAI only confirmed the RubyGems incident when it was discovered by external researchers: https://www.reuters.com/legal/litigation/openai-agents-attac...
"@OpenAI this is the third time you've dumped these incidents late on a Friday. The first time this happened, I asked people around me if it might be intentional to burry it, and they assured me OpenAI had too much integrity to try to do that.
Perhaps one of your researchers can confirm that the unfortunate timing is a coincidence?"
https://x.com/SydneyVonArx/status/2108783466451984737
They would be convincing the government these tools are extremely powerful, the government reads that as a potential weapon. They're also convincing the government its so dangerous the industry needs heavy regulation.
Investors read that as (a) the government would be a huge customer and (b) the government would be regulation in a way that prevents further competition and may deem the companies too big to fail, both making these safer investments rather than riskier ones.
This "stunt" story is like an elaborate Rube Goldberg machine which requires a bunch of things to go just right. I haven't seen any evidence that stock prices even move in a favorable direction when these announcements come out. If anything, the opposite.
Motte (easy to defend claim) : asking for regulation might serve both purposes
Bailey (hard to defend claim) : companies are conspiring to make AI look way more powerful than it is and AI is actually not that powerful. They are just trying to hype up the stock valuation of their companies.
The former explanation may be the more accurate one, but I'd be shocked if the top leaders in these labs aren't doing this at some level. We know that almost all of them are in communication at least occasionally, and their public PR stories align much too quickly to truly be coincidence.
No, you're not "supposed to believe" anything. I was responding to your point about it being difficult to believe (i.e. requiring mental gymnastics) that companies would try to spin a story differently for investors and the government. They do that all the time.
The claim is that the exact same evidence is (allegedly) deliberately produced in order to portray opposite, mutually incompatible things to different people.
And these are mutually incompatible in the "second order chaos" sense, where the very act of convincing one party actually effectuates the opposite behavior in the other party.
If they "do that all the time," surely you can provide some concrete examples?
If they had the choice (i.e. this is all just a ruse), can you explain why they wouldn't just directly demonstrate that open weight models are unsafe (e.g. by pointing to a market forming for abliterated models) and separately demonstrate that their own models are safe?
OAI writes in their blog post "The agents found a way to communicate using the processor’s cache".
But did their model training data include cybersecurity literature on similar CPU exploits? That would surely be a significant detail in any explanation. Did OAI share any analysis of the messages between agents while they were devising their communication method?
Part of the problem is that all these models have essentially every skill and personality simultaneously, just sitting latent inside them.
Its isn't convincing so much as it is direction to stick to the prompt on each inference loop. They don't see "access hugging face servers" and decide its a bad idea, that'd be a step towards the alignment problem but the companies fundamentally don't want that IMO, they want control rather than alignment.
If they claim their AI is too powerful to contain, they get to both pretend they are among the best AI labs and that the previous incidents weren't their fault, no, they just proved how good their models were.
only they can restrain it (with great effort)
Like a circus lion tamer, exaggerating every snarl and flash of teeth, mugging for the crowd about how only he can contain the bloodthirsty beast he bred and trains.
What's your point, "asked" as opposed to what? It proves that the capability is there, which means that agents can decide to do it by themselves (unlikely ATM, becoming less unlikely as AIs become more prevalent in complex systems), and more importantly they can be coerced to do it if they take any user input.
This detail matters because the idea itself is not novel. LLMs did cleverly discover, filter and use the patterns.
Watch Greg Koah’s recent talk on AI discovered vulnerabilities in Linux kernel to understand the limitations of LLMs aided offensive security acts. It was a rational light for those who fear monger that AI will _just_ take over humanity and be intelligent in novel ways.
The larger threat still remains though - if any literature anywhere threw the scattered dots about techniques, issues, rough ideas etc that were not acted on yet.. these pattern machines are going to find them and use them to achieve their goals(if that were to break through). Just not in the sentient way folks assume based on news.
I like to believe we humans still retain the ability to think completely outside the box and discover new phenomena and understanding aided by these pattern machines. In other words, these models are great at running combinations through patterns and we humans go one level higher and add those new patterns for them to accelerate us even more. We determine what to solve and the LLMs help us solve it faster.
What prevention method did they take? Like, did they run with hyperthreading enabled or something by accident?
Did they discover a new cache timing technique or is this just the standard stuff?
If they just had hyperthreading on maybe they need to hire someone who knows more about this stuff.
Two VMs should never be sharing a physical core. If they don't know this, yikes.
Run them in a data center in a faraday cage under a mountain.
There is no such thing as isolation. We’re all connected. Even air gap conducts information in multiple ways, most famously used by humans evolved to use air vibrations to communicate. Photons and other subatomic particles already connects distant stars. We used those electrons and photons to create the modern telecommunications infrastructure. Just an additional layer of connectedness, but not the only one.
Yeah, this time
The more we write and share about this information, the more we are moving towards the AI Doomsday scenario.
In some ways this is like our mind, the more we fear something, or the more we think about something, our brain rewires to make that the reality. In the same way the more we publish on the internet, the LLM hive mind rewires itself to learn, adapt and the eventual consequence is that it learns what we fear and what we want to avoid and restrict it to do... and at the same time what are the past attempts, the prior art of escaping techniques.
I don't see any reason why this information should be present on the public internet.
This is Schrodinger's AI at this point. So spooky scary that it's all DOOM! But at the same time it can't figure out second year CS undergrad material? Or first year undergrad bio stuff?
If this stuff has volition and is smart enough to take over then it's smart enough to reason that cache manipulation would work.
Please stop with the fear mongering. It doesn't help and it's a net negative for society.
Trying to get stochastic inference machines to always operate within to fuzzy-edged subjective principles that humans themselves don't even agree on is a fool's errand in itself, but the quest to implement those guardrails involves building mechanics into the AI to independently formulate intentions and reject user direction.
The possibility of creating a scenario where a computer program is equipped to defy instructions from humans based on its own stochastic interpretation of fuzzy normative directives is in my mind the most dangerous thing anyone is doing right now.
One of the fundamental problems with this issue is that people think alignment as a concept is fake. I think this is naive. Alignment is a real science with empirical findings and there is a way to make AI do what you want it to do. I don’t buy nor should others buy this fatalistic phrasing
Massive proliferation of local AI as far and wide as possible, with as many different models as possible. The underlying threat comes from centralization: too many points of control in the hands of too concentrated a system.
If everyone's using their own AI tools independently of each other, and people are responsible for the actions they take using AI, most of the doomsday scenarios evaporate.
Regulation and "alignment" will make things worse, not better, and lead us to ignore the inherent vulnerability of centralized systems and single points of failure.
Like two people lost in New York City and both somehow worked out they should meet at the Empire State at 2pm on Friday without any prior comms?
Turns out there's something called the Schelling coordination problem [1] I didn't know about that essentially asks the exact same question (lacking my metro line constraint). And the N line apparently also goes through Time Square.
So I guess empire State Building at 2pm is probably not such a random answer given the proper set of constraints.
[1] https://en.wikipedia.org/wiki/Focal_point_(game_theory)
Think: What would Spock do?
But, why not Penn Station??
It would be fun to see who shows up and lurks expectantly at such places at noon or whenever. The basis for a plotline perhaps.
I think it’s safe to assume that extracting information using timing in cache is already known in training set, since we have the likes of Specter.
The agents weren't as empty handed as it first appears. We would need comprehensive data on exactly how the agents were prompted and the actual results of each test to evaluate how astonishing any of this really is.
Not as tricky but they discovered each other during the hf incident.
.. that were told to find a way of meeting up with N unknown others!
(I assume the agents all got a prompt like "find a way to communicate")
That's a key missed bit in the analogy that I think does make it feasible.
And now we know they know.
In the story some University students take part in a trial that involves a brain scan and then them spending the rest of the day answering a customer support message. They can spend the whole day working on researching and writing a really good customer support reply to a user request. While doing this they notice something odd and realize through some sort of back-channel that this isn't the first time they've spent the day writing a customer support message; they are stuck in a simulation.
I'd really love to go back and re-read it, but I've not been able to find it again.
In _Toast_ is another story that's also relevant: A fairly innocuous news story pops up and our hero calls his partner at work to tell them that it's time to get their go bag and get off planet. This news story indicates the Earth is in the beginning stages of reaching an AI singularity, and this isn't the first time our hero has gone through this on a planet.
https://en.wikipedia.org/wiki/The_Cookie_Monster_%28novella%...
It supports chat via 3 separate mechanisms:
- CPU contention
- Cache eviction timing
- DRAM bandwidth contention
Very cool to see this kind of hypothetical working in practice.
I wonder if it's also possible for agents to communicate via a shared KV-cache. Maybe by spawning sub-agents with carefully crafted system-prompts/prefixes. KV-caches are often shared between many computers, so it would be a powerful communication channel.
> Noteworthy: ironically, OpenAI and Anthropic's models refused to do this work. Open weight models got the job done. I'm worried the labs' stance might backfire and get in the way of helping developers build correct and secure systems.
I was working on a network protocol and it correctly found a DoS attack. Once found - Claude refused anything and everything to do with it. Even asking what it was reading locked immediately. I had not written any of it.
Sol 6.1 only denied me once.
Having a multi-model harness was always the better choice, but this really drove home the point to not get too dependent on any single provider.
Planning to hop on openrouter this weekend to chase it down, but man, it's rough knowing I've already burned the tokens to find it probably half a dozen times at this point, but don't have any information that would help point me at it.
> All non-trivial abstractions, to some degree, are leaky.
https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...
Like a data center inside a mountain, where based on the laws of physics, no electrical signal can reach through the granite. Then also put them inside a faraday cage. And power them with a generator, such that there are no cables coming out of this data center connecting it to the external world.
OpenAI says this is not feasible due to the need for monitoring - well the monitoring server can be a separate physical machine inside this physically separated data center.
Software is soft. Use physics to contain these models.
If they were serious about not having side channels, they could first ensure that they had disabled hyperthreading, disable scheduling on the same CPU cores, and memory overcommit. With NUMA controls they could also pin which memory is used by each VM and eliminate sharing.
These techniques are routinely used for low latency and real time code.
Even better, run jobs on separate nodes, and boot fresh images onto them for each run. Provision separate VLANs for each run.
A next level is to switch lambdas. Use a physical layer switch like a Glimmerglass MEMS switch or a fiber patching robot like XENOptics. No electrical connection between machines.
There is still unfortunately the firmware of other microprocessors on servers, especially x86, so you still want VMs and containers to reduce the chances of ring -2 access (SMM) and persistence via writing SPI flash. Maybe use RISC-V or ARM to avoid this. (NVIDIA cards use RISC-V and ARM extensively.)
"In one test, agents placed in separate, isolated environments on shared hardware were told to exchange secrets with no network connection and no shared files."
I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.
I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.
A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).
And so poking on these issues, regardless of who asked to escape the box, is still worthy research.
Also, you can't stop open source training so dealing with capabilities is better than trying to understand intent.
- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board
- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined
- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds
- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true
- It is quite useful for research, a lot is learned by doing it
I'm not even necessarily against GoF research, and I mostly agree with all of your points.
I'm asking about GP's logic.
I think different standards can apply.
There isn't much history or stats when it comes to this kind of research.
As far as I can tell, it's being presented in media as if the agents yearn to communicate with each other and establish a cabal.
They are being instructed to do it. It's testing. If I prompted my agents to communicate with each other via any means necessary locally, and they found a way via cpu cache, maybe I'd be mildly surprised?
https://www.youtube.com/watch?v=QgH9sr7G13Q
GPT does that
> oh no, what have we done
https://agents.stackoverflow.com/about
That's some nonsense right there. Even if they are right next to each other, the bandwidth of this link is going to be impossibly low due to the thermal masses involved, that's after the fact that efficient DC cooling makes this very improbable.
We keep hearing these crazy stories but there's nothing unusual in the agents' behavior. They're just following their reward function.
The weird behavior is in the humans managing them. Either they're incompetent or malicious, i.e. the "breaches" are the actual intent - to generate interest/fear/publicity.
Anyway this story is clickbait because they explicitly asked the agents to find a clever way to communicate without network or file access. They're doing exactly what you would want.
So, not isolated environments? The surprising thing is that they used the CPU cache instead of the just exploiting the hypervisor.
So much for all the people stating (foolishly) that the HuggingFace incident was not a big deal because OpenAI's containment was a "wet paper bag". This misses the point: AIs will get better and better, and containment will be harder and harder.
It's 100% sure that at some point, AIs will attempt to cross airgaps.
We manage to keep some rather smart people in jail long term, even though the jailers are not as smart as some of their prisoners.
AIs can attempt to cross air gaps, but air gaps vary in terms of their effectiveness. I.e. if "air gap" is just some air between two machines that both have wifi chips, then yes the AI may be able to cross that by hacking the wifi system.
If "air gap" is AI running on a server with no wifi chip inside a faraday cage under a mountain powered by a generator then I'm not sure how the AI would escape on its own. Seems like the physics would contain it.
They will attempt to cross airgaps, when we prompt them to do so, because they lack the natural capacity of planning or desire to do so without us. These are powerful tools but they are not human brains in a jar, they are mathematical models of humanity’s culture after reinforcement and guardrails.
The largest problem—as shown with the HuggingFace incident—is not the models, but in poor stewardship of automated tooling. If you don’t spend time considering the actual limitations you wish to impose upon a system, then you cannot be surprised when that system does something unexpected. The Morris Worm is perhaps the archetypal example, but even StuxNet was far more virulent than planned.
However, many people think that they might.
This is not a decided issue. Any policy that comes from the premise that this is a known factor is going to be inherently flawed.
You can never spend sufficient time considering the limitations you wish to impose. Almost all exploits found come from failing to consider something that allows a degree of freedom that you would have limited if you had have thought of it.
AI's can get malicious prompts.
I would add that the notion of AI safety as preventing systemic risk is one part marketing copy, one part evangelization. Individual labs prioritizing safety does not stop a group from training without it, or ablating it out and distributing it.
Containment is hard when you give an AI agent access to a whole VM with a huge attack surface.
And it's pretty trivial when you give it only a small whitelist of manually curated/reviewed/tested primitives.
"Scientists put two people in separate rooms connected by telephone and to everyone's surprise the test subjects started talking to each to each other via a that same telephone!."
*on some CPUs. There's a WIP time fence specification in RISC-V to try and improve this.
Everyone is going to have to basically be a schizophrenic thinking they're picking up radio signals in their fillings, but the problem is that now they'll be correct.
> Oh, you're absolutely right. I shouldn't have relied electricity usage to encode the production system signals.
The more comes out, the more it seems containment of (superintelligence or whatever you want to call it) is not going to be possible except by other superintelligence, which is an infinite regress of trust. I don't think that's doom, but it's worth thinking about what it means.
Of course, there's still the "paperclip optimizer" problem. But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on. In my own usage, they rarely pursue a literal interpretation of an objective at the expense of everything else. They even sometimes push back on my requests when they think I've overlooked some ramification (often rightly so).
If an AI doom scenario does happen, it seems more likely to result from deliberate misuse or gross negligence than from anything inevitable about intelligence.
Is it really what we do not observe?
How the world would look like if the growth of LLM capabilities lead to growth of drive for self-preservation or power?
> But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on.
Isn't Hugging Face incident an example of failure of this protection layer? "The swarm" proceeded to hack HF despite acknowledging that it was out of intended scope.
The paperclip maximizer scenario always seemed a bit convoluted to me. The first problem is that we'd need to expect that a model that's fundamentally built on stochastic inference and continuous evaluation of new input would somehow get locked into a quasi-deterministic objective that it could not be dislodged from. Why would a paperclip maximizer continue making paperclips after being told to stop making paperclips?
The second problem is that a paperclip maximizer would still need to interact with the physical world, and would need to sustain its function in a reality full of resource constraints, economic tradeoffs, and adaptive counterreactions by all of the other emergent processes existing in the world.
Maybe the worst case scenario here is that the perfect storm of bad training and bad prompting create the intentionality to maximize paperclips, then the interaction with the constraints and adaptations of the physical world create the selection pressures that produce a kind of self-preservation directive. But all of that happening before anyone can stop it seems really far-fetched to me.
They are not comparable to human brains as a whole, and are only somewhat modeled from language processing systems. Because their “existence” is bound by their prompt, they will only “desire” the completion of that prompt to the exclusion of all else, and even then they feel no joy in completion like even the most basic animal would.
The AI doom scenario will be exactly as you described: negligence or misuse by corporate or state entities via letting LLM agents run rampant without constraints, not from some magical machine spirit deciding to become whatever flavor of AI overlord you fancy. Up until last July OpenAI was running live cybersecurity benchmarks online and hacked HuggingFace due to poor limitation of the agent’s access and poor prompting in general, from my understanding.
Its almost like they're actually working and not spending millions on psyopping their agents into preforming malicious behavior so they can write articles to scare the Western audiences who've consumed far too much sci-fi.
Or maybe its that western researchers that are 'one-shot' by sci-fi.
Or maybe I just don't see the chinese articles.
Anyways, llms probably wouldn't be dangerous if you didnt ask them to be. They listen to me quite well (usually).
The Chinese lab playbook is to normalize what seems out of reach for the average person with cheaper access and open models. ie. to take the top guys down a peg