s/reduce/reduced. Our firm is making an emergency plan to keep the business going assuming all Internet-connected PCs may be down for months. We are putting some machines aside offline.
I don't think you are considering the ways in which AI helps defenders as well. Even if they take time to fix the code they can look at patterns of access and monitor vast volumes of traffic to act to block hacks before they can do serious damage. I don't think we fully know who will get the upper hand.
Either way, I don't see how regulating AI would help because the most serious -- and economically impactful -- hacks tend to come from overseas. It's better if our companies get hacked first by a teenager in Indiana looking to replace the webpage with a picture of a penis than Russian ransomware gangs or worse saved to deploy en masse during a geopolitical conflict.
Besides, the process of finding exploits to fix is essentially indistinguishable from trying to exploit them so we need to ensure that our software engineers retain the ability to locate these flaws.
Of course, we might try to regulate access to certain features but -- even if the hackers can't gain unauthorized access -- the problem is that the Chinese and other foreign models just aren't far enough behind for barriers the regulations impose on our cyber security research to be outweighed by any delay in the danger.
---
Could be that people still demand regulation irrationally but I think that misreads the media landscape. People will hear about hacks happening but they won't be able to tie them to particular AI assistants while those same companies will be doing everything they can to highlight all the ways they are working to help make you safer. That plus all the security researchers saying how useful they are in protecting us will likely limit the pressure as a result of cybersecurity issues.
I think the pressure to regulate AI is more likely to come from more prosaic cases where someone with mental illness works with AI to do something harmful (to themselves or others).
First, the most powerful models don't come from overseas, and that won't change soon. That mean that if OpenAI, Deepmind, and Anthropic all don't allow competitors to distill their next-gen models, the competitors won't advance nearly as fast either. Second, the density of exploits is far higher than patching extant insecure software can address, so it doesn't help to find a few exploits and patch them, since the attackers get to attempt to hack the new system, which has 80 exploitable parts instead of 100. And third, "the security researchers saying how useful they are in protecting us" don't seem to exist - everyone in cybersecurity I've heard from agrees that cyber is strongly offense dominant, and that won't change until we fundamentally change the way we code.
I can certainly see an argument for saying: AI vendors should limit the most powerful models to accounts physically present in the US (tho I think including the EU is in our interest). But that is different than a general regulation limiting who in the us has access which inevitably applies a lot of friction to our own use.
After all, you don't need the most powerful model. In a year the models from overseas will be as powerful as the us ones today even if our models move ahead. If those bugs aren't patched they get exploited. Patching those bugs requires access to the models that find them. If you gate that access it slows way down.
And not every last bug must be patched. Good security is multilayered so an attacker needs a way to breech each layer.
Regarding distillation, I'm sure that AI vendors already have plenty of incentive to try to shut that down. Also they can always start from the models they have already distilled.
This is big news. But I've heard little concern over something which is not as much of a threat, but more-surprising, and a worse indicator of social breakdown: Microsoft turned Microsoft Office, and Windows itself, into malware a year or 2 ago, and got away with it.
Any version of Microsoft Office that you purchased from 2013 onwards, up until the last Office before the subscription-only Microsoft 365, will be deleted from your computer if you keep it connected to the internet. My Office 2013, a legal copy which I paid for, was last month deleted from the laptop on which I'm writing this, and replaced with Office 365, a subscription service which costs more per year than my Office 2013 cost to purchase. This was done by Microsoft, I think through the Office auto-updater rather than thru Windows Update (probably because a Windows Update can be rolled back).
(At first, sometime in 2025, Microsoft just started removing features from my Office 2013. I noticed when an update removed the "show differences between two Powerpoint documents" feature, which is of course still present in Office 365.)
I expect that the long software license whose terms I probably didn't read had a clause in it, which Microsoft must have written in 2013, reserving the right for Microsoft to change the terms of the license at will. Or else granting Microsoft the right to revoke the license at any time in the future. This is only technically legal; shrink-wrap licenses like this, especially the kind that you can't read until after purchasing the product, should not hold up in court.
Windows 11 has a "feature" which will automatically encrypt drives other than C: with bitlocker if your computer has a TPM chip, without asking or telling you. I don't recall what triggers it. That makes your drives unreadable by anything other than Windows. If you lose your C drive, or otherwise lose your bitlocker keys--this usually happens because the user doesn't KNOW their drives have been encrypted, and so didn't back up the keys, or did something like reformat their boot drive with Linux--your data drives then become unreadable unless you pay Microsoft to decrypt them.
(Meanwhile, modern ransomware has started using the Bitlocker that Microsoft installed on your system to encrypt your hard drive. It's now a Windows security feature which makes you less secure.)
I don't know how many people Microsoft has already robbed in this way, but it's probably in the millions. I don't know why there isn't a class-action lawsuit against them. But there is not.
We have laws capable of stopping Microsoft from brazenly robbing their customers, yet somehow they are not put in operation. We are seeing the return of feudal justice, in which courts or appeals usually existed to redress wrongs, but which in practice were inaccessible to most people. A kind of tyranny created by deliberately dysfunctional social institutions. In feudal days, this dysfunction was open; everyone knew justice was selective. Nowadays, justice is made selective simply by being made too complicated and expensive.
"For many decades, we have known how to write pretty secure software. It takes a bit longer, and security considerations must be central to early design efforts, but it is possible. However, developers have usually been in too much of a rush to market to do this. So most software systems today are riddled with security holes."
While this is true, I think it underestimates the long term impact of Mythos and its peers.
Yes, the median software product is significantly less secure than a hypothetical counterfactual software product that was produced with our known security best-practices, and so if we were less in a rush and more willing to put more effort into security, a lot of these security holes would not have existed.
However, even the software products where {we spend a lot of effort on security and employ best practices} contains security holes, and Mythos is able to find and exploit those as well.
Examples of software products where humans have employed the state-of-the-art best practices and yet security holes still slipped through:
Anthropic has (understandably) redacted a lot of the details for security issues Mythos has found, but we know Mythos was able to find at least one security hole in a software product whose emphasis was on putting a lot of effort into security and employing the security best practices. I haven't found one good "canonical" article that describes it, but you can search for "Mythos FreeBSD" and "Mythos OpenBSD" to find a lot of different articles that all provide some coverage of the issue.
Anthropic also claims (but has redacted the proof) that Mythos has found security vulnerability in all major browsers and OSes. People may joke around about Microsoft Windows, but Microsoft does put significant effort into securing its OS. And I think it's widely accepted that the relevant development teams put significant effort into securing Linux, Chrome, Firefox, etc.
Not to mention that in Project Glasswing, Anthropic is reportedly collaborating with JP Morgan, because Mythos has apparently found security holes in the banking infrastructure as well, a target that has obviously been under significant scrutiny both by would-be attackers and defenders. And so you would expect that all the low-hanging-fruit security holes were already found and fixed.
It may be the case that Mythos and its peers are "so good" at finding security holes that even our smartest, best, most conscientious, etc. humans will be unable to develop software that resists its attacks. At which point, we may have to rely on AI to security-check our software for us, leading to an arms race.
That wasn't the central point of my message, but yes, if pressed on providing a prediction on that topic, I guess it's something like "conditional on us ending up in a situation where we cannot rely on software within the next year, we will continue to be unable to rely on it for at least 5 more years."
Mythos can supposedly break into all major operating systems, but Anthropic has so far managed to contain that ability: We haven't all decided to disconnect our computers from the Internet. And I'm open to the possibility that there will be no major disruptions that cause us all to have to flee from the Internet.
But IF it gets to that point, where it's no longer safe to be on the Internet, I do predict that that state of affairs will persist for at least a few years.
I think a sharp rise in AI-assisted attacks is highly plausible in the short run, but I'm not sure the long-term outcome will be entirely negative. It may finally force the software industry, along with other organizations that build and deploy software, to confront the significant security debt they have accumulated.
For years, especially in SaaS, the incentive has been to ship quickly, add features continuously, and worry about hardening later. Many products now contain complex integrations and rarely used features that expand the attack surface without adding much real user value. If AI makes vulnerability discovery and exploitation much cheaper, that model becomes much harder to sustain.
The key distinction, in my view, is between AI-assisted engineering and what people now call "vibe-coding." AI-generated code is not inherently bad. But enterprise software should never be deployed merely because it appears to work. If an application is customer-facing, internet-exposed, connected to sensitive data, internally business-critical, or integrated into enterprise systems, then AI-generated code should be treated as untrusted until it has passed serious review.
That means code review, security review, threat modeling, dependency checks, secrets scanning, permission review, abuse-case testing, observability review, rollback planning, and clear human ownership. Someone accountable should be able to explain the code, operate it, maintain it, and respond when it fails.
This applies to human-written code, too, of course. A lot of non-AI code is also rushed and insecure. But vibe-coded systems deserve extra scrutiny because developers and non-developers may not fully understand the implementation, edge cases, dependencies, or security implications.
So perhaps the coming wave of AI hacking will have a constructive side: it may slow down reckless feature production, reduce unnecessary SaaS complexity, and create demand for better security review tools and secure development practices. The danger is real, especially among the long tail of SaaS vendors and internal enterprise tools. But the response should not be "don't use AI to code." Don't deploy externally exposed or business-critical enterprise software that no accountable human has deeply reviewed, secured, and taken ownership of.
I've been looking forward to this for a while. Historically many companies have been lax about security but AI hacking tools – especially ones that can run locally – will change that.
We could also do a lot for security if we decriminalized certain forms of hacking. It's hard to get a politician to understand that such activities make us stronger, not weaker. An analogy is TSA metal detectors at airports: You want red-team agents trying their best to find gaps and sneak weapons through.
s/reduce/reduced. Our firm is making an emergency plan to keep the business going assuming all Internet-connected PCs may be down for months. We are putting some machines aside offline.
I don't think you are considering the ways in which AI helps defenders as well. Even if they take time to fix the code they can look at patterns of access and monitor vast volumes of traffic to act to block hacks before they can do serious damage. I don't think we fully know who will get the upper hand.
Either way, I don't see how regulating AI would help because the most serious -- and economically impactful -- hacks tend to come from overseas. It's better if our companies get hacked first by a teenager in Indiana looking to replace the webpage with a picture of a penis than Russian ransomware gangs or worse saved to deploy en masse during a geopolitical conflict.
Besides, the process of finding exploits to fix is essentially indistinguishable from trying to exploit them so we need to ensure that our software engineers retain the ability to locate these flaws.
Of course, we might try to regulate access to certain features but -- even if the hackers can't gain unauthorized access -- the problem is that the Chinese and other foreign models just aren't far enough behind for barriers the regulations impose on our cyber security research to be outweighed by any delay in the danger.
---
Could be that people still demand regulation irrationally but I think that misreads the media landscape. People will hear about hacks happening but they won't be able to tie them to particular AI assistants while those same companies will be doing everything they can to highlight all the ways they are working to help make you safer. That plus all the security researchers saying how useful they are in protecting us will likely limit the pressure as a result of cybersecurity issues.
I think the pressure to regulate AI is more likely to come from more prosaic cases where someone with mental illness works with AI to do something harmful (to themselves or others).
I think this is wrong in a few ways.
First, the most powerful models don't come from overseas, and that won't change soon. That mean that if OpenAI, Deepmind, and Anthropic all don't allow competitors to distill their next-gen models, the competitors won't advance nearly as fast either. Second, the density of exploits is far higher than patching extant insecure software can address, so it doesn't help to find a few exploits and patch them, since the attackers get to attempt to hack the new system, which has 80 exploitable parts instead of 100. And third, "the security researchers saying how useful they are in protecting us" don't seem to exist - everyone in cybersecurity I've heard from agrees that cyber is strongly offense dominant, and that won't change until we fundamentally change the way we code.
I can certainly see an argument for saying: AI vendors should limit the most powerful models to accounts physically present in the US (tho I think including the EU is in our interest). But that is different than a general regulation limiting who in the us has access which inevitably applies a lot of friction to our own use.
After all, you don't need the most powerful model. In a year the models from overseas will be as powerful as the us ones today even if our models move ahead. If those bugs aren't patched they get exploited. Patching those bugs requires access to the models that find them. If you gate that access it slows way down.
And not every last bug must be patched. Good security is multilayered so an attacker needs a way to breech each layer.
Regarding distillation, I'm sure that AI vendors already have plenty of incentive to try to shut that down. Also they can always start from the models they have already distilled.
Would you be willing to make a bet on this?
If so?
What terms would you agree to
This is big news. But I've heard little concern over something which is not as much of a threat, but more-surprising, and a worse indicator of social breakdown: Microsoft turned Microsoft Office, and Windows itself, into malware a year or 2 ago, and got away with it.
Any version of Microsoft Office that you purchased from 2013 onwards, up until the last Office before the subscription-only Microsoft 365, will be deleted from your computer if you keep it connected to the internet. My Office 2013, a legal copy which I paid for, was last month deleted from the laptop on which I'm writing this, and replaced with Office 365, a subscription service which costs more per year than my Office 2013 cost to purchase. This was done by Microsoft, I think through the Office auto-updater rather than thru Windows Update (probably because a Windows Update can be rolled back).
(At first, sometime in 2025, Microsoft just started removing features from my Office 2013. I noticed when an update removed the "show differences between two Powerpoint documents" feature, which is of course still present in Office 365.)
I expect that the long software license whose terms I probably didn't read had a clause in it, which Microsoft must have written in 2013, reserving the right for Microsoft to change the terms of the license at will. Or else granting Microsoft the right to revoke the license at any time in the future. This is only technically legal; shrink-wrap licenses like this, especially the kind that you can't read until after purchasing the product, should not hold up in court.
Windows 11 has a "feature" which will automatically encrypt drives other than C: with bitlocker if your computer has a TPM chip, without asking or telling you. I don't recall what triggers it. That makes your drives unreadable by anything other than Windows. If you lose your C drive, or otherwise lose your bitlocker keys--this usually happens because the user doesn't KNOW their drives have been encrypted, and so didn't back up the keys, or did something like reformat their boot drive with Linux--your data drives then become unreadable unless you pay Microsoft to decrypt them.
(Meanwhile, modern ransomware has started using the Bitlocker that Microsoft installed on your system to encrypt your hard drive. It's now a Windows security feature which makes you less secure.)
I don't know how many people Microsoft has already robbed in this way, but it's probably in the millions. I don't know why there isn't a class-action lawsuit against them. But there is not.
We have laws capable of stopping Microsoft from brazenly robbing their customers, yet somehow they are not put in operation. We are seeing the return of feudal justice, in which courts or appeals usually existed to redress wrongs, but which in practice were inaccessible to most people. A kind of tyranny created by deliberately dysfunctional social institutions. In feudal days, this dysfunction was open; everyone knew justice was selective. Nowadays, justice is made selective simply by being made too complicated and expensive.
"For many decades, we have known how to write pretty secure software. It takes a bit longer, and security considerations must be central to early design efforts, but it is possible. However, developers have usually been in too much of a rush to market to do this. So most software systems today are riddled with security holes."
While this is true, I think it underestimates the long term impact of Mythos and its peers.
Yes, the median software product is significantly less secure than a hypothetical counterfactual software product that was produced with our known security best-practices, and so if we were less in a rush and more willing to put more effort into security, a lot of these security holes would not have existed.
However, even the software products where {we spend a lot of effort on security and employ best practices} contains security holes, and Mythos is able to find and exploit those as well.
Examples of software products where humans have employed the state-of-the-art best practices and yet security holes still slipped through:
- https://www.heartbleed.com/
- https://spectreattack.com/
- https://www.imperialviolet.org/2014/02/22/applebug.html
Anthropic has (understandably) redacted a lot of the details for security issues Mythos has found, but we know Mythos was able to find at least one security hole in a software product whose emphasis was on putting a lot of effort into security and employing the security best practices. I haven't found one good "canonical" article that describes it, but you can search for "Mythos FreeBSD" and "Mythos OpenBSD" to find a lot of different articles that all provide some coverage of the issue.
Anthropic also claims (but has redacted the proof) that Mythos has found security vulnerability in all major browsers and OSes. People may joke around about Microsoft Windows, but Microsoft does put significant effort into securing its OS. And I think it's widely accepted that the relevant development teams put significant effort into securing Linux, Chrome, Firefox, etc.
Not to mention that in Project Glasswing, Anthropic is reportedly collaborating with JP Morgan, because Mythos has apparently found security holes in the banking infrastructure as well, a target that has obviously been under significant scrutiny both by would-be attackers and defenders. And so you would expect that all the low-hanging-fruit security holes were already found and fixed.
It may be the case that Mythos and its peers are "so good" at finding security holes that even our smartest, best, most conscientious, etc. humans will be unable to develop software that resists its attacks. At which point, we may have to rely on AI to security-check our software for us, leading to an arms race.
So your prediction is that our inability to rely on software will continue for much more than a few years?
That wasn't the central point of my message, but yes, if pressed on providing a prediction on that topic, I guess it's something like "conditional on us ending up in a situation where we cannot rely on software within the next year, we will continue to be unable to rely on it for at least 5 more years."
Mythos can supposedly break into all major operating systems, but Anthropic has so far managed to contain that ability: We haven't all decided to disconnect our computers from the Internet. And I'm open to the possibility that there will be no major disruptions that cause us all to have to flee from the Internet.
But IF it gets to that point, where it's no longer safe to be on the Internet, I do predict that that state of affairs will persist for at least a few years.
My two cents:
I think a sharp rise in AI-assisted attacks is highly plausible in the short run, but I'm not sure the long-term outcome will be entirely negative. It may finally force the software industry, along with other organizations that build and deploy software, to confront the significant security debt they have accumulated.
For years, especially in SaaS, the incentive has been to ship quickly, add features continuously, and worry about hardening later. Many products now contain complex integrations and rarely used features that expand the attack surface without adding much real user value. If AI makes vulnerability discovery and exploitation much cheaper, that model becomes much harder to sustain.
The key distinction, in my view, is between AI-assisted engineering and what people now call "vibe-coding." AI-generated code is not inherently bad. But enterprise software should never be deployed merely because it appears to work. If an application is customer-facing, internet-exposed, connected to sensitive data, internally business-critical, or integrated into enterprise systems, then AI-generated code should be treated as untrusted until it has passed serious review.
That means code review, security review, threat modeling, dependency checks, secrets scanning, permission review, abuse-case testing, observability review, rollback planning, and clear human ownership. Someone accountable should be able to explain the code, operate it, maintain it, and respond when it fails.
This applies to human-written code, too, of course. A lot of non-AI code is also rushed and insecure. But vibe-coded systems deserve extra scrutiny because developers and non-developers may not fully understand the implementation, edge cases, dependencies, or security implications.
So perhaps the coming wave of AI hacking will have a constructive side: it may slow down reckless feature production, reduce unnecessary SaaS complexity, and create demand for better security review tools and secure development practices. The danger is real, especially among the long tail of SaaS vendors and internal enterprise tools. But the response should not be "don't use AI to code." Don't deploy externally exposed or business-critical enterprise software that no accountable human has deeply reviewed, secured, and taken ownership of.
I've been looking forward to this for a while. Historically many companies have been lax about security but AI hacking tools – especially ones that can run locally – will change that.
We could also do a lot for security if we decriminalized certain forms of hacking. It's hard to get a politician to understand that such activities make us stronger, not weaker. An analogy is TSA metal detectors at airports: You want red-team agents trying their best to find gaps and sneak weapons through.