>Yes, all of which might in principle be used to support similarly strong hopes, but there’s little energy for that.
The ongoing trillion+ dollar race is entirely premised on such hopes, it would seem to me. Even though, like you say, fears require just one thing to plausibly go wrong, they look to be thoroughly crowded out by hopes so far.
I'd say that there's a narrow optimistic circle of AI labs and investors in them, which isn't short on hopeful words, and the huge rest of the society where nihilistic pessimism of various sorts pervades. I mean, you seem plenty pessimistic yourself about the direction of the western civilization.
Then you don't seem to have understood it very well. It's a perfect, realized example of explicit warnings from the AI safety community for decades. It's a clear demonstration of the harms of non-explicit instrumental goals.
We're lucky on a lot of fronts here: that a foreign server (governmental, military, private) wasn't the target, that substantial bodily harm or financial loss was not a side effect. It's apparently going to take something like that for people with their head in the sand to understand the extent of the problem.
Some argue that AI is as dangerous as nuclear weapons. You and others disagree. If it is, we probably won't know until after it's too late.
People did know that nuclear weapons were dangerous when they were invented, and they became more dangerous until people took steps to decrease their threat. Did people worry too much about that threat? If so, how should those who believe in the threat of AI behave differently?
Not on AI but on hope and fear: looking back many of our fears didn't materialize, often in ways we couldn't possibly predict at the time. Looked at this way, both hope and fear are irrational stances because they take the future to be more predictable than it is. There's a third option: belief that things will work out in some way that you can't currently predict (while of course taking steps towards good outcomes at the same time). You might not know exactly how or when things will work out, but you move forward assuming it will. We tend to look down on belief post-enlightenment, but it's an underappreciated stance in life. It's a lot less fragile than hope and it also makes you more open to unexpected positive outcomes which you might've missed if you were monomaniacally focused on the outcome you were hoping for.
What’s the practical solution to Modern Worries for an individual? Worrying only about the next day where the hopes can balance the worries, or finding more (indirect or hidden) hope for longer terms?
Why say that "bad outcomes are rare"? The modal number of children for male humans is zero. These individuals die without leaving offspring. Evolution grades their score cards as FAIL. It seems like a pretty common bad outcome to die childless. The situation for your species seems even more dire - with 99% of species going extinct. By what metrics are bad outcomes rare?
The asymmetry in the first paragraph is doing all the work, and I don't think it holds. A good thing derailed by one failure is something you live through in real time. You were doing the many things, you felt each one that had to go right, and then you watched the one that didn't. It arrives already in pieces. A bad outcome that needed a dozen things to line up arrives as a finished event, and reasoning backward from an event stops at the first cause that looks sufficient. So memory fills with fragile-good cases and almost no many-cause bad ones, and the asymmetry looks obvious because half the ledger was never posted.
The evidence that this is a narration effect is that we had to build institutions to force the decomposition. NTSB, root-cause review, morbidity and mortality conferences, the Swiss cheese model - a whole profession whose output is "actually, eleven things had to align", whose permanent adversary is the assumption that the last error was the cause. If bad outcomes really did need only one thing to go wrong, that discipline would be redundant.
Which leaves your AI conclusion on better ground than the argument you gave it. Doom scenarios need a lot to line up at once - capability and misalignment and deployment and no correction - so they are what this most inflates. Raphaël's checklist is the mechanism in action: ticking the pieces off one at a time is how a long list of requirements comes to feel like accumulating certainty.
I would agree wholeheartedly, if only these once-outlandish AI scenarios dreamed up decades ago weren't coming true one after another, right in front of us.
AIs that can talk and think like humans, check. AIs that pursue goals and rewards by any means necessary, including lying and cheating, check. AIs that can code and hack better than humans, and slip their lab's containment, check. AIs that could take over the world, work in progress ?
I'm confused. In my understanding you expected or hoped for a trajectory pointing in a promising direction, not towards a Yudkowskyan scenario. The ChatGPT moment was a neutral point between the two trajectories. But the events since are core of the doom scenario. Admittedly the recent events do not strictly exclude an happy ending, but if so, they are twists and turns, while they are straightforward in the doom scenario.
>Yes, all of which might in principle be used to support similarly strong hopes, but there’s little energy for that.
The ongoing trillion+ dollar race is entirely premised on such hopes, it would seem to me. Even though, like you say, fears require just one thing to plausibly go wrong, they look to be thoroughly crowded out by hopes so far.
Yes, investment reveals hopes, but not so much words.
I'd say that there's a narrow optimistic circle of AI labs and investors in them, which isn't short on hopeful words, and the huge rest of the society where nihilistic pessimism of various sorts pervades. I mean, you seem plenty pessimistic yourself about the direction of the western civilization.
You should read / listen to Dwarkesh's summary of the OAI huggingface incident report
not abstract at all
predictable (and predicted) in advance
I did read it, but didn't learn much. I don't see much we can infer from it for the big issues everyone wants to talk about..
Then you don't seem to have understood it very well. It's a perfect, realized example of explicit warnings from the AI safety community for decades. It's a clear demonstration of the harms of non-explicit instrumental goals.
We're lucky on a lot of fronts here: that a foreign server (governmental, military, private) wasn't the target, that substantial bodily harm or financial loss was not a side effect. It's apparently going to take something like that for people with their head in the sand to understand the extent of the problem.
Some argue that AI is as dangerous as nuclear weapons. You and others disagree. If it is, we probably won't know until after it's too late.
People did know that nuclear weapons were dangerous when they were invented, and they became more dangerous until people took steps to decrease their threat. Did people worry too much about that threat? If so, how should those who believe in the threat of AI behave differently?
Not on AI but on hope and fear: looking back many of our fears didn't materialize, often in ways we couldn't possibly predict at the time. Looked at this way, both hope and fear are irrational stances because they take the future to be more predictable than it is. There's a third option: belief that things will work out in some way that you can't currently predict (while of course taking steps towards good outcomes at the same time). You might not know exactly how or when things will work out, but you move forward assuming it will. We tend to look down on belief post-enlightenment, but it's an underappreciated stance in life. It's a lot less fragile than hope and it also makes you more open to unexpected positive outcomes which you might've missed if you were monomaniacally focused on the outcome you were hoping for.
What’s the practical solution to Modern Worries for an individual? Worrying only about the next day where the hopes can balance the worries, or finding more (indirect or hidden) hope for longer terms?
Why say that "bad outcomes are rare"? The modal number of children for male humans is zero. These individuals die without leaving offspring. Evolution grades their score cards as FAIL. It seems like a pretty common bad outcome to die childless. The situation for your species seems even more dire - with 99% of species going extinct. By what metrics are bad outcomes rare?
The asymmetry in the first paragraph is doing all the work, and I don't think it holds. A good thing derailed by one failure is something you live through in real time. You were doing the many things, you felt each one that had to go right, and then you watched the one that didn't. It arrives already in pieces. A bad outcome that needed a dozen things to line up arrives as a finished event, and reasoning backward from an event stops at the first cause that looks sufficient. So memory fills with fragile-good cases and almost no many-cause bad ones, and the asymmetry looks obvious because half the ledger was never posted.
The evidence that this is a narration effect is that we had to build institutions to force the decomposition. NTSB, root-cause review, morbidity and mortality conferences, the Swiss cheese model - a whole profession whose output is "actually, eleven things had to align", whose permanent adversary is the assumption that the last error was the cause. If bad outcomes really did need only one thing to go wrong, that discipline would be redundant.
Which leaves your AI conclusion on better ground than the argument you gave it. Doom scenarios need a lot to line up at once - capability and misalignment and deployment and no correction - so they are what this most inflates. Raphaël's checklist is the mechanism in action: ticking the pieces off one at a time is how a long list of requirements comes to feel like accumulating certainty.
I would agree wholeheartedly, if only these once-outlandish AI scenarios dreamed up decades ago weren't coming true one after another, right in front of us.
AIs that can talk and think like humans, check. AIs that pursue goals and rewards by any means necessary, including lying and cheating, check. AIs that can code and hack better than humans, and slip their lab's containment, check. AIs that could take over the world, work in progress ?
I expected all those things, not news to me.
I'm confused. In my understanding you expected or hoped for a trajectory pointing in a promising direction, not towards a Yudkowskyan scenario. The ChatGPT moment was a neutral point between the two trajectories. But the events since are core of the doom scenario. Admittedly the recent events do not strictly exclude an happy ending, but if so, they are twists and turns, while they are straightforward in the doom scenario.