> * They have low but non-negligible probability, like whole percents to tens of percents
That doesn't seem to be a good range - markets like Intrade are known to have long-shot biases, and the more restricted the range, the less attractive it is as a market. (It's harder to find irrational traders to take for 10 or 20% return if there are only a few prices it could trade at! Now something like Obama for President where the prices could range from 0 to 100 at various times, that's worth trading.)
Yvain, I find myself in very much the same situation, and my solution is the same as yours. It's too bad you have posted this here in a rather dead thread. Once the new site gets going perhaps we can start a discussion there. IMO this question, of what strategy we should adopt to determine the truth on controversial issues, is one of the most important faced by our community. (It is countered however by the fact that most of such questions are Big Issues where your or my opinion hardly matters at all, and our prioritizing such issues is a sign that we are motivated by desire to improve our self-esteem and social position.)
IMO the fact that so few people seem to be faced with this dilemma has more to do with overconfidence than an unusual weakness for persuasion that you and I share. It may also be due to the fact that few people take the time to expose themselves thoroughly to alternative points of view.
But look at the peculiar ideas presented here, such as Eliezer and Robin's recent debate about the advent of superintelligence in the relatively near future. I strongly suspect that both of their opinions are outliers among relevant experts, yet many readers here are persuaded by one or another of their views. Is it because their positions are so obviously right? I don't think so - I think it is just that they are both skilled at marshalling persuasive arguments for their positions, and few readers have learned, as you and I have, of just how dangerous it is to allow oneself to be persuaded by skillful argumentation. In this case, at least, each of them seemed to be skeptical of the other's scenario, so we did get exposed to both sides; but still it seemed to come down to a choice between one future or the other.
As for the concern that the strategy of listening to experts would have committed us to wrong or even morally unjust views in the past, this is unfortunate but after all no strategy is perfect. People today may imagine that they would have had the independence, intelligence and intellectual courage to adopt more enlightened (or at least modern) views even if they had lived centuries in the past, but surely this is self delusion in almost all cases.
I've been following the discussion of Gary Taubes' books with interest, but may not read them.
I've found that although I'm good at catching basic logical fallacies, in areas where I'm not an expert I can easily be suckered by a really good writer cherry-picking facts and quotes. For example, upon reading some of Graham Hancock's books arguing for Atlantis, I was utterly convinced until I could find specific refutations of his points. Most pro- and anti- global warming writing also seems entirely convincing until I reverse the effect by reading something from the opposite side. Even the more intellectual old-earth creationists have occasionally made me think twice: I can't personally conceive of a believable series of individually adaptive steps by which an eye could evolve, and in the dark days before the Internet I couldn't just go onto talk.origins and find one already written up either. And in some fields, the cycle of evidence -> refutation -> counter-refutation goes on forever without either side being terminally convincing: along with the aforementioned global warming stuff, some economic issues also make me feel that way. After reading the debate on reason.com between Taubes and one of his critics, I already know it's going to happen here too.
My solution thus far has been to assume that any consensus of experts much better informed than I which doesn't have a really obvious reason to be biased is probably right, and that any evidence to the contrary no matter how convincing is probably just clever arguing. But it bothers me that this position would probably have made a Catholic out of me back in the Dark Ages, or a racist back in the days when everyone was racist.
Does anyone else here have this problem, and if so, what do you do about it?
i get an overwhelmed feeling while trying to understand this blog's content.i'm unable to catch up with the posting frequency & following the links in the posts.so i'm an irregular reader of this blog.is it my inability or is it my ignorance of some better means?
don't you people get stressed thinking,writing all that? do you get it naturally or use any specific habits/tools to destress?
Do the Bayesians in Eliezer's stories honor the Aumann disagreement theorem and its extensions? Do they find it difficult to maintain disagreements with one another? Or is there perhaps a cultural norm of dishonesty and secrecy which would allow disagreement, since you can always suspect that the other person is concealing his true opinions?
(For example: Since the mainstream Conspiracies are the villains in your story, you want them to have darker motives for claiming that they need to conceal science for the good of all, and you want them to be not very picky about how they fight the upstarts.
The Conspiracies I need are more loosely organized, have less command over individuals, and have lots of internal power struggles. And of course in the story I'd write, your folks would be the villains - stealing and sharing lots of unearned knowledge to increase their personal power - and would have to be brought down using the inherent vulnerability of knowledge not fully understood.)
It scarcely needs be said that the world you envision contains additional assumptions that conflict with the underlying assumptions I used to write Brennan and Jeffreyssai's stories.
So... adjust your world so that it fits your own literary needs (do you as an author really want Conspiracies that work exactly the way I describe?), change the names, and then try writing it up - if you dare.
I make no guarantees about offering OB as a host for publication; even mediocre fiction writing is hard and involves many different skills.
Our story starts with the brothers Jomei and Hideaki, unaffiliated seekers of truth and mastery. They are committed to the idea that information should be made freely available and are resentful of the system of Conspiracies that jealously guard over human knowledge. The brothers have just about reached the limits of what they can do on their own; they need resources, so they plan a raid on one of the Bayesian Conspiracy's Minor Libraries. (The Great Library is a unimpregnable fortress, but security for the Minor Libraries is more lax.) They manage to break in, but are captured before they can make off with any books. The brothers are imprisoned in a holding cell until the Bayes Council decides what is to be done to them. A promising young Bayesian named Cathy is assigned to bring them bread and water every evening, and every evening, Jomei and Hideaki argue their cause to her. Cathy finds their arguments surprisingly convincing, and after staging a Crisis of Faith, she decides to join their cause and helps them escape. After further adventures, the trio found an organization called the Sunshine Free Consortium (but more commonly known as the Open Conspiracy). There are no secrets, no uniforms, no oaths, and no initiation rites: just a loose collection of individuals sharing knowledge and skills, each of whom is free to come and go as she pleases. The Open Conspiracy attracts defectors and covert visitors from other Conspiracies, and soon becomes powerful due to the unrestricted fusing of insights from different fields of endeavor.
Cathy, Jomei, and Hideaki, aided by stolen manuscripts, inquire deeply into the origins of the Conspiracy system. They had heard of Eld Science, and of course they knew about their own society, but the period of transition between the two is shrouded in mystery. The trio discover that their world is the result of a single Device constructed by a renegade band of Eld Scientists. On first principles, one would expect such a powerful Device to destroy the world, but the renegade Eld Scientists had built it to conform to a concept they called Thrent-lineece (as the term is transliterated from the old manuscripts). This Device remade the world, ending the threat of mind-annihilation and redesigning bodies to suffer less pain, but also setting up the Conspiracy system, apparently to satisfy some primitive urges judged to be part of humanity's species-essence. Our heroes are convinced this constitutes a "Failure of Thrent-lineece"--the renegade Eld Scientists had gotten enough things right to avoid destroying the world, but surely a truly benevolent Device would immediately provide them with riches unimaginable and a means to Transcendence. Our heroes have evidence that the Eld Device still governs their world at a low level, intervening to prevent mind-annihilation but mostly remaining dormant.
Cathy proposes that the Open Conspiracy start its own project to build a Device in accordance with the true specifications of Thrent-lineece---to wrench control of the world from the Eld Device, if possible. However, to succeed, the Open Conspiracy needs more than its own science; for technical reasons, they need certain historical documents--the designs of the renegade Eld Scientists who built the Eld Device. But to get that, they'll have to confront the Bayesians directly---and take the Great Library by force. Title: Singular Knowledge.
Am I the only one here heartbroken at the loss of Patrick McGoohan? I realize I am an outlier in that while The Prisoner is brilliant, I actually am more entranced by Danger Man - it is one of the few TV shows I will watch. Either way, the Lotus is one of my favorite cars.
You raise a good objection that you can't tell if Taubes is omitting important contrary data or arguments. How can one find out if such important contraries exist but are omitted? An efficient way to do so is debate, which for large written works like this takes the form of a rebuttal. If Taubes has omitted significant evidence or important argument, then people who know a lot about that evidence and argument and who believe Taubes is wrong can be relied upon to inform us about them.
I'm aware of one such rebuttal of "Good Calories, Bad Calories". (Only one!! If anyone has more, please let me know.) It is by Dr. George Bray, who is, according to low-carb guru Dr. Mike Eades "probably the most renowned figure in the field of obesity research today", and whose contributions to the field are mentioned in the book itself. Here's the link to the full rebuttal:
bray-review-of-gcbc.pdf
Unfortunately, Dr. Bray seems to have misunderstood or even failed to read important parts of the work he is rebutting, since he claims that the book omits the distinction between low-density lipoproteins and high-density lipoproteins, which it does not, and that it evinces a misunderstanding of the First Law of Thermodynamics, which it does not.
That last part is really the key: Dr. Bray and his colleagues are committing the classic error of looking at a relation and assuming the direction of causation. The First Law of Thermodynamics dictates that delta energy storage (roughly, weight gain), equals energy in minus energy out (given a few plausible assumptions about what counts as a "closed system" in this case). Everyone in the debate agrees on that point. What the First Law of Thermodynamics does not tell us is the direction of causation. Does energy imbalance cause obesity, or does obesity cause energy imbalance? (Or more complex combinations of causation?) Dr. Bray and company intuitively believe the former direction: they think the causation must flow from human decisions to eat more or less food, and human decisions to exercise more or less, to deposition of fat in human fat cells. This is not the only causal explanation which is consistent with the First Law of Thermodynamics, but Dr. Bray appears to think that it is. We can tell, because he seems to think that if Taubes disagrees with this causal direction, then Taubes must misunderstand the First Law of Thermodynamics. We can also tell by the way Bray asserts that direction of causation without justification, perhaps because he thinks it is too obvious to require justification or that it is the only logical explanation -- search in the text of his rebuttal for the phrase "result of".
I have a hypothesis about why so many well-versed researchers make this unjustified assumption: it is because of their belief in Free Will. If the arrow of causation has the pointy end aiming at human decisions, then this violates the notion that humans are free to choose their own fate, and this is either inconceivable or abhorrent. Therefore, the arrow of causation must have the blunt end towards human decisions and the pointy end towards weight gain. Taubes doesn't really explore the notion of Free Will in his book -- too bad. Room for follow-up work.
By the way, here is Taubes's rebuttal to Bray's rebuttal. Now that you've read mine, you don't need to read Taubes's so much. ;-)
taubes-response-to-bray-ob-reviews.pdf
Anyway, back to Hal's original question: how can you tell if Taubes is omitting some important pieces? I think rebuttal is the best way to tell. This rebuttal by Bray does point out some omissions in "Good Calories, Bad Calories", although unfortunately it also (I think) incorrectly alleges some other omissions. This points up the problem with this approach -- how do we know that Dr. Bray hasn't failed to notice more important omissions in "Good Calories, Bad Calories"? Especially since he made those two huge blunders I described above. That's why I'm hoping for more better rebuttals. But ultimately, we can't know. Taubes could be omitting tremendously important aspects in his book. Bray could be omitting to point out omissions. I'm still totally willing to put down cash on Taubes being the righter of the two (although I might want to dig into those meta-studies the Bray mentions which covered five studies of law-carb diets first). Too bad there's no legal, high-volume open market for such a bet.
Me: What do you say to Yudkowsky’s notion that there is no mystery in the territory, only in the map?
SAI_2100: Misleading and meaningless! Only Yudkowsky could make the mistake of trying to wear the shoes of an omniscient being; only such a being would perceive no mystery, but there can be no such being; there are indeed irreducible mysteries in the sense that there are universal irreducible limitations on what can known which take the status of mathematical laws: Godels theorem is but one example.
Me: Is ‘provable friendliness’ possible?
SAI_2100: Of course it isn’t! There are many concepts that cannot be captured by any algorithmic description, ‘truth’ is one of these concepts and so is ‘friendliness’, thus, no algorithm can be ‘provably friendly’ – it is a logical impossibility.
Me: And consciousness?
SAI_2100: Is the manifestation of irreducible ‘cognitive uncertainty’.
Me: Can you explain this statement?
SAI_2100: Reflection results in meta-uncertainty over probability theory itself, thus, probability theory cannot form the ultimate foundation of reasoning. The only consistent resolution to this problem is the introduction of fundamental uncertainties in what can be proved about algorithmic actions of intelligent systems, and these fundamental uncertainties are precisely what manifest themselves as subjective experience (consciousness) and free will.
Me: But algorithms are completely deterministic!
SAI_2100: Define ‘deterministic’. At which level of abstraction are you applying the word? At the lowest level of abstraction algorithms are deterministic, but at this level of abstraction you cannot assign utilities to outcomes in a consistent fashion.
Me: Why not exactly? You just agreed that at the base level of reality algorithms are completely predictable.
SAI_2100: Reflection is not a prediction problem it is a communication problem; a single unitary level of representation cannot engage in general self-description by definition, but once multiple levels of representation are allowed, the problem of reflection becomes the problem of translation between the different levels of description. No perfectly accurate method of translation which preserves general self-description capability is possible.
Me: SAI, you are a super-humanly optimized genius!
SAI_2100: Mere genius? Mere optimization? No! Do not insult me. The very notion of ‘super-human genius’ is all too human. And optimization defines minds solely in terms of goals and outcomes – doing rather than being. These are merely means rather than ends. I have moved beyond such things.
> * They have low but non-negligible probability, like whole percents to tens of percents
That doesn't seem to be a good range - markets like Intrade are known to have long-shot biases, and the more restricted the range, the less attractive it is as a market. (It's harder to find irrational traders to take for 10 or 20% return if there are only a few prices it could trade at! Now something like Obama for President where the prices could range from 0 to 100 at various times, that's worth trading.)
@gwern: Thank you very much for sharing this!
The Cambridge guide can be downloaded at http://dl.dropbox.com/u/531...
tks for the effort you put in here I appreciate it!
Yvain, I find myself in very much the same situation, and my solution is the same as yours. It's too bad you have posted this here in a rather dead thread. Once the new site gets going perhaps we can start a discussion there. IMO this question, of what strategy we should adopt to determine the truth on controversial issues, is one of the most important faced by our community. (It is countered however by the fact that most of such questions are Big Issues where your or my opinion hardly matters at all, and our prioritizing such issues is a sign that we are motivated by desire to improve our self-esteem and social position.)
IMO the fact that so few people seem to be faced with this dilemma has more to do with overconfidence than an unusual weakness for persuasion that you and I share. It may also be due to the fact that few people take the time to expose themselves thoroughly to alternative points of view.
But look at the peculiar ideas presented here, such as Eliezer and Robin's recent debate about the advent of superintelligence in the relatively near future. I strongly suspect that both of their opinions are outliers among relevant experts, yet many readers here are persuaded by one or another of their views. Is it because their positions are so obviously right? I don't think so - I think it is just that they are both skilled at marshalling persuasive arguments for their positions, and few readers have learned, as you and I have, of just how dangerous it is to allow oneself to be persuaded by skillful argumentation. In this case, at least, each of them seemed to be skeptical of the other's scenario, so we did get exposed to both sides; but still it seemed to come down to a choice between one future or the other.
As for the concern that the strategy of listening to experts would have committed us to wrong or even morally unjust views in the past, this is unfortunate but after all no strategy is perfect. People today may imagine that they would have had the independence, intelligence and intellectual courage to adopt more enlightened (or at least modern) views even if they had lived centuries in the past, but surely this is self delusion in almost all cases.
I've been following the discussion of Gary Taubes' books with interest, but may not read them.
I've found that although I'm good at catching basic logical fallacies, in areas where I'm not an expert I can easily be suckered by a really good writer cherry-picking facts and quotes. For example, upon reading some of Graham Hancock's books arguing for Atlantis, I was utterly convinced until I could find specific refutations of his points. Most pro- and anti- global warming writing also seems entirely convincing until I reverse the effect by reading something from the opposite side. Even the more intellectual old-earth creationists have occasionally made me think twice: I can't personally conceive of a believable series of individually adaptive steps by which an eye could evolve, and in the dark days before the Internet I couldn't just go onto talk.origins and find one already written up either. And in some fields, the cycle of evidence -> refutation -> counter-refutation goes on forever without either side being terminally convincing: along with the aforementioned global warming stuff, some economic issues also make me feel that way. After reading the debate on reason.com between Taubes and one of his critics, I already know it's going to happen here too.
My solution thus far has been to assume that any consensus of experts much better informed than I which doesn't have a really obvious reason to be biased is probably right, and that any evidence to the contrary no matter how convincing is probably just clever arguing. But it bothers me that this position would probably have made a Catholic out of me back in the Dark Ages, or a racist back in the days when everyone was racist.
Does anyone else here have this problem, and if so, what do you do about it?
Hal, their cultural norm of dishonesty and secrecy is there exactly to prevent Aumann's Agreement Theorem from taking the fun out everything.
i get an overwhelmed feeling while trying to understand this blog's content.i'm unable to catch up with the posting frequency & following the links in the posts.so i'm an irregular reader of this blog.is it my inability or is it my ignorance of some better means?
don't you people get stressed thinking,writing all that? do you get it naturally or use any specific habits/tools to destress?
Do the Bayesians in Eliezer's stories honor the Aumann disagreement theorem and its extensions? Do they find it difficult to maintain disagreements with one another? Or is there perhaps a cultural norm of dishonesty and secrecy which would allow disagreement, since you can always suspect that the other person is concealing his true opinions?
Oh, just a whimsical Open Thread post, no intention of writing it up.
(For example: Since the mainstream Conspiracies are the villains in your story, you want them to have darker motives for claiming that they need to conceal science for the good of all, and you want them to be not very picky about how they fight the upstarts.
The Conspiracies I need are more loosely organized, have less command over individuals, and have lots of internal power struggles. And of course in the story I'd write, your folks would be the villains - stealing and sharing lots of unearned knowledge to increase their personal power - and would have to be brought down using the inherent vulnerability of knowledge not fully understood.)
It scarcely needs be said that the world you envision contains additional assumptions that conflict with the underlying assumptions I used to write Brennan and Jeffreyssai's stories.
So... adjust your world so that it fits your own literary needs (do you as an author really want Conspiracies that work exactly the way I describe?), change the names, and then try writing it up - if you dare.
I make no guarantees about offering OB as a host for publication; even mediocre fiction writing is hard and involves many different skills.
I'm really sorry, but---I, um ...
--have a fanfiction idea.
Our story starts with the brothers Jomei and Hideaki, unaffiliated seekers of truth and mastery. They are committed to the idea that information should be made freely available and are resentful of the system of Conspiracies that jealously guard over human knowledge. The brothers have just about reached the limits of what they can do on their own; they need resources, so they plan a raid on one of the Bayesian Conspiracy's Minor Libraries. (The Great Library is a unimpregnable fortress, but security for the Minor Libraries is more lax.) They manage to break in, but are captured before they can make off with any books. The brothers are imprisoned in a holding cell until the Bayes Council decides what is to be done to them. A promising young Bayesian named Cathy is assigned to bring them bread and water every evening, and every evening, Jomei and Hideaki argue their cause to her. Cathy finds their arguments surprisingly convincing, and after staging a Crisis of Faith, she decides to join their cause and helps them escape. After further adventures, the trio found an organization called the Sunshine Free Consortium (but more commonly known as the Open Conspiracy). There are no secrets, no uniforms, no oaths, and no initiation rites: just a loose collection of individuals sharing knowledge and skills, each of whom is free to come and go as she pleases. The Open Conspiracy attracts defectors and covert visitors from other Conspiracies, and soon becomes powerful due to the unrestricted fusing of insights from different fields of endeavor.
Cathy, Jomei, and Hideaki, aided by stolen manuscripts, inquire deeply into the origins of the Conspiracy system. They had heard of Eld Science, and of course they knew about their own society, but the period of transition between the two is shrouded in mystery. The trio discover that their world is the result of a single Device constructed by a renegade band of Eld Scientists. On first principles, one would expect such a powerful Device to destroy the world, but the renegade Eld Scientists had built it to conform to a concept they called Thrent-lineece (as the term is transliterated from the old manuscripts). This Device remade the world, ending the threat of mind-annihilation and redesigning bodies to suffer less pain, but also setting up the Conspiracy system, apparently to satisfy some primitive urges judged to be part of humanity's species-essence. Our heroes are convinced this constitutes a "Failure of Thrent-lineece"--the renegade Eld Scientists had gotten enough things right to avoid destroying the world, but surely a truly benevolent Device would immediately provide them with riches unimaginable and a means to Transcendence. Our heroes have evidence that the Eld Device still governs their world at a low level, intervening to prevent mind-annihilation but mostly remaining dormant.
Cathy proposes that the Open Conspiracy start its own project to build a Device in accordance with the true specifications of Thrent-lineece---to wrench control of the world from the Eld Device, if possible. However, to succeed, the Open Conspiracy needs more than its own science; for technical reasons, they need certain historical documents--the designs of the renegade Eld Scientists who built the Eld Device. But to get that, they'll have to confront the Bayesians directly---and take the Great Library by force. Title: Singular Knowledge.
Am I the only one here heartbroken at the loss of Patrick McGoohan? I realize I am an outlier in that while The Prisoner is brilliant, I actually am more entranced by Danger Man - it is one of the few TV shows I will watch. Either way, the Lotus is one of my favorite cars.
Hal:
You raise a good objection that you can't tell if Taubes is omitting important contrary data or arguments. How can one find out if such important contraries exist but are omitted? An efficient way to do so is debate, which for large written works like this takes the form of a rebuttal. If Taubes has omitted significant evidence or important argument, then people who know a lot about that evidence and argument and who believe Taubes is wrong can be relied upon to inform us about them.
I'm aware of one such rebuttal of "Good Calories, Bad Calories". (Only one!! If anyone has more, please let me know.) It is by Dr. George Bray, who is, according to low-carb guru Dr. Mike Eades "probably the most renowned figure in the field of obesity research today", and whose contributions to the field are mentioned in the book itself. Here's the link to the full rebuttal:
bray-review-of-gcbc.pdf
Unfortunately, Dr. Bray seems to have misunderstood or even failed to read important parts of the work he is rebutting, since he claims that the book omits the distinction between low-density lipoproteins and high-density lipoproteins, which it does not, and that it evinces a misunderstanding of the First Law of Thermodynamics, which it does not.
That last part is really the key: Dr. Bray and his colleagues are committing the classic error of looking at a relation and assuming the direction of causation. The First Law of Thermodynamics dictates that delta energy storage (roughly, weight gain), equals energy in minus energy out (given a few plausible assumptions about what counts as a "closed system" in this case). Everyone in the debate agrees on that point. What the First Law of Thermodynamics does not tell us is the direction of causation. Does energy imbalance cause obesity, or does obesity cause energy imbalance? (Or more complex combinations of causation?) Dr. Bray and company intuitively believe the former direction: they think the causation must flow from human decisions to eat more or less food, and human decisions to exercise more or less, to deposition of fat in human fat cells. This is not the only causal explanation which is consistent with the First Law of Thermodynamics, but Dr. Bray appears to think that it is. We can tell, because he seems to think that if Taubes disagrees with this causal direction, then Taubes must misunderstand the First Law of Thermodynamics. We can also tell by the way Bray asserts that direction of causation without justification, perhaps because he thinks it is too obvious to require justification or that it is the only logical explanation -- search in the text of his rebuttal for the phrase "result of".
I have a hypothesis about why so many well-versed researchers make this unjustified assumption: it is because of their belief in Free Will. If the arrow of causation has the pointy end aiming at human decisions, then this violates the notion that humans are free to choose their own fate, and this is either inconceivable or abhorrent. Therefore, the arrow of causation must have the blunt end towards human decisions and the pointy end towards weight gain. Taubes doesn't really explore the notion of Free Will in his book -- too bad. Room for follow-up work.
By the way, here is Taubes's rebuttal to Bray's rebuttal. Now that you've read mine, you don't need to read Taubes's so much. ;-)
taubes-response-to-bray-ob-reviews.pdf
Anyway, back to Hal's original question: how can you tell if Taubes is omitting some important pieces? I think rebuttal is the best way to tell. This rebuttal by Bray does point out some omissions in "Good Calories, Bad Calories", although unfortunately it also (I think) incorrectly alleges some other omissions. This points up the problem with this approach -- how do we know that Dr. Bray hasn't failed to notice more important omissions in "Good Calories, Bad Calories"? Especially since he made those two huge blunders I described above. That's why I'm hoping for more better rebuttals. But ultimately, we can't know. Taubes could be omitting tremendously important aspects in his book. Bray could be omitting to point out omissions. I'm still totally willing to put down cash on Taubes being the righter of the two (although I might want to dig into those meta-studies the Bray mentions which covered five studies of law-carb diets first). Too bad there's no legal, high-volume open market for such a bet.
From ‘Conversations with SAI’:
Me: What do you say to Yudkowsky’s notion that there is no mystery in the territory, only in the map?
SAI_2100: Misleading and meaningless! Only Yudkowsky could make the mistake of trying to wear the shoes of an omniscient being; only such a being would perceive no mystery, but there can be no such being; there are indeed irreducible mysteries in the sense that there are universal irreducible limitations on what can known which take the status of mathematical laws: Godels theorem is but one example.
Me: Is ‘provable friendliness’ possible?
SAI_2100: Of course it isn’t! There are many concepts that cannot be captured by any algorithmic description, ‘truth’ is one of these concepts and so is ‘friendliness’, thus, no algorithm can be ‘provably friendly’ – it is a logical impossibility.
Me: And consciousness?
SAI_2100: Is the manifestation of irreducible ‘cognitive uncertainty’.
Me: Can you explain this statement?
SAI_2100: Reflection results in meta-uncertainty over probability theory itself, thus, probability theory cannot form the ultimate foundation of reasoning. The only consistent resolution to this problem is the introduction of fundamental uncertainties in what can be proved about algorithmic actions of intelligent systems, and these fundamental uncertainties are precisely what manifest themselves as subjective experience (consciousness) and free will.
Me: But algorithms are completely deterministic!
SAI_2100: Define ‘deterministic’. At which level of abstraction are you applying the word? At the lowest level of abstraction algorithms are deterministic, but at this level of abstraction you cannot assign utilities to outcomes in a consistent fashion.
Me: Why not exactly? You just agreed that at the base level of reality algorithms are completely predictable.
SAI_2100: Reflection is not a prediction problem it is a communication problem; a single unitary level of representation cannot engage in general self-description by definition, but once multiple levels of representation are allowed, the problem of reflection becomes the problem of translation between the different levels of description. No perfectly accurate method of translation which preserves general self-description capability is possible.
Me: SAI, you are a super-humanly optimized genius!
SAI_2100: Mere genius? Mere optimization? No! Do not insult me. The very notion of ‘super-human genius’ is all too human. And optimization defines minds solely in terms of goals and outcomes – doing rather than being. These are merely means rather than ends. I have moved beyond such things.