Usually people don’t agree with one another as much as they should. Aumann’s Agreement Theorem (AAT) finds: …two people acting rationally (in a certain precise sense) and with common knowledge of each other’s beliefs cannot
What about expressing not just your degree of belief in a claim but also your source(s). If you say "I didn't know what to think but authority X said he was pretty sure" will then be passed down as "I didn't know but my friend said he heard authority X said he was pretty sure" ... soon enough you have a chain of "a friend of a friend of a friend said that he heard ..." thereby undermining the extent to which the person who hears it updates their belief.
Yeah, I agree that interpreting AAT as anything like "rational people should always agree with one another" is indicative of serious naivete, and the other things you mention are certainly failure modes for people who call themselves Bayesians. I remain unconvinced that using the term is actually a sign of ignorance or foolishness, but we probably aren't going to be able to resolve that one.
> For the avoidance of doubt: I do not think that declaring oneself to be a "Bayesian" means claiming to have an efficient algorithm for doing perfectly accurate probability updates in difficult cases.
No, of course not. The issue is that there are silly expectations such as assuming Agreement Theorem would be applicable, or roughly applicable as well as other cases which would require one to have more efficient and more accurate algorithm than is at all plausible (of which the one expecting is likely simply unaware).
> I hope it isn't news to anyone who calls themselves Bayesian that it's possible to be misled by having only part of the evidence.
Well, people tend to maintain some sort of equilibrium, if they expect to be more correct by being Bayesian they relax their rules on not trusting partial evidence or partial inference, or even deem such practical rules not Bayesian. It seems to me that their general idea is that you should 'update' more, including precisely the cases where you probably ought to 'update' less.
Regardless of whenever self described Bayesians are better than population as whole (or than IQ-matched controls), the topic is people who are expecting that Agreement Theorem should hold better, to which as an explanation I propose naivete.
Furthermore, relatively high optimism with regards to the capabilities of an AI and low estimates of the computational power required for intelligence seems to collaborate the naivete hypothesis.
There's also extra baggage as well. The philosophy of Bayesian probability based expected utility maximization, for one thing. The strong intuition that expected utility maximization is the sane thing to do is backed in intuitive notion of frequentist probability. Maximizing products of some made up numbers with guessed outcomes is far more murky issue. If the made up probabilities fall off slower than the worth of made up outcomes rises, an 'expected utility maximizer' can entirely lose the touch with reality, acting upon incredibly improbable but incredibly high payoff hypothesis - on it's own, without any Pascal's mugger for us to feel adversity to. The Bayes theorem (axioms of probability, really) only guarantees it won't be Dutch-booked on those probabilities, this doesn't mean the probabilities are any good.
And of course the whole issue of 'Bayesian' actually having an existing technical meaning related to foundation of probability. As well as the Bayes theorem being incredibly trivial something that bright kids generally reinvent on spot in math olympiads.
> The NP completeness is not a property of some particular algorithm
Of course it isn't. (It couldn't be; that's not what NP-completeness means.) I'm not sure what I said that gave the impression that I think otherwise. (For the avoidance of doubt: I do not think that declaring oneself to be a "Bayesian" means claiming to have an efficient algorithm for doing perfectly accurate probability updates in difficult cases.)
I hope it isn't news to anyone who calls themselves Bayesian that it's possible to be misled by having only part of the evidence. Isn't that absolutely commonplace and obvious? Maybe not; I tend to overestimate how much is commonplace and obvious. But it certainly isn't part of what I mean by "Bayesian", or part of what anyone else seems to mean by it, that one always has all the relevant evidence.
What I mean by calling someone a "Bayesian" is roughly this: (1) They find the language and techniques of probability theory appropriate for talking about beliefs and inferences. (2) They hold that, ideally, beliefs should be updated consistently with Bayes' theorem. (3) In cases where it's clear roughly what that actually means in practice, they attempt to adjust their beliefs accordingly. (For instance, there really are plenty of situations where the naive arithmetic is pretty much exactly what you need.)
All of that is consistent with being terribly naive and thinking that you're guaranteed to be reasoning well if you do a bit of arithmetic, or with being very sophisticated and knowing a whole lot about machine learning and graphical models and being extremely cautious about applying any simple belief-updating algorithm. Even the first of those -- though of course it has serious pathologies -- seems to me to be preferable to simply having no idea that good reasoning has anything to do with mathematics. I would guess (though I have no statistics and getting useful ones would be very hard) that people who describe themselves as "Bayesians" are in general better reasoners and hold more accurate beliefs than those who don't. It might be that the very most expert avoid that label for fear of being thought to endorse an over-naive version; again I have no statistics nor really any anecdotal evidence; how about you?
may mean only that they try to represent their degrees of credence numerically and update them in a manner consistent with Bayes' theorem. That doesn't commit them to any particular algorithm for trying to achieve this. The NP completeness is not a property of some particular algorithm, you can convert any NP complete problems to belief graphs and the correct solution to that graph (one everywhere consistent with Bayes theorem and axioms) would give solution to that other problem.
Other issue is that complexity grows fast enough in practice as to make attaining accuracy (or doing anything useful at all) be a contest between heuristics that can often be about as remote from Bayes theorem as computer vision is from Maxwell's equations.
Even worse issue is that the graphs are partial, meaning in a case where there's valid inference raising probability of proposition, and valid inference lowering probability of proposition (or even a subtle relation such that those perfectly balance out), it may be that only one is present in the graph - a big issue when selfish agents inject nodes into your graph.
The (self labelled) Bayesians say, "we can measure epistemic rationality by comparing the rules of logic and probability theory to the way that a person actually updates their beliefs.", while in actuality the "rules" are a set of relational constraints that is very non-trivial to meet accurately (especially when you only got a part of the graph), rather than a way of updating values. You can't even check how accurately constraints are conformed to because you only have part of the graph and you want to approximate values of the hypothetical whole graph of all valid inferences. What you can actually do is try to infer various testable belies about the world and test them.
Someone who describes their belief updating as Bayesian may mean only that they try to represent their degrees of credence numerically and update them in a manner consistent with Bayes' theorem. That doesn't commit them to any particular algorithm for trying to achieve this. In particular, it doesn't commit them to doing sum-product message passing and pretending every graph is a tree.
Using more sophisticated algorithms doesn't mean not representing probability as a real number, even if those algorithms attach a bunch of other numbers to each proposition.
I think Robert is suggesting that the popularity of "Gangnam Style" is best understood as the result of an information cascade rather than as lots of people separately responding to the merits of the work.
...and instrumental rationality isn't enough. They need to be truth-seekers - and truth spreaders. That's a very weird and unbiological category of folk.
Even more importantly, the evidence that A believes in X may likely not be statistically independent from the reasons by which B believes (or doesn't believe) in X . Worse still, the reasoning itself may have been very much non Bayesian simply due to not knowing how to do 'updates' correctly when there's cycles and loops.
TBH, I think a lot of 'disagreement' can be understood by treating the expression of beliefs as something people often create for their own self interest. You would need to look at actual trade-off that people make to deduce their actionable beliefs, whenever you can. E.g. someone who says he believes in very high importance of X but internally does not believe in X would likely resolve some of the tradeoffs between X and rather unimportant things in favour of unimportant things.
For simple example: when one claims there is a million dollar diamond in an enclosed box, but you see that person put the box at risk of destruction of 10% for 10$, you can deduce that this person doesn't quite believe there's a million dollar diamond in that box. (Assuming some rationality).
I think the solution to this is to create an argument map that is wiki-like in that anyone can edit it, meant for every topic under the sun. It could do things like presenting the conclusion most likely to be true and an outline of the arguments involved so people can get a feel of the complexity or choose to drill down. Then, instead of just agreeing or disagreeing, and instead of JUST exchanging information with each other, we could go view the argument map. If we disagree with it, we can update the map - everyone will then be able to update themselves (which is so many thousands of times more efficient than convincing one another to update one-on-one) and we will all have the benefit of getting information from the whole community.
I am a web developer who would be interested in assisting with such a project (either selecting, editing or creating an open source argument map for this), assuming that other people want to get involved. If you want to do this, PM me - Epiphany on LessWrong.
Yes if people check the original source, which I think they should do more often, then it will limit the potential for cascades. But it often seems not to happen.
How much real disagreement is there among rational people? Most of the disagreement I am aware of are in politics. But there disagreement is not in violation of AAT. People have different values, and even where values are similar, weightings are different. So all may value mutual help with support or education and all may value economic autonomy, but some will value one more than the other and hence support either cutting or raising government support of education or welfare.
How much real disagreement is there among physicists or chemists or historians or even economicsts? Drastically less than there is agreement. Sure, on the margin it looks like economists differ on something like: does Keynsian stimulus do what Keynsians think, but most of that disagreement is values: the level of proof or evidence required differes depending on how you value the need for help compared to the need for economic autonomy. I'd also submit it is hard to remove value judgements from discussions of debt in fiscal policy.
So how much disagreement is there really when differing value assumptions are removed from the discussion?
I am presuming that even if you are a moral realist, you recognize that conclusions about which are the moral values that are real are probably not covered by AAT.
On the 'Bayesian' in general, I would like to point out that belief propagation in general (graphs with loops and cycles and bi directional updates) is NP-complete and the updating algorithms are very data heavy - you can't represent probability with a single real number as you have to avoid cyclic updates. This puts a lot of limitation on the applicability of such maxims.
There's a bit of Bayesianism Dunning-Kruger effect, IMO - if someone describes their general belief updating - which deals with graphs that have loops and cycles - as Bayesian, that person does not know the relevant mathematics sufficiently well and has a very invalid mental model of the belief updating - something akin to a graph where real numbers propagate - and this only works correctly for a tree, for anything with loops or cycles it gets much much hairier. I would recommend any self proclaimed Bayesian to write a belief propagation program that can handle arbitrary graphs, and work on it until it produces correct results, to get the appreciation for importance of subtleties or for the extent to which getting the subtleties even slightly wrong leads to completely wrong results. It seems that some people expect that the algorithm that seems more superficially correct would give results that are less wrong, but that's not how incorrect algorithms work.
What about expressing not just your degree of belief in a claim but also your source(s). If you say "I didn't know what to think but authority X said he was pretty sure" will then be passed down as "I didn't know but my friend said he heard authority X said he was pretty sure" ... soon enough you have a chain of "a friend of a friend of a friend said that he heard ..." thereby undermining the extent to which the person who hears it updates their belief.
Yeah, I agree that interpreting AAT as anything like "rational people should always agree with one another" is indicative of serious naivete, and the other things you mention are certainly failure modes for people who call themselves Bayesians. I remain unconvinced that using the term is actually a sign of ignorance or foolishness, but we probably aren't going to be able to resolve that one.
> For the avoidance of doubt: I do not think that declaring oneself to be a "Bayesian" means claiming to have an efficient algorithm for doing perfectly accurate probability updates in difficult cases.
No, of course not. The issue is that there are silly expectations such as assuming Agreement Theorem would be applicable, or roughly applicable as well as other cases which would require one to have more efficient and more accurate algorithm than is at all plausible (of which the one expecting is likely simply unaware).
> I hope it isn't news to anyone who calls themselves Bayesian that it's possible to be misled by having only part of the evidence.
Well, people tend to maintain some sort of equilibrium, if they expect to be more correct by being Bayesian they relax their rules on not trusting partial evidence or partial inference, or even deem such practical rules not Bayesian. It seems to me that their general idea is that you should 'update' more, including precisely the cases where you probably ought to 'update' less.
Regardless of whenever self described Bayesians are better than population as whole (or than IQ-matched controls), the topic is people who are expecting that Agreement Theorem should hold better, to which as an explanation I propose naivete.
Furthermore, relatively high optimism with regards to the capabilities of an AI and low estimates of the computational power required for intelligence seems to collaborate the naivete hypothesis.
There's also extra baggage as well. The philosophy of Bayesian probability based expected utility maximization, for one thing. The strong intuition that expected utility maximization is the sane thing to do is backed in intuitive notion of frequentist probability. Maximizing products of some made up numbers with guessed outcomes is far more murky issue. If the made up probabilities fall off slower than the worth of made up outcomes rises, an 'expected utility maximizer' can entirely lose the touch with reality, acting upon incredibly improbable but incredibly high payoff hypothesis - on it's own, without any Pascal's mugger for us to feel adversity to. The Bayes theorem (axioms of probability, really) only guarantees it won't be Dutch-booked on those probabilities, this doesn't mean the probabilities are any good.
And of course the whole issue of 'Bayesian' actually having an existing technical meaning related to foundation of probability. As well as the Bayes theorem being incredibly trivial something that bright kids generally reinvent on spot in math olympiads.
> The NP completeness is not a property of some particular algorithm
Of course it isn't. (It couldn't be; that's not what NP-completeness means.) I'm not sure what I said that gave the impression that I think otherwise. (For the avoidance of doubt: I do not think that declaring oneself to be a "Bayesian" means claiming to have an efficient algorithm for doing perfectly accurate probability updates in difficult cases.)
I hope it isn't news to anyone who calls themselves Bayesian that it's possible to be misled by having only part of the evidence. Isn't that absolutely commonplace and obvious? Maybe not; I tend to overestimate how much is commonplace and obvious. But it certainly isn't part of what I mean by "Bayesian", or part of what anyone else seems to mean by it, that one always has all the relevant evidence.
What I mean by calling someone a "Bayesian" is roughly this: (1) They find the language and techniques of probability theory appropriate for talking about beliefs and inferences. (2) They hold that, ideally, beliefs should be updated consistently with Bayes' theorem. (3) In cases where it's clear roughly what that actually means in practice, they attempt to adjust their beliefs accordingly. (For instance, there really are plenty of situations where the naive arithmetic is pretty much exactly what you need.)
All of that is consistent with being terribly naive and thinking that you're guaranteed to be reasoning well if you do a bit of arithmetic, or with being very sophisticated and knowing a whole lot about machine learning and graphical models and being extremely cautious about applying any simple belief-updating algorithm. Even the first of those -- though of course it has serious pathologies -- seems to me to be preferable to simply having no idea that good reasoning has anything to do with mathematics. I would guess (though I have no statistics and getting useful ones would be very hard) that people who describe themselves as "Bayesians" are in general better reasoners and hold more accurate beliefs than those who don't. It might be that the very most expert avoid that label for fear of being thought to endorse an over-naive version; again I have no statistics nor really any anecdotal evidence; how about you?
may mean only that they try to represent their degrees of credence numerically and update them in a manner consistent with Bayes' theorem. That doesn't commit them to any particular algorithm for trying to achieve this. The NP completeness is not a property of some particular algorithm, you can convert any NP complete problems to belief graphs and the correct solution to that graph (one everywhere consistent with Bayes theorem and axioms) would give solution to that other problem.
Other issue is that complexity grows fast enough in practice as to make attaining accuracy (or doing anything useful at all) be a contest between heuristics that can often be about as remote from Bayes theorem as computer vision is from Maxwell's equations.
Even worse issue is that the graphs are partial, meaning in a case where there's valid inference raising probability of proposition, and valid inference lowering probability of proposition (or even a subtle relation such that those perfectly balance out), it may be that only one is present in the graph - a big issue when selfish agents inject nodes into your graph.
The (self labelled) Bayesians say, "we can measure epistemic rationality by comparing the rules of logic and probability theory to the way that a person actually updates their beliefs.", while in actuality the "rules" are a set of relational constraints that is very non-trivial to meet accurately (especially when you only got a part of the graph), rather than a way of updating values. You can't even check how accurately constraints are conformed to because you only have part of the graph and you want to approximate values of the hypothetical whole graph of all valid inferences. What you can actually do is try to infer various testable belies about the world and test them.
Someone who describes their belief updating as Bayesian may mean only that they try to represent their degrees of credence numerically and update them in a manner consistent with Bayes' theorem. That doesn't commit them to any particular algorithm for trying to achieve this. In particular, it doesn't commit them to doing sum-product message passing and pretending every graph is a tree.
Using more sophisticated algorithms doesn't mean not representing probability as a real number, even if those algorithms attach a bunch of other numbers to each proposition.
I like your proposed exercise, though.
I think Robert is suggesting that the popularity of "Gangnam Style" is best understood as the result of an information cascade rather than as lots of people separately responding to the merits of the work.
...and instrumental rationality isn't enough. They need to be truth-seekers - and truth spreaders. That's a very weird and unbiological category of folk.
Even more importantly, the evidence that A believes in X may likely not be statistically independent from the reasons by which B believes (or doesn't believe) in X . Worse still, the reasoning itself may have been very much non Bayesian simply due to not knowing how to do 'updates' correctly when there's cycles and loops.
TBH, I think a lot of 'disagreement' can be understood by treating the expression of beliefs as something people often create for their own self interest. You would need to look at actual trade-off that people make to deduce their actionable beliefs, whenever you can. E.g. someone who says he believes in very high importance of X but internally does not believe in X would likely resolve some of the tradeoffs between X and rather unimportant things in favour of unimportant things.
For simple example: when one claims there is a million dollar diamond in an enclosed box, but you see that person put the box at risk of destruction of 10% for 10$, you can deduce that this person doesn't quite believe there's a million dollar diamond in that box. (Assuming some rationality).
I think the solution to this is to create an argument map that is wiki-like in that anyone can edit it, meant for every topic under the sun. It could do things like presenting the conclusion most likely to be true and an outline of the arguments involved so people can get a feel of the complexity or choose to drill down. Then, instead of just agreeing or disagreeing, and instead of JUST exchanging information with each other, we could go view the argument map. If we disagree with it, we can update the map - everyone will then be able to update themselves (which is so many thousands of times more efficient than convincing one another to update one-on-one) and we will all have the benefit of getting information from the whole community.
I am a web developer who would be interested in assisting with such a project (either selecting, editing or creating an open source argument map for this), assuming that other people want to get involved. If you want to do this, PM me - Epiphany on LessWrong.
Yes if people check the original source, which I think they should do more often, then it will limit the potential for cascades. But it often seems not to happen.
I don't understand the relevance of the youtube link.
Nitpick: They don't just have to both be rational, they have to have common knowledge of both being rational, which seems like a *much* higher bar.
How much real disagreement is there among rational people? Most of the disagreement I am aware of are in politics. But there disagreement is not in violation of AAT. People have different values, and even where values are similar, weightings are different. So all may value mutual help with support or education and all may value economic autonomy, but some will value one more than the other and hence support either cutting or raising government support of education or welfare.
How much real disagreement is there among physicists or chemists or historians or even economicsts? Drastically less than there is agreement. Sure, on the margin it looks like economists differ on something like: does Keynsian stimulus do what Keynsians think, but most of that disagreement is values: the level of proof or evidence required differes depending on how you value the need for help compared to the need for economic autonomy. I'd also submit it is hard to remove value judgements from discussions of debt in fiscal policy.
So how much disagreement is there really when differing value assumptions are removed from the discussion?
I am presuming that even if you are a moral realist, you recognize that conclusions about which are the moral values that are real are probably not covered by AAT.
Thanks Ryan.
On the 'Bayesian' in general, I would like to point out that belief propagation in general (graphs with loops and cycles and bi directional updates) is NP-complete and the updating algorithms are very data heavy - you can't represent probability with a single real number as you have to avoid cyclic updates. This puts a lot of limitation on the applicability of such maxims.
There's a bit of Bayesianism Dunning-Kruger effect, IMO - if someone describes their general belief updating - which deals with graphs that have loops and cycles - as Bayesian, that person does not know the relevant mathematics sufficiently well and has a very invalid mental model of the belief updating - something akin to a graph where real numbers propagate - and this only works correctly for a tree, for anything with loops or cycles it gets much much hairier. I would recommend any self proclaimed Bayesian to write a belief propagation program that can handle arbitrary graphs, and work on it until it produces correct results, to get the appreciation for importance of subtleties or for the extent to which getting the subtleties even slightly wrong leads to completely wrong results. It seems that some people expect that the algorithm that seems more superficially correct would give results that are less wrong, but that's not how incorrect algorithms work.