How Much Defer To Experts?

Bryan Caplan argues the public has very poorly informed political beliefs, best described as "irrational," and his main evidence is that they are strong and yet differ greatly and systematically from beliefs of professional economists, after correcting for ideology, income, and other characteristics.  Dan Klein and Tyler Cowen complain that, compared to most economists, Bryan’s beliefs unusually far from median economists’ beliefs.  Tyler: 

Dan’s key point is that you, in fact, differ radically from the professional consensus (as Arnold writes as well) and that the argument is self-undercutting if you simultaneously erect expert consensus as a relevant benchmark. I don’t see that you have yet replied to this.

Bryan replies: 

I only embrace a presumption of expert competence; it’s where I start, not necessarily where I finish.  Still, Tyler is entirely correct to say that I’m not an average economist. How do I account for the difference? In large part, I think that other economists have failed to fully free themselves from anti-market bias; … the best I can do is argue issue-by-issue, and see if I can make a dent. … Another large chunk comes down to different values. … I’m a staunch libertarian. I’m extremely meritocratic. … I look down on patriotism and piety of every kind.

It seems to me that Bryan’s intuition is that a large disagreement with experts is a strong positive sign of irrationality, but that a small disagreement is only a weak sign about rationality.  Even if Bryan is an unusual economist, he still mostly agrees with them, making his overall disagreement "small."  But this raises the obvious question:  how much disagreement with experts is enough for a strong sign of irrationality?