Diagnosis Futures

From ’97 to ’99 I was a RWJF Health Policy Scholar (at UC Berkeley), and my final project and presentation was on what I called “treatment futures”, i.e., the idea of using decision markets to forecast treatment-conditional health outcomes for individual patients. I proposed:

  1. At major treatment decision point, post sanitized medical record & options to web.
  2. Subsidize [betting] markets estimating treatment-conditional outcomes (e.g. lifespan).
  3. Anyone can trade or add treatment options.
  4. Market estimates inform treatment choice.
  5. Outcome determines market asset values.

I also posted on this in ’07. Yesterday I learned that a new startup, CrowdMed, is spending $1.1M to try a related idea. They will have ordinary people “bet” on particular patient diagnoses. I put “bet” in quotes because they only bet donations, and they don’t tell users how individual predictions, individual winnings, and consensus estimates on patients are related. That is apparently part of their patented secret sauce – you’ll just have to trust them.

A patient pays $200 to post their problem, and promises to eventually declare a “correct” diagnosis. Each player is given $5 to start, and can only spend winnings on donating to Watsi patients. So if after several years hard work, you do much better than average, and end up with $20, you might donate that much – woo hoo! Player incentives to diagnose correctly are diluted further by the fact that they only predict what the patient will say is their diagnosis, not the true diagnosis. And players don’t get to look at a full medical history, just a few paragraphs of description.

Patients mainly pay for possible diagnoses to suggest to their doctor to consider, diagnoses that players believe might find supporting evidence, if only the patient’s doctor would consider them. So patients have to believe that their doctor will believe that these volunteer amateur detectives have useful diagnosis suggestions to pursue, ones the doctor would not have otherwise considered. Seems a pretty high bar to me.

My conditional forecasting concept could help patients even if patient doctors don’t believe in it, but it does require players to wait longer to find out if they win. And I think that players deserve a much higher fraction of the patient payments than this startup seems willing to give them — I expect CrowdMed incentives are way too weak. Many seem to have decided that the big idea in “crowd-sourcing” is getting amateurs to do for free what you’d otherwise have to pay professionals to do. Me, I think you usually need to pay good money to get good info, even when you do it right.

Added 3p: The CrowdMed founder replies in the comments; I respond also.

GD Star Rating
loading...
Tagged as: ,
Trackback URL:
  • http://www.gwern.net/ gwern

    That does sound rather weaksauce… what do they plan to do if the best predictor just turns out to be someone plugging patient info into an expert system or Bayes net, or a base-rate predictor?

    • Frank White

      Unless they’ve got specific domain expertise, most people can probably do no better on average than WebMD, so I suspect that plugging in the symptoms (or your best estimate of them) into WebMD and outputting whatever WebMD gives you is probably Nash for this kind of incentives.

  • IMASBA

    “3. Anyone can trade or add treatment options.”

    Anyone? Even laymen? This sounds like a moneymaking scheme with an “eat shit, 10 trillion flies can’t be wrong” justification. What’s next, replacing universities with prediction markets (if most people think the world is 6000 years old, then that has to be the truth and we’ll teach it in all schools)?

    Btw, this whole thing leads to the death of creativity: if everything is based on prediction markets which are based on the outcomes of previous prediction markets then there is no progress. If going along with the “cancer option” leads to less loss of life than going along with the “TBC option” then prediction markets will converge towards cancer being the right diagnosis, until the prediction market dries up because it’s too uniform, diluting the reward too much. But what if it’s a disease that hasn’t been discovered yet? Who is going to do research into undiscovered diseases when the standard practice has become the prediction market which “solves” questions by comparing existing ideas? So prediction markets can never be more than a tangential experiment running parallel to the scientific method.

    • http://www.facebook.com/martinL.epstein Martin Epstein

      If almost everyone thinks the Earth is 6 thousand years old then a prediction market can’t come soon enough. Once the Earth is confirmed to be 4.5 billion years old I will make a boatload of money. A prediction market is not a survey.

      Also, “undiscovered disease” would be a perfectly fine prognosis to bet on.

      • IMASBA

        “If almost everyone thinks the Earth is 6 thousand years old then a prediction market can’t come soon enough. Once the Earth is confirmed to be 4.5 billion years old I will make a boatload of money.”

        Yeah, except the patient would already be dead when the majority of the prediction market participants are shown their mistake. You could reduce the chances of that happening by only allowing doctors to participate, but even then you’re not making optimal use of their potential skills because specialization is more efficient (as humans have known for thousands of years): it’s better to have your cancer diagnosed and treated by an oncologist and your aneurysm diagnosed and treated by a neurologist then to take the average diagnosis and average treatment of both these doctors for both ailments. And the thing is, the prediction market is based on money, so that opens the door for manipulation and speculation which no sane person wants in medicine and science in general.

        “Also, “undiscovered disease” would be a perfectly fine prognosis to bet on.”

        If it’s an option on the list, in any case it requires independent research which is not bound to the outcomes of prediction markets.

      • davesmith001

        I don’t think you understand how prediction markets work or at the very least you underestimate the power of requiring people to put their money where their mouth is. Your chosen example of earth age is a great one to disprove your own point. One of the reasons so many people believe in a young earth is that it costs them nothing to do so.

      • IMASBA

        Prediction markets that influence policy only get corrected after the damage is done. In addition they hinder independent experts and because they involve money they are vulnerable to manipulation and speculation (let’s have a prediction market on the effects of fracking on the environment and let the Koch Brothers particiapte, what could possibly go wrong?). In general it’s this religious belief of the market always being right (put a lot of people who know nothing about a subject together and somehow their average answer will the truth) that I have beef with. Why not have a prediction market on the validity of string theory or the question of the existence of an afterlife while we’re at it?

        Oh and I have no doubt that in the Southern US or the Middle East a prediction market on the origins of life would not end with evolution coming up on top. If people are willing to donate money to missionaries then they’re willing to bet money on creationism.

        In those situations when prediction markets are as good as the experts that usually says more about the experts than the prediction market and anyone with bit of common sense and knowledge could have known that in advance.

      • davesmith001

        You could be right that prediction markets might not work/be subject to some kind of manipulation in these types of circumstances but in many cases the act of buying and selling has proven to be an effective way to solve problems and disseminate problems. I’d like to see us gain much more experience from prediction markets.

      • IMASBA

        Markets are fine for (most) goods and services, not for truth finding. Like I said, the only time prediction markets beat experts is when the experts aren’t really experts and then you might as well have tossed a coin to make a decision.

      • Stephen S

        If that’s the case, then prediction markets serve the intensely useful function of discerning which fields are susceptible to real expertise.

      • http://www.facebook.com/CronoDAS Douglas Scheinberg

        Once the Earth is confirmed to be 4.5 billion years old I will make a boatload of money. A prediction market is not a survey.

        No, you won’t make your money until whomever is judging the contract is convinced. If it’s 1870 and the Church of England is sponsoring a prediction market on Darwin’s theory of evolution by natural selection, would you bet on Darwin?

  • Jared Heyman

    Jared Heyman here, the founder of CrowdMed. Dr. Hanson – I’ve been following your work for years and appreciate all you’ve done for prediction markets in both academia and industry.

    A few points of clarification regarding CrowdMed:

    – We don’t explain to our traders (aka, “Medical Detectives”) how their individual predictions, individual winnings, and consensus estimates on patients are related simply to avoid confusion. As you know, many commercial prediction markets have failed in the past because they overwhelmed lay participants with technical detail. Our strategy is more like Google’s — keep the user interface as simple as possible, only telling users what they need to know, and maintain the technical intricacies under the hood.

    – We debated furiously whether to use real-money trader incentives, play money incentives (i.e., points), or charity-based incentives. We settled upon a combination of the second and third. Though real-money incentives can certainly motivate “information discovery” as you economists say, it could also motivate market manipulation. Besides, intrinsic rewards (like a sense of contribution, purpose, meaning, and competence) can be equally or more powerful than extrinsic rewards (like money.) In user surveys, CrowdMed traders have reported being primarily motivated by helping another human being – something which no reward system can capture. They way we see it, even the money that traders can win for Watsi is just gravy.

    – It’s up to our patients how much information they chose to disclose to the Medical Detectives. Some provide only high-level information about their illness, others upload their entire medical record. It’s in a patient’s best interest to provide just enough “clues” for the Medical Detective community to solve their case, as any unnecessary information will only discourage them from reviewing it in the first place.

    – Yes, our process does necessitate a patient to convince their doctor to take CrowdMed results seriously. We expect for doctors to initially be skeptical of our results, but once they prove consistently accurate and insightful, then physician adoption should get much easier for us.

    – Our process also necessitates the patient to accurately report their final correct diagnosis to us. We’re not worried about this one – if you ever speak with a patient who’s suffered for months or years with a chronic disease, you’ll find that she often knows much more about her condition than her doctor, and is certainly capable of confirming her correct diagnosis once it’s identified.

    – We’re not trying to get “amateurs to do for free what you’d otherwise have to pay professionals to do.” Our objective is to harness the wisdom of crowds to *supplement* the work of
    professionals. CrowdMed isn’t trying to replace doctors, we’re trying to empower patients to speak with their doctors more intelligently about their disease and its most probable root causes.

    Feel free to reach out to me directly if you’d like to learn more about CrowdMed. I welcome your feedback of our system once you have a more complete understanding of how we designed it and why.

    • http://overcomingbias.com RobinHanson

      Your plan is for CrowdMed to keep the vast majority (>95%) of the money that patients pay, leaving only a tiny fraction for the detectives who provide the info you sell, offering those detectives instead “a sense of contribution, purpose, meaning”, a compensation you think sufficient for them but not for yourselves. How exactly is that not “trying to get amateurs to do for free what you’d otherwise have to pay professionals to do”? And how do you expect to motivate your detectives to compete to show their “competence” if you won’t tell them the exact rules to your game?

      • Jared Heyman

        What makes you think that CrowdMed keeps >95% of the money that patients pay? We don’t limit the charitable payouts for any of our markets, and in fact it’s mathematically possible that our payout *exceeds* what we charge the patient. You could accuse us of having a flawed business model, but not being greedy.

        Again, we don’t believe that our Medical Detectives are motivated by money (or at least, we don’t wish to attract those that are.) We’re a business, and as such we’re obligated to charge for our service and produce a return for our shareholders. I find nothing immoral or internally inconsistent in charging patients for the service we provide while not providing a cash-based incentive for our Medical Detectives, as long as this model motivates the right behavior and satisfies all of our constituents.

      • Stephen Diamond

        What makes you think that CrowdMed keeps >95% of the money that patients pay?

        The question obviously deserves an answer, although “we don’t limit the charitable payouts” is phrased in jargon I must admit not understanding.

        Robin in his innuendo seems to take cynical delight in wielding the weapons of “forager morality” by inveighing against greed. (It seems that being aware of psychological underpinnings can serve as either caveat or as justification. The guns of critical thought can be fired for hypocrisy.)

        As a dispassionate observer who has absolutely no affection for markets ( http://tinyurl.com/blhdluc ), I would say it would be misleading to call your enterprise a prediction market, since Robin’s objections mark defining criteria for those entities.

        But whether your enterprise will provide value is a purely empirical question. If Robin wants to argue against it, he needs to point to evidence that only true prediction markets work.

        ADDED. The comment by Robin most relevant to his “cynical delight” is:

        I think that players deserve a much higher fraction of the patient payments than this startup seems willing to give them.

      • http://overcomingbias.com RobinHanson

        You are free to tell us what % of patient payments you expect will go to detectives, but in the absence of your telling us I have to estimate based on the parameters you’ve given us.

        Yes, there have been successful play money markets, but there is also a vast graveyard of such markets that failed badly because their creators thought it sufficient to just announce a website. It takes far more than javascript to make a community that cares enough to offer thoughtful expert contributions while forgoing material compensation. You’ve said you aren’t going to pay detectives but you haven’t said how you expect to actually provide them the “sense of contribution” you plan to rely on.

      • Jared Heyman

        The percent of patient payments that go to our detectives is highly variable, ranging between 0% (if nobody suggests or bets on the correct diagnosis) to over 100% (if the correct diagnosis is suggested early in the market and lots of detectives bet on it.) Once we’ve run enough cases to establish a stable average payout I’m happy to report it to you.

        I know that play money markets have enjoyed mixed results over the years. From what I’ve read, they tend to work best when the market topic is of intrinsic interest to the participants (sports markets for sports fans, box office markets for movie buffs, etc.) We’re trying to recruit detectives who like solving medical mysteries for its own sake, just like some people like solving crossword puzzles. On top of that, we stress on our website that they’re helping to solve a real-world medical mystery and potentially help to save a life, plus of course the charitable contribution piece. Not everyone will be motivated by these things, but our hypothesis is that we can recruit enough people in the world who are. If our hypothesis proves incorrect then we’ll quickly change course — stubborn adherence to flawed beliefs is a luxury that few startups can afford.

    • IMASBA

      I like the idea of not using real money with the argument that that would open the door to manipulation and that the amounts would have to be huge to really pose a risk for traders (and that would exclude a lot of people from joining).

      I also like how you’re not trying to replace doctors or overrule them. If the contribution consists of suggestions that can help doctors remember some obscure, but important detail they had forgotten since med school or otherwise inspire them to think in a direction they hadn’t thought about then the project can be a *healthy* addition.

      Could the disagreement between crowdmed and Robin Hanson be based on the difference between practice and theory? the difference between the entrepeneur and the (libertarian-leaning) economist?

  • russ andersson

    Jared:

    I admit to being disappointed in your post …I expected better from you and your team.

    You would be wise to listen to Hanson as he is telling you something very important. The structure of what you are planning on implementing is flawed i.e. it won’t work!

    Prediction markets are not magical devices, they are based on simple mathematical principles, they have significant limitations, and they only work well under very specific scenarios. Scenarios which your proposed structure lacks. I know this because I created the largest set of prediction market data on the planet, and hanson
    knows this because he invented them. So rather than argue points which are irrelevant you should pay attention to the advice Robin is kindly giving you.

    In order for prediction markets to work, at the very least, you need

    1. a group of people that have “information”
    2. who can process/express that information through
    3. a methodology that aggregates this information.

    Jared, your structure barely has any of these essential elements in it. UI is the least of your worries my friend. Medical detectives meets prediction markets with a twist of altruism thrown in. Seriously?

    The structure that you are describing is a joke! It hasn’t worked well anywhere else. Ever.

    I am dissapointed that you think “medical detectives” reading case histories of patients and other self-reported data and then trading with each other for the good of others is going to produce any prediction remotely reliable. I’m more dissapointed that Thiel gave you capital for this hair brained scheme.

    You are extremely naive to:
    A).think you can make this structure work when all before you have failed
    B). come onto this blog and argue with the guy that invented the concept
    C). while starting your post off with some pc rambling drivel about thanking Hanson for his contribution to the field (both academic and industry no less) and then basically dismissing his opinions.

    To succeed, you need to do a lot better than this Jared. A lot lot better. You might be capable of it, but you need to try something else by having more clarity on: what information is needed to improve medical diagnoses, where you can get this information, how you can share this information, who/which structure is best positioned to process that information, and the collective incentives to make the entire scheme work.

    Right now you limited clarity on any of these important points.

    • Jared Heyman

      Russ, I’m open to learning from those who know more about prediction markets than myself, and certainly count Dr. Hanson within that group. My contention is that one should have a thorough understanding of how our system works and why before criticizing it.

      I’m happy to speak with anyone out there who has created a similar system in the past and failed — it’s much better to learn from the mistakes of those who came before us than repeat them.

      We’ve tried to think through each of the issues that you mention and have come up with the best model we could think of. If there are specific paths we’ve chosen that you (or anyone) knows to be incorrect, then I welcome that specific feedback.

      • Russ Andersson

        I have been to your site/read a few background
        articles, and have a reasonable grasp of how your proposed system is intended to work … based on the frequency that your premise/pitch is changing relative
        to what it was only a few months ago, it seems like you don’t have as many answers as you claim you do.

        The fundamental issue you are facing is how to get
        more reliable medical diagnoses into the hands of patients. What you seem to be offering at the moment is part literature review part bragging about how smart you are versus the medical community and part croudsourcing buzzword speak. I’m far from sold.

        Let’s assume your challenge is to diagnose the
        likely set of potential afflictions more reliably than existing medical practice does. Prove that you can. AND have your audience believe you can.

        I would suggest analyzing how related problems get
        solved today. Its not through a prediction market or any of the high end math you seem to be so fond of. A related problem might be understanding how people figure out what is wrong with their car? You are doing the same thing, the obvious parallel is helping patients figure out what is wrong with them. The
        car in this instance is merely more complex. As are the tests. So what might be helpfull is to study the elements of how the discovery process for diagnosing cars problems work and then figure out how to apply those findings to diagnosing medical conditions using a scalable business model. Look at the thousands of DYI websites/communities that work, they work for clearly observable reasons.

        The way your average bear finds out what is wrong
        with their car, is to 1. find an audience with experience, and 2. post car symptoms to that community, and 3. allow the community to trouble shoot it by asking the poster to perform tests and then report back their findings and then 4. repeating. (Does it have park, or not, if it does, then its likely fuel related etc. It has no spark, its electrical, have you checked your battery … etc), so the process basically it’s a series of symptoms followed by suggested tests repeated. Its basic troubleshooting Jared. No need for calculus.

        Maybe you consider a parallel approach thing, linking
        symptoms, with suggested clinical tests, followed by feedback, followed by more tests, until you discover the root cause. The low hanging fruit here is that the
        overwhelming majority of doctors are so stupid that they don’t know which clinical tests to order, and what these results actually mean. Speak to any credible Clinical Lab Scientist about the disconnect between the test
        administration and the doctors who ask for and interpret the tests and you will find out that this disconnect is massive. The CLSs feel incredibly sorry for patients everytime the lab phone rings and there is a docter on the other side of the call. They know gross stupidity is nearby. Little wonder medicine is in the state it is, imagine getting car advice from a mechanic that doesn’t understand which tests to perform and what the results mean.

        Perhaps developing a discussion board framework for
        iteratively linking symptoms with appropriate clinical tests while providing explanations for what the findings mean would be a huge improvement over current day practice. So maybe get a community of CLSs, find out how to incentivize them, post some symptoms, order appropriate clinical tests, and repeat. Soon best practices and patterns will emerge etc, and you will be on your way. You make money by facilitating the ordering of the correct clinical test. Leave the treatment to to the docters.

        The holy grail would be to develop a system that can
        make more refined medical predictions (a challenge of prediction), AND then translate them into actionable treatment decisions (a challenge of decision
        analysis). Maybe I know how to do this, and perhaps I will tell you how to do so for free, but the business model, economics/incentives won’t scale, and your
        financial sponsors won’t think there is a credible business there because there isn’t. So maybe rather than building a prediction market that will never work, perhaps go and build a discussion board community frequented by CLSs, that help patients to tell their idiot doctors which tests to perform

        Keep it simple. you asked for feedback and here it is, download PHPBB, find a few CLS to review patient histories/symptom descriptions, order some tests, repeat, and then report back to us here … and we will tell you want to do next …

        see how it works Jared, no need for math: symptom description, test, feedback, more tests … eventually root cause discovery.

        Good luck.

    • IMASBA

      Russ, I wouldn’t be so damning if I were you. According to economics 101 theory no good can come from free software either, yet a lot of good things come from it in practice. There will be skilled people out there with motivations other than the prospect of making money. Yes, this will be culturally and time dependent but CrowdMed is operating in a certain time frame in a certain culture, it’s not being forced on the whole universe forever.

      • Stephen Diamond

        Also, maybe Jared Heyman shouldn’t be overly concerned about the commercial judgment of someone who thinks people really like ads.

      • russ andersson

        IMASBA – you are right. I regret being so outspoken in my criticisms. The team is doing their best and I wish them well. The point I was attempting to make is that to build a business of scale, one needs to have strong incentives at every point in the delivery chain. These incentives are usually financial, but they can also be status, altruistic, entertainment, etc. The evidence suggests that free/low paying/altruistic prediction markets don’t work and structures that supply enough incentives to scale have yet to be discovered.

        Your point about free software needs further analysis. Little software is truely “free” i.e nobody pays … these companies usually have a business model where somebody is paying, (the economic buyer/true customer) while users use it for “free.” There is a clear difference between users and customers. Google is a good example of that, users use it for free, but advertisers pay. There is also the fremium approach whereby users get a “free” version of the software, but for it to be useful, they need to upgrade and pay for premium versions. Open source is built on consulting, value added services, and upgrades. RedHat, mySQL etc are not free if you want to use them in any meaningful way. Its the old gillette story of giving away the razors but charging for the blades. So software tends not really free in the full sense of the word, its free to some, but somewhere money is usually changing hands.

      • IMASBA

        I used to create elaborate, high quality mods for videogames that I shared online for free and I know plenty of useful programs that were created 100% for free by everyone involved. Believe, me, free software works. What it doesn’t do is work all the time, at any place, in any culture and at any scale, that is something we have to keep in mind and it’s why someone like you would generally (and most of the time rightfully so) advice against betting on it. It cannot be commercialized, generalized, or reproduced on demand, just like memes and virals can’t. CrowdMed can work, just don’t expect it to work for a 100.000 employee company in China 20 years from now, but that’s not a problem, it will help some people in the here and now (which is the goal) and gather data that can be used for future diagnostics software.