Missing Credentials

The typical modern credential (i.e., standard worker quality sign of widely understood significance) is based on a narrow written declared test of knowledge given early in one’s career on a pre-announced date at a quiet location. In this test, there is a list of questions to which one gives answers, answers then graded by independent judges who supposedly look only at the answers, and don’t take into account other things they know about the testee. In this post I want to point out that a much larger space of credentials are possible.

For example, you could be evaluated on actual products and contributions, based on your efforts over a long period, instead of being evaluated on short tests. You could be tested via tasks you must perform, instead of questions you must answer. After all, mostly we want to know what workers can do, not what questions they can answer. Since much of real question answering in the world is done verbally, test question-answering could also be done verbally, instead of in writing. And it could be done with frequent distractions and interruptions, as with most real question-answering.

However expressed, judges could take your first response as a starting point to ask you more questions (or give you more tasks), and dig deeper into your understanding. Judges could know you well, and choose questions specifically for you, and interpret your answers given all they know about you. This is, after all, closer to how most question-answering in the world actually goes.

Tests could be done at random days and times, and spread all through your career. Tests might be disguised as ordinary interactions, and not revealed to be tests until afterward. These approaches could discourage cramming for tests and other strategies that makes you good only at tests, and not so much at remembering or using your knowledge at other times.

Finally, you could be tested on your ability to integrate knowledge from a wide range of topic areas, instead of on your knowledge of a narrow topic area. Yes you could show that you know many areas via passing tests for many areas, but that won’t show that you have integrated these diverse areas usefully together in your mind.

Of course I’m not saying that these variations are never explored, just that they are used much less often than the standard credential test. This vast space of possible credentials suggests that a lot of innovation may be possible, and I’m naturally especially interested in helping to develop better credentials for abilities that I have which are neglected by the usual credentials. For example, I’d love to see a polymath credential, for those who can integrate understanding of many fields, and a conversation credential, on one’s ability to get to the bottom of topics via a back & forth interaction.

The narrow range of most credentials compared to the vast possible space also seems to confirm Bryan Caplan’s emphasis on school as emphasizing and screening conformity. Yes the the usual kinds of tests can often be cheaper in many ways, but the lack of much variation even when credentials are very important, and so worth spending a bit more on, suggests that conformity is also an issue. It really does seem that people see non-standard tests as illicit in many ways.

The dominance of the usual credential test can also be seen as a way our society is unfairly dominated by the sort of writing-focused book-smart narrowly-skilled people who happen to be especially good at such tests. These people are in fact usually in charge of designing such tests.

GD Star Rating
loading...
Tagged as: ,
Trackback URL:
  • buybuydandavis

    Government should turn away from funding higher education to funding credentials.

    There is infinite free information and training that people could use to prepare for those credentials.

  • dagon_net

    I don’t know how many resumes and CVs you look at, but test results are a trivially small part of modern credentials. References, accomplishments, prior positions, credit rating, and publications matter far more for most adults.

    • http://www.gwern.net/ gwern

      Indeed. If we were speaking of ‘modern credentials’, we would have to consider that the overwhelming majority come in the form of educational degrees.

      > The typical modern credential is based on a narrow written declared test
      of knowledge given early in one’s career on a pre-announced date at a
      quiet location.

      This description does not fit the high school graduation, associates degree, bachelor’s degree, master’s degree, PhD, JD, MD, or pretty much any kind of higher educational credential I can think of. The closest might be the GED, which is 4 tests – except ironically, that’s also penalized by the labor market as being worse than the high school graduation. The JD does have the bar exam and historically one could take that without a degree or after an apprenticeship, but increasingly few states allow that. Within the degrees, math and physics have a reputation for use of high-stakes testing, but even there, they’re only a tiny fraction of degrees granted and the tests are typically only a fraction of the grade.

      • http://overcomingbias.com RobinHanson

        Most school degrees require one to pass many tests in specific classes.

      • http://www.gwern.net/ gwern

        You’re engaging in a motte-and-bailey now. Most degrees use a wide variety of low-stakes tests administered in many formats (often not multiple-choice or administered at specific times as single high-stakes tests) as one of many parts of the requirements to graduate but frequently test scores are not even in the official list – most college graduation requirements are about *credit-hours*, specific combinations of courses taken, average grades not being abysmal, and (of course) being fully financially paid up after one’s 4 years of time invested. SAT-style testing is merely one element and not even obviously the majority compared to the other things like writing papers (not tests). A very far cry from

        > The typical modern credential is based on a narrow written declared test
        of knowledge given early in one’s career on a pre-announced date at a
        quiet location.

        No. They’re not.

      • Vitalik Buterin

        In most courses in university that I took, 50-55% of the final grade came from a final exam, and another 25-30% from a midterm. Everything else was never more than 20%, and often less. Of course, there are many courses that you have to take with tests for each one and it’s not One Big Standardized Test to Rule them All, but Robin’s description does resonate with me.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        How else but examinations to test performance in classes consisting of lectures and reading? Seems to me your and Robin’s quarrel should be with the methods of education rather than how its mastery is credentialed.

        Written exams aren’t very important in graduate school, where the objective is to prepare the student for an academic career. Consistent with that objective, conducting research and writing research papers is emphasized. It’s not clear to me that the use of written tests is well tailored to the objectives of the college-level curriculum – transmitting declarative knowledge.

      • Vitalik Buterin

        Personally, I think surprise tests (and more tests, not just one or two per semester) would have been a marginal improvement over pre-planned exams, as they create less perverse incentive to game the system by cramming, though I know that some people disagree.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        In high school, I always resented surprise exams as paternalistic. [Where I went to college, class attendance wasn’t even mandatory.]

      • http://www.gwern.net/ gwern

        It doesn’t resonate with me. In most of my courses, the final exam made up a decent chunk (often ~50% like you), but many instead had final papers, and the ones where there was a final exam, it was never multiple choice standardized tests like Robin claims. They were typically long essays (philosophy, social sciences), free response (derivations in math courses), or computer graded (program submissions run against test suites in CS). This was over 3 different programs in 3 colleges, 2 public and 1 private.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        I didn’t read Robin as saying college examinations were usually multiple choice standardized exams.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        So why exactly didn’t you say that to begin with?

    • http://overcomingbias.com RobinHanson

      Among those other items, only a credit rating can be seen as a credential which doesn’t need much local knowledge to interpret.

  • Lord

    This is largely a numbers and cost issue. A teacher with 20 or 30 students much less 200 or 300 or 2000 or 3000 can’t spend that much time and what time they could spend could easily end up superficial and subjective. It becomes easy to spend more time assessing than learning, defeating the point. Beyond those mass settings these other methods are often used, whether for hiring, promotion, publishing, prizes, and probably amount to more in total and certainly more in importance after a few years afterwards. There is a problem with them though, in that they become very difficult to comparable. There is an incompatibility between diversified assessment and common standards, and you can’t easily arrive at the latter through the former.

  • Scott Draper

    preform = perform?

    • http://overcomingbias.com RobinHanson

      Yup; fixed; thanks.

  • Anonymous

    >It really does seem that people see non-standard tests as illicit in many ways.
    People are always looking for someone to blame when something goes terribly wrong, so diverging from the standard option is risky. (“We failed our best students? Bad luck; we just did what all schools do.”)

    If a company thinks it can gain from employees that are good at a certain non-standard test, they should be able to subsidize the schools that perform that test (indirectly, by explicitly being biased in favor of graduates from those schools).

  • Robert Koslover

    Life is a continuing test, upon which some perform far, far better than others. The set of those who succeed in their overall life tests only partially overlaps the set of those who score well on various official, authoritative, and/or standardized tests. Some, especially those who may do comparatively well early in life on formal tests but perhaps less well later on other measures of overall life success, consider this situation to be terribly unfair. It is important for society as a whole to never allow such bitter/resentful folks to become too powerful; they tend to hold the common man in contempt and will often not act in society’s best interests.

    • consider

      Life is a continuing test? An overall life test?? You don’t believe St. Peter is up there debating whether you are currently performing at a cumulative B+ or A- level, right? (Only St. Pete gets a straight A)

      We are all roaming the Earth for a very short time and then die (except for Robin who will be frozen first) – not long enough for an evaluation and who would we hand it into anyway?

      • Robert Koslover

        Heh. Actually, I really wasn’t trying to speak from a religious perspective, but I can understand why you might see it that way. Rather, by saying that life is a test, I’m just trying to point out that essentially every choice one makes, and every action (or inaction) one takes, has consequences. The test is whether you ultimately decide or act in a way that advances your goals, your value to others, or any other figures of merit that matter to you. Ultimately, you must judge whether you have passed or failed any particular test. That said, others, and nature, may have a strong impact on that conclusion. Some of life’s little tests may turn out to be VERY important after all, so… choose wisely! E.g., https://vimeo.com/54484129

      • consider

        I disagree with much of this for several reasons including the sheer bad luck many people run into. Life is not a series of controlled tests and there is often much more arbitrariness than in school tests.

      • Robert Koslover

        I didn’t mean to assert life’s tests were well controlled. And yes, plenty of life’s tests have a strong component of luck. But as they say, “that’s life.”

  • Sid

    A major advantage of the quiet, sit-down, written test is fairness – both perceived and actual; but especially perceived fairness. The rest of the suggestions for testing– especially the continuous, day-to-day type of testing – are perceived to be easily hacked by in-groups.

    This is not to say that the evaluation of standard testing cannot be unfair. But I think it is perceived (and I think mostly correctly) to be fair.

    • http://overcomingbias.com RobinHanson

      What sort of hacking do you have in mind?

      • Sid

        Well, nothing fancy. You may feel, rightly or wrongly, that the judges unfairly rated you poorly compared to someone else – someone, say of the judges’ own race/country/gender/college/city – then you cannot easily appeal the judges’ decision. Especially so if the testing is some kind of sum over everyday interactions. It’ll just be your word against theirs.

        Whereas if there is a written test with a (quasi)-objective standard for right and wrong, then you can appeal the judges’ decision to a higher third-party. So if the judges did want to tilt the scales, they can’t do much short of tampering with your answers (which is hard).

      • http://overcomingbias.com RobinHanson

        Ordinary tests also have judges, so that can’t be the difference. I think you are suggesting that there would be less inter-judge reliability in their evaluations with these alt credentials. But that isn’t obvious.

      • Sid

        Yes, I’m trying to point out the possibility of less inter-judge reliability. But also that ordinary testing involves hard-to-tamper records of the test answers that can be examined by other judges.

  • Tom Hynes

    Military officer promotion is good example. Some of it is testing, some of it is officer fitness reports, some of it is doing well in a battle.

  • Ryan Reynolds

    Isn’t this description of frequent personalized testing a description of what actually happens in life anyway? There’s no shortage of people saying they don’t have any formal education but that didn’t hold them back, and succeed anyway, and are (one might argue correctly) evaluated on their commercial or entrepreneurial success and not for their lack of formal credentials.

    The converse is also true: a highly regarded degree from an expensive school is discounted heavily if the bearer of the degree is otherwise a total failure elsewhere in life. I would argue that the test itself is not the sole, or often even the primary identifier of social worth.

    Or, to rephrase that, it’s not clear to me that society is dominated by book learnt but otherwise clueless people. This seems like a testable proposition which should be measured before merely stating it as fact.

    • http://overcomingbias.com RobinHanson

      All this “frequent personalized testing” doesn’t make a standard credential whose meaning is widely understood.

      • Ryan Reynolds

        Is the objective to arrive at a single accurate standardized measure of worth? That sounds like a fever dream for a central planner, not an efficient market.

        There is a reason why there are a range of different tests, with different perceived values: because people disagree on their fundamental value. There’s no magical valuation formula for a corporation, why should a skill set be any different?

      • http://overcomingbias.com RobinHanson

        The idea is to substitute for existing credentials based on the standard sort of tests.

  • Pingback: Recomendaciones | intelib

  • Wat

    Credentialing in general is used as a sorting tool for social darwinistic applications. There is no assumption, as there was more or less in aboriginal cultures, that the testee will retain a place of standing in the society in general without some specific credentialing. We just let people go into the wilderness if you don’t “earn your place”. This is a root of violence in our society, among other pathologies. I think everyone should be assured of a dignified, secure existence if they meet the simplest test of being a good faith societal participant.

  • Joe

    The domination of school as primary credential might also be a function of a huge government subsidy one kind of test (school) crowding out other forms of credentialing.

  • http://www.truthcoin.info/ psztorc

    No! : )

    The credential only works at all, because the decision-maker knows exactly what it means. One’s “ability to take a simple test” might not mean very much, but at least it is the ~same test for everyone.

    The credential is, itself, a solution to a signal-to-noise problem. If the tests involve this much randomness and chaos, I would probably just ignore them altogether.

    And this process is supported by resume, letters of recommendation, awards, etc.

    > It really does seem that people see non-standard tests as illicit in many ways.

    Yes, there is an inherent ulterior motive (unfortunately). Anyone for whom the current system isn’t working has an incentive to roll the dice on a new system.

    • http://overcomingbias.com RobinHanson

      But it isn’t at all obvious that overall these alternative approaches have more noise relative to signal.

      • http://www.truthcoin.info/ psztorc

        I think so. Until I (the decision maker) the receiver of the signal, am familiar with it, the singal is 100% noise.

  • https://twitter.com/CyborgTribe Kevin Hill

    As there are as many credentials as judges, the problem then becomes one of finding the best judges of performance and then finding out their ratings on particular people.

    Sounds a lot like networking to me…

  • Frederic Bush

    There are too many opportunities for corruption and discrimination when the tests are arbitrary. There’s a reason that governments use formal, written tests for the civil service.

    • http://overcomingbias.com RobinHanson

      Who said anything about arbitrary?

      • Frederic Bush

        Would you prefer Kafkaesque?

        A lot of the ideas you propose involve giving a lot of flexibility to the judge to decide the terms of the test. That is arbitrary, at the whim of the arbiter.

        To me the exemplar of the improved credentialing task is the blind audition for an orchestra, which is successful because it removes a factor from the judge’s control rather than granting the judge more control. The more control the judge has the more the successful applicant is going to resemble the judge in ways that do not involve competence at a task.

  • Unanimous

    I don’t think society is dominated by people who excel at exams. In fact qualifications don’t appear very relevant at all.

  • http://www.quora.com/Richard-Treitel/answers Richard T

    Combining this with another idea found in your recent posts, what if judges were to express their predictions by speculating in a futures market on testees’ future performance? Apart from the obvious problem of testees bribing judges, I mean.

  • Richard Kennaway

    Sounds like an interview followed by a probationary appointment and on the job assessments. Are you suggesting something new here, or seeing if anyone recognises the status quo when described in unfamiliar terms?

    • http://overcomingbias.com RobinHanson

      The job assessments would have to be standardized so they could be meaningful to people in very different contexts.

  • Ronfar

    Something much like what you have described here has been implemented in engineering schools as a “senior design project”…