Most schools assign each student a “grade point average”, i.e., a number that averages over many teacher evaluations of that student. Many schools also assign each teacher an “average student evaluation”, i.e., a number that averages over many student evaluations of that teacher. Many workplaces similarly post evaluations which average worker performance ratings across different tasks. And sport leagues often post rankings of teams, which average over team performance across many contests.
All trends in higher education are to make grades and assessment of students less informative than they were even 30-50 years ago. Look at the SAT, it was renormed in the early 90s so that overnight the number of double Verbal/Math 800s increased sixfold thus making it harder for high scorers to distinguish them from those a tier lower. This change has brought NO complaints from the edu establishment but instead has led them to double down by claming that even the watered down SATs are too discriminatory so that all elites have now eliminated SATs for admission.
A current attempt to that initially started by the promoter of GPCs would be dismissed without a second look by universities. Especially by those high status places that know well how to implement these stat techniques. THEY know how to distinguish good from bad students but want employers to treat all their graduates -- the useful and the useless alike -- as stellar products.
Thanks Robin. My points is that the subpopulation /is/ segmented (mainly: between schools, and then all levels up; the data is very sparse since students only follow classes on one school; the model 'should' go hierarchical here). That segmentation makes the models for proper ranking on skill a lot harder than those in the Github (alas, no data there). Not talking about things like the US 'sub'-populations, although even that would become a theme I imagine if you'd try to run this model at scale.
I have some experience in building models from a research perspective versus building models for a production environment (both private and public). The difference in the amount of work is several orders of magnitude.
An example, say your child has top-5% GPA in their school, but gets a top-20% skill ranking. The school is just not that good. You'd be upset, since that is the difference between Ivy league and no-name university (trying to think US here). So you would try and get to the bottom of the model. You then find out that the models is creating some crazy fixed or random effect on a logit that pretty much dooms all the students in the school. The researcher would say: hey, that's just metrics in action and on the whole the model works better than GPA. The (government?) entity running the model would get sued into apoplexy.
The normal information a college knows, an employer receives, or acquaintances might share with each other is GPA + major. In a way, that is just as efficient as GPC for those who know avg GRE and LSAT scores for each major. In fact, is it not more efficient for communication since it (lossily) separates out what part of this person's profile is subject matter difficulty and what part is consciousness?
"Such statistical models are pretty easy to construct, and most universities contain hundreds of people who are up to this task."
I take some issue with this statement. Creating a model that adequately segments a population reliably and robustly across a very diverse educational landscape (very sparse matrix) is not as trivial as estimating a logistic fixed effects model. You can easily rank results per class, perhaps per school, but properly comparing over schools / districts / subjects while retaining 'fairness' in ranking is a hard problem. Sure, many models will beat the naïve GPA since the drawbacks are so evident. But given the way the outcome will influence real lives, the ethical modeller would need a very high degree of certainty the model approaches the 'true' ranking of ability.
I like the idea and think there are at most a few hundred people in the world that can build this model.
Whilst I'm more bearish on the acceptability, I still think this is a great proposal, but has a chance only if it is as "average-like" as possible: only include dumy variables for subject and student, and nothing else at all. No gender, state, school, race, etc. The intercept and student coefficient add up to the GPC.
Yes, this is exactly right. However, there are other organizations - mostly companies - that are actually looking to identify and invest in real talent. It's interesting that though there's money at stake, corporate decisionmaking about hiring is shockingly primitive. I've long been arguing that if some ambitious companies offered very talented high-school graduates a good alternative to college, the graduates and companies could both benefit.Edit: What I meant to imply, but should have just stated, is that companies doing scouting have every reason to use a system like the one outlined, even if university administrators don't. The only thing that could stop them is a lack of the right sort of data inputs, which will be denied them for "privacy" reasons.
Arizona State University claims to be the most innovative university in the US, (they win this every year https://www.usnews.com/best.... They should try it.
I've long favored this idea, though it doesn't need to be linear regression. Something based on item-response theory would also work. https://mc-stan.org/docs/2_...
Not very hard. We did speak with the Engineering College about it. They already calculate a good enough solution, a "technical GPA" (taken from core STEM courses), which they use as a graduation requirement but don't transcript. They have less need of the cGPA as they have less variation in grading across departments, though.The actual, implemented, solution to reducing departmental grading disparities is probably going to look pretty different. In the College of Liberal Arts and Sciences, Chemistry stands out as having hard classes, and I've heard that they're under pressure from the administration to raise grades. All the incentives line up for them to agree - it's easier for individual professors, students and administrators if grades are higher (less complaints, more degrees, and more tuition). This dynamic means that grades are rising about 0.1 of a Grade Point every decade (https://www.gradeinflation.... - Chemistry's current harsh grading would be average grading in the 1980's. We're running out of headroom on the 4-point scale, which makes even calibrated GPAs less discriminatory. As a consequence, it's only a slight exaggeration to say that in STEM we now pretty much just use calculus to judge academic ability (in grad school applications, for example, which no longer require GREs). I was trying to make progress on a different issue. Workers at other big Midwestern schools had suggested that STEM courses discriminate against women and other groups, as they get lower grades than you would expect in STEM courses, given their GPAs. We developed the calibrated GPA to correct the obvious flaw in this approach (the observed GPA is a bad yardstick as different groups of people take different courses, which are graded differently) and found that there isn't any resolvable discrimination - people do as well as you would expect if you adjust for course difficulty in their overall GPA. STEM courses are harder for everyone.
All trends in higher education are to make grades and assessment of students less informative than they were even 30-50 years ago. Look at the SAT, it was renormed in the early 90s so that overnight the number of double Verbal/Math 800s increased sixfold thus making it harder for high scorers to distinguish them from those a tier lower. This change has brought NO complaints from the edu establishment but instead has led them to double down by claming that even the watered down SATs are too discriminatory so that all elites have now eliminated SATs for admission.
A current attempt to that initially started by the promoter of GPCs would be dismissed without a second look by universities. Especially by those high status places that know well how to implement these stat techniques. THEY know how to distinguish good from bad students but want employers to treat all their graduates -- the useful and the useless alike -- as stellar products.
Thanks Robin. My points is that the subpopulation /is/ segmented (mainly: between schools, and then all levels up; the data is very sparse since students only follow classes on one school; the model 'should' go hierarchical here). That segmentation makes the models for proper ranking on skill a lot harder than those in the Github (alas, no data there). Not talking about things like the US 'sub'-populations, although even that would become a theme I imagine if you'd try to run this model at scale.
I have some experience in building models from a research perspective versus building models for a production environment (both private and public). The difference in the amount of work is several orders of magnitude.
An example, say your child has top-5% GPA in their school, but gets a top-20% skill ranking. The school is just not that good. You'd be upset, since that is the difference between Ivy league and no-name university (trying to think US here). So you would try and get to the bottom of the model. You then find out that the models is creating some crazy fixed or random effect on a logit that pretty much dooms all the students in the school. The researcher would say: hey, that's just metrics in action and on the whole the model works better than GPA. The (government?) entity running the model would get sued into apoplexy.
The normal information a college knows, an employer receives, or acquaintances might share with each other is GPA + major. In a way, that is just as efficient as GPC for those who know avg GRE and LSAT scores for each major. In fact, is it not more efficient for communication since it (lossily) separates out what part of this person's profile is subject matter difficulty and what part is consciousness?
No, don't remove outliners. Normalizing variables makes no difference. There are not missing data in grade transcripts.
I don't see why we'd want to segment populations. See the Added to the post for actual prior stat analyses, which are pretty straightforward.
"Such statistical models are pretty easy to construct, and most universities contain hundreds of people who are up to this task."
I take some issue with this statement. Creating a model that adequately segments a population reliably and robustly across a very diverse educational landscape (very sparse matrix) is not as trivial as estimating a logistic fixed effects model. You can easily rank results per class, perhaps per school, but properly comparing over schools / districts / subjects while retaining 'fairness' in ranking is a hard problem. Sure, many models will beat the naïve GPA since the drawbacks are so evident. But given the way the outcome will influence real lives, the ethical modeller would need a very high degree of certainty the model approaches the 'true' ranking of ability.
I like the idea and think there are at most a few hundred people in the world that can build this model.
Should we remove outliers?Should we normalize variables?What should we do if there is missing data?Etc.
These are not trivial decisions. And there will be inconsistencies across schools.
Colleges are better equipped to calculating all of this. And they are highly incentivized to get this right.
Whilst I'm more bearish on the acceptability, I still think this is a great proposal, but has a chance only if it is as "average-like" as possible: only include dumy variables for subject and student, and nothing else at all. No gender, state, school, race, etc. The intercept and student coefficient add up to the GPC.
Yes, this is exactly right. However, there are other organizations - mostly companies - that are actually looking to identify and invest in real talent. It's interesting that though there's money at stake, corporate decisionmaking about hiring is shockingly primitive. I've long been arguing that if some ambitious companies offered very talented high-school graduates a good alternative to college, the graduates and companies could both benefit.Edit: What I meant to imply, but should have just stated, is that companies doing scouting have every reason to use a system like the one outlined, even if university administrators don't. The only thing that could stop them is a lack of the right sort of data inputs, which will be denied them for "privacy" reasons.
Why doesn't the weather service just give you all their weather data, and let you make your own weather forecasts?
Instead of creating a grade point coefficient, why not just give schools all the data, and let them figure it out themselves?
Arizona State University claims to be the most innovative university in the US, (they win this every year https://www.usnews.com/best.... They should try it.
Yes, I immediately thought of a Rasch IRT as the most direct approach. Same big idea though.
I've long favored this idea, though it doesn't need to be linear regression. Something based on item-response theory would also work. https://mc-stan.org/docs/2_...
Not very hard. We did speak with the Engineering College about it. They already calculate a good enough solution, a "technical GPA" (taken from core STEM courses), which they use as a graduation requirement but don't transcript. They have less need of the cGPA as they have less variation in grading across departments, though.The actual, implemented, solution to reducing departmental grading disparities is probably going to look pretty different. In the College of Liberal Arts and Sciences, Chemistry stands out as having hard classes, and I've heard that they're under pressure from the administration to raise grades. All the incentives line up for them to agree - it's easier for individual professors, students and administrators if grades are higher (less complaints, more degrees, and more tuition). This dynamic means that grades are rising about 0.1 of a Grade Point every decade (https://www.gradeinflation.... - Chemistry's current harsh grading would be average grading in the 1980's. We're running out of headroom on the 4-point scale, which makes even calibrated GPAs less discriminatory. As a consequence, it's only a slight exaggeration to say that in STEM we now pretty much just use calculus to judge academic ability (in grad school applications, for example, which no longer require GREs). I was trying to make progress on a different issue. Workers at other big Midwestern schools had suggested that STEM courses discriminate against women and other groups, as they get lower grades than you would expect in STEM courses, given their GPAs. We developed the calibrated GPA to correct the obvious flaw in this approach (the observed GPA is a bad yardstick as different groups of people take different courses, which are graded differently) and found that there isn't any resolvable discrimination - people do as well as you would expect if you adjust for course difficulty in their overall GPA. STEM courses are harder for everyone.
A great many educators do understand linear regressions, making their results not black boxes to them.