Discussion about this post

User's avatar
disqus_2YvG85xdvS's avatar

All trends in higher education are to make grades and assessment of students less informative than they were even 30-50 years ago. Look at the SAT, it was renormed in the early 90s so that overnight the number of double Verbal/Math 800s increased sixfold thus making it harder for high scorers to distinguish them from those a tier lower. This change has brought NO complaints from the edu establishment but instead has led them to double down by claming that even the watered down SATs are too discriminatory so that all elites have now eliminated SATs for admission.

A current attempt to that initially started by the promoter of GPCs would be dismissed without a second look by universities. Especially by those high status places that know well how to implement these stat techniques. THEY know how to distinguish good from bad students but want employers to treat all their graduates -- the useful and the useless alike -- as stellar products.

Expand full comment
Willem Kerstholt's avatar

Thanks Robin. My points is that the subpopulation /is/ segmented (mainly: between schools, and then all levels up; the data is very sparse since students only follow classes on one school; the model 'should' go hierarchical here). That segmentation makes the models for proper ranking on skill a lot harder than those in the Github (alas, no data there). Not talking about things like the US 'sub'-populations, although even that would become a theme I imagine if you'd try to run this model at scale.

I have some experience in building models from a research perspective versus building models for a production environment (both private and public). The difference in the amount of work is several orders of magnitude.

An example, say your child has top-5% GPA in their school, but gets a top-20% skill ranking. The school is just not that good. You'd be upset, since that is the difference between Ivy league and no-name university (trying to think US here). So you would try and get to the bottom of the model. You then find out that the models is creating some crazy fixed or random effect on a logit that pretty much dooms all the students in the school. The researcher would say: hey, that's just metrics in action and on the whole the model works better than GPA. The (government?) entity running the model would get sued into apoplexy.

Expand full comment
21 more comments...

No posts