Long ago when I first started teaching undergrads, I noticed that while they were equally bad at math and writing, they could more easily see that they were bad at math. So they were happier to get writing assignments. Math assignments made them face the uncomfortable fact that they were bad at something valuable, while writing assignments did not. So students liked my classes better when I assigned writing more.
Science fiction stories are usually set in a possible future world, and you might think that some people would specialize in working out the details of such worlds, while others would specialize in setting stories in those worlds. But while people (like me) who work out future scenarios are well aware that we are not good at writing stories in those worlds, it seems that the people who are good at writing stories don’t believe that they need any help figuring out the details of their worlds. Early in their career most sf authors have already collected a lifetime supply of story settings for their future stories; they have little interest in collaborating with world builders.
In the world of ideas, some people are especially good at finding and exploring interesting ideas, and some people are especially good at writing about such ideas in ways that are compelling and engaging to wide audiences. The people who are good at ideas usually know that they aren’t so good at writing about them, are generally interested in collaborating with those better at writing. But for the most part, the people who are good at writing well to wide audiences are not much interested in such collaboration. Those good writers mostly believe that the ideas that they have are among the best; no need to work together with idea people, as they are idea people.
In general, we see distorted behaviors resulting from the fact that some abilities are both respected and hard to judge. The people with the easier to judge abilities tend to assume that they are also the best at those other opaque but respected abilities. And so our world is full of people who rise to prominence because they are best at what we can more easily judge, who tell themselves and us that they are also best at important things that are much harder to judge. And we seem willing to believe, even if this seems quite implausible.
Worldbuilding ends up as role-playing game settings instead of novels.
So far as story-telling goes, it's widely considered that characterization > plot > world-building. So, unfortunate as it is for world-building autists like us, top story-tellers don't miss out much from having somewhat unrealistic worlds if they get the first two right, which encompass what is widely considered to be the essence of literary skill.