It is of course problematic, not to mention silly, to separate “literature” from”spec fic” or “genre” (the latter two which I will treat as synonyms from hereafter). Many great (and some not so great) books fall firmly in both camps. (Murakami, Atwood, Bulgakov, Marquez, to name a few of the great, in my opinion.) But both people I actually know and people I encounter in the online world often reveal a prejudice toward literature by referring to things like “better prose” “more readability” and possessing “higher standards.”
Readability is completely subjective and so too are higher standards. But I actually completely agree that prose in literature tends to be more emphasized, of a more polished nature. (Which doesn’t always mean better– writers like Flynn, Franzen, Sedaris, and Vowell all personally leave me cold.) But is that the sole criterion of good writing? I would argue no. Even writers I absolutely worship like Graham Greene, Raymond Carver, Chuck Palahniuk, etc are simply lacking creativity compared to spec fic writers. I mean, by definition, they lack a speculative element.
Sometimes less polished prose aside, writers of spec fic are doing everything else that lit writers are (character development, plotting, dialogue) on top of creating an entire new world for their characters. This can involve new flora, fauna, weather, cultures, technology, people, currencies, languages, and so much more.
In terms of imagination, what we call literature is drastically lacking. While you could uncharitably criticize genre fiction as favoring dark lords and farmboys (aka “pixies and dragons“) over polished writing, it’s equally true (and equally unfair) to say that literature skews overwhelmingly toward middle-class angst (aka misery porn) is kind of boring, no matter how prettily the words are put together.
Unless you would argue that prose quality should be the single criterion to judge all books, that’s why I would dispute the use of terms like “higher standard.” Especially when each emphasizes an entirely different standard. Different strokes for different folks, with no need for elitism.