Super interesting that you enjoy fiction so much. What I struggle with most is that visual language is often very dense in information, but I can't do a lot with it. Imagine something like this:
"Light spilled in through the high windows, tinting the hallway into beautiful autumn colors. It looked as if the sunlight was dancing, but of course nothing moved except the dust suspended in the air."
I would read this and think: cool, I bet this would look amazing if I could see it, but all the information I can actually use from these sentences is "A hallway has high windows, it's maybe morning or evening". Everything else is either visual or obvious to me. So fiction books are more exhausting, because I constantly filter out things that I can't really use. It's like I'm reading a text where a person constantly rambles and can't get to the damn point. I'm really curious how or why this is different for you? Another thing I find annoying is, that usually when reading fiction books, you constantly have to amend your mental model. I presume this is relatively easy for people without aphantasia, although I might be wrong. Let me explain with this example:
"blah" said A. "blah?" B responded. A said "blah blah" as he stood up from his chair. "blah!" B said back, while A turned right and walked out the door.
This order is the exact opposite my brain expects. I'd like know the room layout and who is sitting/standing where first, then the characters can interact with each other in my already complete internal model. This might be a me-thing, but if non-aphantasia people can image images as easy as I can imagine sounds, making changes to the model must be super easy.
Also, I do think fiction books and non-fiction history books are very different. Simply because an author can build a world, story and characters to convey some deeper meaning or overarching theme, or use strong imagery or metaphores. All of that is more uncommon for historic books from my experience. The above example in a history book would probably look something more like "Orange light entered the hallway through the high windows". And even if non-fiction history books were similar to fiction, history is a tiny part of non-fiction! There are tons of other subcategories that differ greatly from fiction.

This is the second leak-type-meme I've stumbled across and now I'm hooked