People would probably buy smaller phones if given the correct choices, I think the iPhone 15 size is good yeah. iPhone is often a fashion statement, so people usually want the highest end they can get and manufacturers tend to put more features in their bigger more expensive models.
There's no options in the mid-range for smaller phones, most are over 6.5 inches. So people are forced into buying them, there's definitely a large market for small phones, but manufacturers force them into larger models. I suspect they are easier/cheaper to make, so that's what we get. Also bigger numbers = better right, so they'd rather crank the marketing on battery, screen etc.
Of course there's some limitations yeah, though the S24 and iPhone 15 have great cameras, more than enough for the majority of people. Most people aren't power users and won't even look at specs in-depth, just a glance at what it looks like, storage and price. Size would definitely be a factor people would consider (and already do when they can) and it's a real shame there aren't more options.
To be honest looking at the teardowns it's impressive what Apple and Samsung have managed to fit in their phones, I think the Xiamoi 14 Ultra isn't doing itself any favours with that camera placement though, seems inefficient having the modules in the middle of the phone?