I also feel this is true, and I wonder why that is. I remember when Jonathan Franzen turned down Oprah's Book Club, an honor for which most authors would commit actual murder, because he was afraid it would scare away the already dwindling audience of men. Something weird happened in the late 90s.
Here's an incisive writeup in Vox that explains the Franzen controversy and sheds some light on the 'why don't men read fiction anymore' question as well.
Now that you mention it.. Growing up my dad bought books 📚 all the time from Amazon (like I do now) but hardly ever read them. He claimed he was too busy but I always suspected there was more to it.
I used to read fiction voraciously. In my 40s, dealing with a protracted and bitter divorce, I stopped being able to concentrate long enough to read ANYTHING. Now I'm reading more often again, but fiction keeps defeating me. I really don't know why and I hate it.