I do think variances in reporting are some of it. I have some visualization skills and wouldn't claim to be "aphantasic", but on the flip side, there is an enormous dividing line between real-world images and these imagined images. While awake I would never confuse the two.

I also suspect a lot of people are massively overreporting the quality of their images. It's very easy to imagine something and think you understand it, until asked to produce something based on it. This isn't just limited to mental images, it's the source of the "why don't you 'just'..." questions, and comes up all the time in building, engineering, artistic endeavors, etc. Your brain is easily convinced it has a thought with more detail than it actually has. I really only trust people who have taken a lot of time to train the relevant system if they claim they have these things. For instance, I really can design some somewhat non-trivial programs in my head... but I've been doing this for ~25 years, and only recently would I say I'm getting to the point where these designs are sometimes good enough that they're not taking serious body blows betwixt conceptualization and realization.