The replication failure in psychology is inevitable. The whole field boils down to taking an incredibly complex system (human beings) ... and measuring a few variables on them across a few people.
Most of the "statistical" justifications stem from assuming distributions are normal and samples are uniformly collected (both false).
This is why I trust the works of clinical psychologists like Jung and Freud so much more seriously than the current approaches.
It just makes a lot more sense to observe something complex descriptively, rather than start with a yes/no hypothesis and then test it with a few dozen people.
This 'study of reproducibility ' found the opposite: different sampling generally failed to produce different results. These contrived lab situations are remarkably relevant to the larger population, which has quite limited variation.
The issue was sloppy science, not a poor approach.
This interpretation of the reproducibility crisis is not supported by this massive international study, at least according to the last paragraphs of the original article (which quote someone directly addressing your concern).
The results they found were that the studies that could not be successfully reproduced generally failed across all tested cultures, while the ones that could be successfully reproduced generally worked across all tested cultures. So it seems that there are some results in statistical psychology that are robust, but that researchers have not been sufficiently reliable at identifying which results those are.
I'm not an expert on experimental psychology, but I'm fairly certain that the principles of how experiments are conducted in the sense that they control for other factors, choose sufficiently large samples etc., are pretty sound. An often mentioned problem in psychology research is "p-hacking", where researchers manipulate the data in a way that validates their hypotheses statistically, which seems like a more logical reason for a reproducibility crisis than the reasons you mentioned.
Also just because there are some problems in the field should not cause you to dismiss science completely in favor of completely subjective speculating about how the mind works, which certainly also has value, but I don't think it's warranted to dismiss actual science in favor of it.
But they're just not -- that's exactly my point. If you test a bunch of Harvard undergrads (the most common subject type in these studies) -- that's not a uniform sample. In any way whatsoever.
Furthermore, even if you get a high p-value ... your experimental design can introduce so much noise.
Even when you do find these high p-value correlations -- they don't really ever say much about "why" ... because they can't. So much of that is the study design.
Let's say I make a study on how well people navigate a maze under the influence of alcohol vs. not. You can probably get a high p-value that they do worse while under the influence of alcohol... but it reveals very little about why.
So much of that is dependent on the particular maze. In fact I bet I could design a maze specifically to prove whatever conclusion I wanted. That is the whole problem in psychology. There is very little focus given to actually characterizing cognition in any meaningful way.
I like the clinical psychologists because they attempt to do exactly that, even though they have less "scientific" findings.
Humans are complicated, but we're also fairly predictable in some respects. Hungry people tend to go get food. The field is about the predictable aspects. We're not impossible to study.
Sure, we tend to want to eat once we're hungry but when we go just a stop above hunger, meaning the desire or non-desire for sexual reproduction, then things get really, really complicated really, really fast. For example just the other day I was reading about a 92-year old guy who was choosing daily sex over him allowing his body to get physically better, while on the other hand we've got asexual 20-year olds from places like Japan who don't really think about sex at all. There's no way for a lab-made psychology experiment to make sense of our libido (or lack of).
But sex is one of the fundamental aspects of the human existence. If psychology cannot say anything meaningful about our sex life then it cannot say anything meaningful about us at all. Yes, it can probably guess when we should purchase or sell some stocks, like people following guys like Kahneman do, but their studies do almost nothing to “illuminate” us on what us humans really are on the inside.
I agree - descriptive studies also matter. I think in the last century, we have overdone statistics. It is time to counterbalance with a more descriptive approach as well.
Most of the "statistical" justifications stem from assuming distributions are normal and samples are uniformly collected (both false).
This is why I trust the works of clinical psychologists like Jung and Freud so much more seriously than the current approaches.
It just makes a lot more sense to observe something complex descriptively, rather than start with a yes/no hypothesis and then test it with a few dozen people.