Fully agree with you that the way much (but not all) of the publication process currently works is bad for everyone involved (including the taxpayer).
But I'd like to push back against the following.
> the way we say "hasn't been peer reviewed" implies that if it has been published in a journal then it's factual.
I think that's ascribing meaning that isn't actually there. Peer review _does_ tell you something. Certainly not "guaranteed correct" or even "guaranteed high quality" but it is definitely some sort of signal to the reader. Consider that this is nothing more than proactively pointing out that a linked paper which closely resembles work that would typically be affiliated with a journal (a signal gated by an editor) be peer reviewed (another signal) and those reviews possibly made public (yet another signal) does not have any of these signals. That is to say, it's an important disclaimer in order to avoid the appearance of attempting to deceive the audience.
I don't think you're wrong, I just don't think those signals are very strong. I do think they used to be stronger, but paper discovery isn't really dominated that way anymore. Word of mouth was always common, but now that's expanded to social media as well as things like Google scholar or semantic. Hell, most journals and conferences are hard to search through. Maybe I'll land on their page from a Google search of the paper or when looking for those topics, but that's not any different from ArXiv. And truth is that I still have to do the same thing when landing in either location. I still have to read the abstract, give it a sniff test, and read it to varying depth depending on what's needed. There's a signal, I just haven't found it to be that meaningful
I never claimed it was useful for targeted searches by modern standards. It's primarily a time saving measure from my perspective. If I come across a paper and it is in a publication that has a track record of publishing shoddy results I am likely to prioritize other works (assuming that I have a large potential pool of them to investigate).
If a nature paper claims a seemingly impossible result I'll most likely spend the time to at least glance through it and see if maybe my preconceptions were mistaken. If something on arxiv looks like nonsense I'm not going to waste any time on it (unless someone else told me it's worth it).
> It's primarily a time saving measure from my perspective.
Your perspective as what? Help me understand the point you are arguing from. A researcher? A general member of the public? Grad student? Something else?
I'm specifically asking this because there are different perspectives from the inside than the outside. There's a lot of internal problems that the general public isn't going to see. I also do not think papers should be aimed at the public. They should be aimed at other researchers. There are spaces for communicating with the public and it uses very different language. But these papers are supposed to be our communication to one another. We do it in the open. Doing this more openly means no gatekeeping, which putting them behind paywalls is also gatekeeping. But we need a space to talk to one another. The best way to do that is out in the open, mostly because we don't know who we're talking to. But just because it is visible by all doesn't mean everyone is our audience.
> If I come across a paper and it is in a publication that has a track record of publishing shoddy results I am likely to prioritize other works
Taking a stab in the dark here, but I'm guessing the answer isn't "researcher" to the above question? The reason I say this is because this seems like a weird scenario. There's so many venues that except those in your specific domain you're probably not going to know what is a reputable venue from one that isn't. It's also not exactly an easy thing to verify. If it is your field, well, you can tell 99% of the time from just the abstract.
> If a nature paper
You may wish to have a look at this conversation[0]. Or look at the controversy section of the wiki which provides more examples of my earlier claim about routinely rejecting good papers.[1] A good place to always look at is Retraction Watch[2,3]
Rejecting good papers is different from accepting bad ones. False negatives (the former) don't waste my time as a reader whereas false positives might. From my perspective the goal (which is a means to an end) is to improve the signal to noise ratio.
Not to sound uncaring but I'm not losing sleep if someone got his feelings hurt because his groundbreaking paper was rejected by whatever high profile journal. That used to be potentially damaging but these days we have arxiv so you're still the original researcher to make the discovery. Suck it up and resubmit elsewhere. Rejection is an integral part of the job.
To answer your earlier question I am sometimes a researcher and sometimes a layman. There are many reasons to consult the academic literature after all.
> There's so many venues that except those in your specific domain you're probably not going to know what is a reputable venue from one that isn't. It's also not exactly an easy thing to verify.
You are massively overthinking this. If of the last N nonsense results 5% of them were in a given journal you bet I'm going to remember the name and associate it with bullshit. I don't need to "verify" anything it's nothing more than a personal bias regarding a given publication venue. Nonetheless I find it (and quite a few other metrics) useful in rapidly prioritizing a list of hundreds of potentially relevant results when deciding which threads to start pulling on for my current question.
That's a more important question than you might think (at least in my experience) since you generally don't have time to pull on all the threads and ask all the new questions that come up in doing so. It's a multi headed search that generally prioritizes depth so the quality of the commentary and references in the papers that you spend significant time on is of the utmost importance to the end result.
But I'd like to push back against the following.
> the way we say "hasn't been peer reviewed" implies that if it has been published in a journal then it's factual.
I think that's ascribing meaning that isn't actually there. Peer review _does_ tell you something. Certainly not "guaranteed correct" or even "guaranteed high quality" but it is definitely some sort of signal to the reader. Consider that this is nothing more than proactively pointing out that a linked paper which closely resembles work that would typically be affiliated with a journal (a signal gated by an editor) be peer reviewed (another signal) and those reviews possibly made public (yet another signal) does not have any of these signals. That is to say, it's an important disclaimer in order to avoid the appearance of attempting to deceive the audience.