Vanity publishing and predatory publishing are two models which allow publication without any of the normal processes of editorial or peer review.

In vanity publishing, the author usually pays for the publication of the material, which is usually these days delivered via print on demand, incurring little to no cost to the publishing house. Vanity presses often have no selection criteria and provide none of the normal services of a publishing house, such as legal review, proofreading, copy editing or fact checking. There is a list of vanity presses and self-publishing houses at Wikipedia:List of companies engaged in the self-publishing business. The best known is probably lulu.com.

In academia, vanity publishing has also been referred to as "write-only publishing". There may be no charge to the author, and the publisher may make their money by selling copies at high prices solely to libraries of record (large university libraries, the Library of Congress etc). Publishers called out for this practice include Lambert Academic Publishing, IGI Global^[1][2][3] and Edwin Mellen Press. These books may publish fringe thought and may be used to inflate the reputation of otherwise mundane authors.

Predatory open access publishers charge high transaction fees to authors and publish based on payment, not credible peer review. This came to prominence in part through the work of Jeffrey Beall, who maintained "Beall's List" of predatory publishers and journals. Beall was silenced by legal threats from a publisher he had criticised. New models are emerging for assessing journal quality, including the Directory of Open Access Journals (DOAJ) and Cabell's blacklist.

Red flags for predatory publishing include:

• Unheard-of publishers with dozens of journals in unrelated areas
• Groups of journals with very similar or overlapping scope
• Journals with absurdly broad subject areas
• Journals with few published articles per issue, or few published issues
• Journal name very similar to a more established journal (e.g. Journal of Social Sciences vs Journal of Social Science)
• Location of the publisher/editorial board not matching with the title of the journal. For example, a mostly Indian editorial board for a journal named American Journal of .... Or an exclusively Chinese editorial board for a journal called International Journal of ...
• The topic of the article does not match the topic of the journal, for example a mathematics article published in a journal of medicine, or vice versa
• No impact factor or use of fake impact factor and other misleading metrics (see [1])
• Absent from standard indexes for similar journals (e.g. medical journals not indexed in PubMed, astronomical journals not indexed in ADSABS, open access journals not indexed in DOAJ)
• Indexed only in low-selection (Google Scholar) or sham (Index Copernicus) databases
• Cited in hoax publication tests
• Listed by Beall's List or Cabell's blacklist
• No DOI number, invalid ISSNs
• Spammed by WP:SPAs, especially where usernames match author names
• Online reports of spam solicitation for papers

Keep in mind that a journal meeting one or more of those criteria does not guarantee that it is predatory, but the more criteria are met, the likelier it is. Predatory publishing is also a relatively new phenomenon, so publications established prior to the 2000s are unlikely to be predatory.

In the real world, it is possible that a work published in non-peer reviewed venue like a vanity press or a predatory journal represents excellent scholarship. One can write an absolutely correct and rigorous analysis of a topic that would pass peer-review and would be considered authoritative in the field by experts, but publish it elsewhere for one reason or another. For example, Grigori Perelman submitted his proof of the Poincaré conjecture to arXiv, an unreviewed preprint repository. Perelman's proof has been examined by many mathematicians, who certified it as a correct and brilliant proof, for which he was offered the 2006 Fields Medal (he declined). In this case, Perelman's papers are considered reliable sources, because independent reliable sources consider them to be excellent scholarship.

Vanity presses venues can also by used by honest researchers for non-nefarious print on demand purposes, like a reference work intended for distribution in a handful of research centres and libraries, with no commercial potential.

However, Wikipedia cannot conduct such an expert analysis of sources, and must instead rely on the analysis of other experts in the field. Even if you are personally qualified to conduct such an analysis, on Wikipedia it would be considered original research, and is not allowed. Nonetheless, some non-peer reviewed sources are known to be more likely than others to be reliable based on their reputation.

• arXiv – a repository of eprints, mostly from published experts affiliated with recognized institutions. Each section is moderated to filter unserious and nonsense scholarship. Moderation is not equivalent to peer-review, but the papers hosted on arXiv typically represent an honest (and sometimes preliminary) attempt at scholarship (i.e. preprints). Many will become published; many won't. Some papers are even postprints, in which case they are as reliable as the publication venue that accepted them. Wikipedia cannot establish "X was discovered in 2020" based on a preprint paper, but it could be used to write "A team of researchers claimed to have discovered X". Other considerations like WP:DUE and WP:SPS will apply however.
• viXra – very much like arXiv, except without any moderation or control on who can submit a paper. This leads to the vast majority of experts avoiding to upload their research there, because they do not want to be associated with papers like viXra:0912.0036, leaving the venue to be mostly used by the fringe.

We know there is a difference in reliability of these two venues because arXiv papers are extensively cited in research and that recognized experts make extensive use of this venue, while viXra is almost universally ignored by the research community. In the words of Gerard 't Hooft:^[4]

When a paper is published in viXra, it is usually a sign that it is not likely to contain acceptable results. It may, but the odds against that are considerable.

On Wikipedia, we, unlike scholars, cannot assess individual published works. We must instead rely on the reputation of the venue in which something is published. Preprints in reputable repositories like arXiv, bioRxiv, PeerJ Preprints, socArXiv and so on, are typically treated as self-published expert sources and can only be used to support basic uncontroversial claims. Vanity presses and predatory journals are, on the other hand, treated like viXra above, for both failing to conduct proper review, and for their willingness to publishing the work of researchers who are actively avoiding peer review. And, like with viXra above, while it is possible to have good research published in a predatory journal or vanity press, it is unlikely. In those cases Wikipedia errs on the side of caution and does not consider those references to be valid, unless it can be demonstrated that a particular paper or book is recognized as reliable in reliable, independent and secondary sources.

• Wikipedia:Reliable sources/Noticeboard
• Wikipedia:Wikipedia CiteWatch – The CiteWatch, a list of questionable sources cited on Wikipedia. See large disclaimer at the top of the page before using.
• User:JzG/Predatory – A list of questionable sources, with links to search for DOIs and URLs, associated with those sources.
• Wikipedia:Unreliable/Predatory Source Detector – The Unreliable/Predatory Source Detector, a user script to facility the detection of questionable sources cited on Wikipedia. See large disclaimer at the top of the page before using.