Everyone is talking about Chat GPT. Let's put it to the test: "Can you write a Wikipedia article (English Wikipedia) for the Mussel Inn in Onekaka? Complete with references to secondary reliable sources using Wikipedia code ready for inserting into Wikipedia?"

ChatGPT: "Unfortunately, I cannot write a complete Wikipedia article as I am an AI language model and my training data only goes up until 2021. However, I can provide you with the information and structure of a typical Wikipedia article, along with examples of how to properly format and cite references. You can then use this information to write the article."

It then gives me a sample article. I've pasted that onto the article page.

The structure of the article is good. The language – not so. I would tag it with "advert". It gives a couple of sources and one is secondary and independent (recent article in the NZH) and the other the entry in the Lonely Planet. The first fact mentioned in history is wrong. Maybe I was just given a typical structure. Let's see. I'll suggest some sources to Chat GPT; I wonder whether it'll cope with that. I also want something a bit longer; I was given 1180 Bytes of readable prose. Schwede66 00:55, 1 February 2023 (UTC)[reply]

Me: "The Mussel Inn is much older than what you say; it was founded on 5 December 1992. Can you please read the article on page 8 of https://issuu.com/charlotterichards3/docs/2022_december02gbweekly (where the 30th birthday party is reported on), use this as one of the suggested sources, and write a longer article of about 3500 bytes of prose?"

The resulting article is shown on User:Schwede66/ChatGPT2. That's 2064 of readable prose the way Wikipedia defines it or 2372 characters when I drop this into word. Maybe I should get it to read up on readable prose size.

Me: "That article is shorter than what I intended it to be. Could you please read https://en.wikipedia.org/wiki/Wikipedia:Article_size to learn how Wikipedia computes article size, and then produce an article of about 3500 bytes of readable prose?"

ChatGPT: "Certainly! Here is a revised article for the Mussel Inn in Onekaka, New Zealand:" I'll put this at User:Schwede66/ChatGPT3. Schwede66 01:17, 1 February 2023 (UTC)[reply]

Well, that didn't work. The resulting article is of the same length, but ChatGPT has dropped the section headings for some reason (they are excluded from the prose size calculation, but I didn't ask it to suppress the headers). Ok, ChatGPT isn't good at reading and interpreting Wikipedia guidance pages. Maybe it works better when I spell things out: "Your article sounds like an advert. Please use neutral language, present the information in a formal and factual way. And please reinstate the headings." Here's the resulting article. I shall draw some conclusions:

Overall, ChatGPT is mostly useless at writing a Wikipedia article. Concerningly, it makes up references (the NZH article does not exist). It does not read policy pages and thus cannot learn from that. When I give it a reference to read, it doesn't appear to be able to. It does appear to be able to learn from information that I give it. The only useful output was an article that had a meaningful structure. All in all, it appears that us Wikipedia volunteers aren't out of a job just yet. Schwede66 01:41, 1 February 2023 (UTC)[reply]