The following discussion is an archived debate. Please do not modify it. Subsequent comments should be made in a new section. The result of the discussion was Request Expired.

Operator: ShakingSpirit

Automatic or Manually Assisted: Automatic

Programming Language(s): Python (pywikipedia framework)

Function Summary: To detect and notify on talk pages any dead external links found within articles

Edit period(s) (e.g. Continuous, daily, one time run): Weekly to Monthly

Edit rate requested: 30 edits per minute

Already has a bot flag (Y/N): N

Function Details: I noticed that there's a fair number of dead link bots based on the weblinkchecker.py script included with the pywikipedia framework, but (to the best of my knowledge) none are actually up, alive and kicking. After playing with the weblinkchecker.py script for a while it's easy to see why; although a very easy bot to set up, it requires a very large amount of time to run (a few weeks at the very least per sweep) because, even using something like Special:Export, going through every single article on Wikipedia and grabbing links is no small task! Even once it's done this once, it needs to do at least two sweeps spaced apart to check that the link isn't just down temporarily. ShakingBot abandons the idea of checking through every article as it stands, and instead uses the monthly externallinks.sql database dump. It goes though the following process:

1. Check through every article alphabetically in externallinks.sql for bad links, and log any it finds
2. 7 days later, run though this check again, and note any links which were marked 'bad' in both checks
3. Go though the list of 'bad' links and check to see if each link is still used in the 'live' article (as it may have been removed since the dump was taken) - disgard those which aren't
4. Add a note to the article's talk page, letting users know that the link is bad, and should be reviewed or removed

Please note that ShakingBot will not remove links by itself, as some links are important and should be replaced with similar pages or an archived version of the original link - this requires human intervention.
Currently I am working on detecting if ShakingBot previously noted a dead link which hasn't been removed since the last run, and if so, note it to file, and to a noticeboard in my userspace. I hope to have this finnished for if and when ShakingBot is approved.

Seems pretty safe, wanna trial it for a bit -- Tawker 03:12, 27 December 2006 (UTC)[reply]

That would be great, but will need to be for about a month to get in results; it takes a long time as it is, and I'm still optimizing the number of threads it uses for my server's specific CPU and bandwidth ShakingSpirit^talk 03:55, 27 December 2006 (UTC)[reply]

For a trial, choose a select few articles, and have it work on them. No need to have it be the whole encyclopedia. Cheers, ✎ Peter M Dodge ( Talk to Me • Neutrality Project ) 16:25, 27 December 2006 (UTC)[reply]

Two questions:

• Would it be possible (make more sense?) to work with the lists of bad links being generated by the system, at Wikipedia:Dead external links, or to otherwise get a list from the Wikipedia database?

That would be ideal if it was possible, but to the best of my knowledge the lists on Wikipedia:Dead external links haven't been updated for nearly a year now. ShakingSpirit^talk 04:11, 28 December 2006 (UTC)[reply]

• Why not TAG the external link (similar to {{citation needed}}) IN the article, rather than posting to the talk page? John Broughton | Talk 01:05, 28 December 2006 (UTC)[reply]

I thought about tagging them with {{dlw}}, but this doesn't say "please fix me!" like a note on the talk page does, and personally I think it makes inline refs look ugly and confusing. I'd like to hear other people's opinions though ^_^ ShakingSpirit^talk 04:11, 28 December 2006 (UTC)[reply]

Please look at Wikipedia:Dead external links — Iamunknown 01:47, 29 December 2006 (UTC)[reply]

Please look at what I've written above; Wikipedia:Dead external links is very much out of date. ShakingSpirit^talk 05:44, 29 December 2006 (UTC)[reply]

Not sure how I missed that. Maybe you could update the lists with the ShakingBot. I for one still occasionally go through the Wikipedia:Dead external links lists and try to help out. — Iamunknown 20:28, 29 December 2006 (UTC)[reply]

That's certainly a possibility, as I've already completed the code to post persistent dead links to User:ShakingBot/Persistent dead links, and it wouldn't take too much effort to re-organize them to Wikipedia:Dead external links's style and post them there instead/as well ShakingSpirit^talk 07:09, 30 December 2006 (UTC)[reply]

Regarding the inline tag, <{{dlw}}> is long and ugly. I was thinking of something like this: [1]^{[bad link]}. (I've not used a template here, but rather a modified version of Template:Fact, for illustration.) John Broughton | Talk 03:14, 30 December 2006 (UTC)[reply]

I guess that would work, as long as the link wasn't inside a <ref> or somesuch, as that breaks it. I'll try various other ways of tagging and see if it's feasible ^_^ ShakingSpirit^talk 07:09, 30 December 2006 (UTC)[reply]

Combining John Broughton and Iamunknown's ideas above, what about tagging the links with something like ^{[bad link]} and as part of that template, adding the page to a category (in the same way that ^{[citation needed]} adds the page to Category:Articles with unsourced statements)? This has the advantage of the list of bad link pages being dynamic, and doesn't need to have fixed links manually removed and new links added in bulk as the ones on Wikipedia:Dead external links do. Thoughts, opinions? ShakingSpirit^talk 07:23, 30 December 2006 (UTC)[reply]

I think that would definitely be appropriate. It might help stem the tide of what I've experienced in fixing dead links, which is the removal of them. It would keep the content (links) and the process (a link to Wikipedia:Dead external links) in the same spot (the article), whereas previously both the content and process were both totally separate from the spot. Great idea! — Iamunknown 22:34, 31 December 2006 (UTC)[reply]

30 edits per minute? Or do you mean 30 page-gets per? I'm afraid, though, that this seems to very much fall foul of Wikipedia:BOT: "Bots that download substantial portions of the page database are prohibited. Instead, download the database dumps. See also Wikipedia:Mirrors and forks." I'd suggest if you can do that, and then check only the "positive" pages from the dump against the live database, that would be much less of a resource hit. Alai 05:01, 12 January 2007 (UTC)[reply]

What is the status of this bot? —Mets501 (talk) 20:56, 25 January 2007 (UTC)[reply]

Don't pin me to this but I think it is still in test phase. RED skunk^TALK 01:24, 2 February 2007 (UTC) P.S. Shaking Spirit has gone inactive for a while.[reply]

I notice that he "unidled" long enough to remove an image upload notification from his talk page, but given the lack of a response here, or to queries about this matter on his talk page, I think this is certainly at the "archival" stage. Alai 16:53, 8 February 2007 (UTC)[reply]