The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was Approved.

Operator: Nettrom (talk · contribs)

Automatic: Specify whether supervised or unsupervised --> Automatic or Manually assisted: Automatic, supervised

Programming language(s): Perl, Python (with pywikibot), SQL

Source code available: Currently not

Function overview: Post suggestions to a limited number of new users in research project

Links to relevant discussions (where appropriate):

Edit period(s): daily, over a period of eight weeks

Estimated number of pages affected: 500

Exclusion compliant (Y/N): Y

Already has a bot flag (Y/N): Y

Function details: SuggestBot already posts suggestions to talk pages of users who have requested to get them. As part of a research study, we want to post a suggestions to a limited number of new users in order to see if we can make them contribute more than they otherwise would. Through statistic analysis of en-WP data we have identified some criteria which we think identify the more prolific users, and now want to test our hypothesis with an experiment. We will not post suggestions to users who have been warned or banned for vandalistic edits, or similar actions. Unless the users in the experiment specifically ask for regular suggestions, they will only have received a one-time posting on their user talk page.

I'd just like to point out that SuggestBot has already been the subject of two published research articles. The first is "SuggestBot: Using Intelligent Task Routing to Help People Find Work in Wikipedia" from 2007, and the second is "The Diffusion of a Task Recommendation System to Facilitate Contributions to an Online Community." from 2009. Nettrom (talk) 20:37, 11 June 2010 (UTC)[reply]

This seems to be directly in line with the proposals at WP:Research. If nothing else, the people working on that proposal should be invited to participate in this BRFA. Please do so. Were I unaware of that proposal, I would say "go get strong consensus at WP:VPR, advertised at WP:CENT, WP:AN, and anywhere else you can think of, and come back when that's done", which IMO would be a rather higher hurdle. Anomie⚔ 21:32, 11 June 2010 (UTC)[reply]

I have made a post to WP:Research and invited anyone interested to participate, thanks for the suggestion! The other option does sound like a slightly difficult hill to climb, yes. Nettrom (talk) 01:35, 12 June 2010 (UTC)[reply]

Can you please provide some bibliographic details for those articles (direct links, preferably)? ElKevbo (talk) 22:13, 19 June 2010 (UTC)[reply]

Sorry for not seeing this sooner. You can find an abstract of the first article through our research lab's home page, SuggestBot: (…) -- GroupLens Research, and also through ACM's Digital Library: [1] The second article was published in Journal of Computer-Mediated Communication, and an abstract should be available through [2]. Let me know if there's additional questions. Cheers, Nettrom (talk) 19:34, 25 June 2010 (UTC)[reply]

More importantly, I'd also like to point out that I have made a similar request on Norwegian Wikipedia, as we would like to run a parallel study across both language versions of Wikipedia, also using SuggestBot's Norwegian sister bot AnbefalingsBot. If you read Norwegian, the request and its discussion is here (if needed I can summarise it). Cross-language Wikipedia research like this is currently rare, which is one of the motivations for this study. Nettrom (talk) 13:26, 12 June 2010 (UTC)[reply]

Does the bot care if the user has been previously welcomed or not? Note that some welcome messages are also warnings. Gigs (talk) 22:40, 13 June 2010 (UTC)[reply]

In the process of selecting users, we will check for the common warning templates on their user talk page, and if any is present they'll be excluded from the study. Apart from that I've made a note to make manual checks of their user talk page before we post suggestions, as the number of users is small enough to allow for that. Are you also wondering if our suggestions will incorporate a welcoming message to the users? I've thought of that, and am considering having a group that gets it, and one that doesn't, to see if it has any effect, but the presence of other users and their welcoming messages might make it difficult, so I'm not totally sure. Nettrom (talk) 02:09, 14 June 2010 (UTC)[reply]

Well, yes welcoming first might be good. Just be aware that some of the welcome templates are also warnings, like {{welcome-coi}}. Gigs (talk) 02:58, 14 June 2010 (UTC)[reply]

Ah, I wasn't aware of that, thanks for pointing them out! I've found several others too, and will make sure we take those into account, in addition to the manual checks. Nettrom (talk) 15:00, 14 June 2010 (UTC)[reply]

This request sounds reasonable and low-risk for editors. I have some general concerns about this process but I'll post them elsewhere so as not to derail this discussion. ElKevbo (talk) 22:13, 19 June 2010 (UTC)[reply]

Approved for trial (50 edits or 5 days). Please provide a link to the relevant contributions and/or diffs when the trial is complete. MBisanz ^talk 02:40, 23 June 2010 (UTC)[reply]

Since the whole task is only 200-250 pages, and suggest-bot is well established, I would suggest approving the whole task. That way, base-lining dates and so forth will be easier for the researchers. Rich Farmbrough, 07:02, 23 June 2010 (UTC).[reply]

After having discussed the study again with my advisers it appears we might have to scale our sample size, perhaps doubling it. While that might appear to be a huge amount of users, I'd like to note that it's roughly 1.7% of our identified population size, and about 1% of active registered users, so we'll be leaving most Wikipedians to themselves. I'll have more exact numbers after next week's meeting. I'd be happy to do a 50-user trial to demonstrate that we'll be targeting the right users and stay away from disruptive behaviour, though. Cheers, Nettrom (talk) 19:46, 25 June 2010 (UTC)[reply]

Update 2010-06-29: After having revisited our statistics and calculations for the size of the study, we have found it necessary to increase our size to about 500 users, and increase the experiment period to eight weeks (we originally planned for four). Revisiting the numbers I posted on June 25, the time extensions cuts the percentages in half, thus we'll be reaching less than one percent of either user base, meaning we're still leaving most Wikipedians alone.

Unless there are concerns raised, I will commence a 50-user trial as soon as possible. We're currently fixing a data gathering bug in SuggestBot and should have that resolved in a couple of days. Nettrom (talk) 18:13, 29 June 2010 (UTC)[reply]

Suggestions: One of the concerns raised about the process outlined in Wikipedia:Subject Recruitment Approvals Group and Wikipedia:Research was the potential annoyance factor of making uninvited requests for assistance to a volunteer group who are often already quite stretched. Using SuggestBot as a means of recruiting research subjects is a good idea, as people who receive SuggestBot have already opted in for suggestions on how they might further assist Wikipedia. I see this a good way forward.

• As a means of recuiting more subjects, might it be possible for SuggestBot to also contact users who opt in for researching?
• Additionally, could SuggestBot, as a one off when the trial is succesfully completed and the project is given the green light, include in its message to all users a link for people to opt in for future research projects. This would have the benefit of picking up users who read the talkpage of users who receive SuggestBot. SilkTork *^YES! 17:50, 1 July 2010 (UTC)[reply]

For this study we aim to discover if our suggestions have an effect, and have therefore made preparations to make sure that is the only thing we're looking for. Adding a message asking users to opt in will only make things more difficult for us, for instance by triggering undesired side effects, and therefore I do not wish to do that.

SuggestBot is built with a specific purpose in mind, while the recruitment process you are describing is of a more general nature. I am not sure SuggestBot is the right bot to do that job, and would instead recommend discussing that kind of problem over on Wikipedia Talk:Research or Wikipedia Talk:Subject Recruitment Approvals Group. Cheers, Nettrom (talk) 20:13, 6 July 2010 (UTC)[reply]

Sorry - I made a mistake. I was under the impression that this was a trial run to recruit research subjects. I see that is not the case. Please disregard my remarks. SilkTork *^YES! 09:25, 8 July 2010 (UTC)[reply]

Trial complete.

Update 2010-07-07: We've now completed the 50-user trial. See SuggestBot's contribution history, edits starting from 16:05 on July 2 to 11:23 on July 7, with the exception of User talk:Sillybillypiggy, User talk:Birdy god, and User talk:White Shadows, which were all regular requests for suggestions posted on User:SuggestBot/Requests and thus use a different template. Cheers, Nettrom (talk) 16:42, 7 July 2010 (UTC)[reply]

{{BAGAssistanceNeeded}}

Update 2010-07-19: I'd appreciate some response on this request from a BAG member at this point. Before the trial started we also approached the Welcoming committee members to notify them of our intentions. Neither there nor here (nor from the WP:Research crowd) have we seen any strong opposition from the community, and if there are any issues that the BAG members think haven't been discussed or some actions we haven't taken, I'd appreciate getting to know about them so I can work on it. Cheers, Nettrom (talk) 15:48, 19 July 2010 (UTC)[reply]

 Approved. MBisanz ^talk 04:48, 26 July 2010 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.