The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was  Approved.

Operator: Maximilianklein (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 18:50, Monday, January 7, 2019 (UTC)

Automatic, Supervised, or Manual: automatic

Programming language(s): Python

Source code available: https://github.com/notconfusing/hostbot-ai

Function overview: User:Jtmorgan and User:Maximilianklein have planned, and received consent to run an A/B experiment between the current version of HostBot and a newly developed-AI version. The AI version uses a machine-learning classifier based on ORES to prioritize which users should be invited to the TeaHouse whereas the current version uses rules. The point is to see if we can improve user retention by turning our attention to the most promising users.

The two versions would operate simultaneously. Both versions would log-in as "User:HostBot" so that the end-users would be blinded as to what process they were interacting with.

The A/B experiment would run for 75 days (calculated by statistical power analysis).

Links to relevant discussions (where appropriate): Wikipedia_talk:Teahouse#Experiment_test_using_AI_to_invite_users_to_Teahouse

Edit period(s): Hourly (AI-version) and Daily (rules-version)

Estimated number of pages affected: ~11,000

Exclusion compliant (Yes/No): No

Already has a bot flag (Yes/No): Yes

Function details: All technical details on meta:Research:ORES-powered_TeaHouse_Invites.

Just posting here to confirm that I am excited to collaborating with Maximilianklein on this experiment. I've been wanting to improve HostBot's sampling criteria for a while now, and other Teahouse hosts have asked for it. J-Mo 19:33, 7 January 2019 (UTC)[reply]

Thought I'd drop by to voice my support, both for the experiment and for Maximilianklein. During the earlier discussion, I posted a couple of question on their talk page and got both a timely and thoughtful reply. I'm also interested in learning about the outcomes of this experiment, looking forward to them! Cheers, Nettrom (talk) 15:20, 15 January 2019 (UTC)[reply]

• So are you looking to be approved for a 75-day trial? Primefac (talk) 00:37, 20 January 2019 (UTC)[reply]

  @Maximilianklein and Jtmorgan: ^^ --TheSandDoctor ^Talk 17:48, 21 January 2019 (UTC)[reply]

  @TheSandDoctor:, a 75-day trial would be excellent, thank you. Maximilianklein (talk) 20:01, 21 January 2019 (UTC)[reply]

  @Primefac: ^^ Dreamy Jazz 🎷 ^{talk to me | my contributions} 11:03, 22 January 2019 (UTC)[reply]

• Approved for trial (75 days). Please provide a link to the relevant contributions and/or diffs when the trial is complete.. Primefac (talk) 19:47, 22 January 2019 (UTC)[reply]

 Comment: - HostBot seems to be having a few issues. Which version is this? See here. RhinosF1^(chat)_{(status)(contribs)} 08:21, 18 February 2019 (UTC)[reply]

 Resolved- They were on trial version. RhinosF1^(chat)_{(status)(contribs)} 07:54, 19 February 2019 (UTC)[reply]

Next check in: April 7. — xaosflux ^Talk 12:02, 19 March 2019 (UTC)[reply]

@Xaosflux, Maximilianklein, and Jtmorgan: Its after April 7 - the 75 days should be over, right? --DannyS712 (talk) 00:55, 10 April 2019 (UTC)[reply]

{{OperatorAssistanceNeeded}} what are the results of the trial? — xaosflux ^Talk 00:56, 10 April 2019 (UTC)[reply]

@DannyS712 and Jtmorgan: thanks for the ping. Indeed 75 days are over. Is it possible to ask for a 25-day extension? The reason being that it took some time to de-bug the new bot while it was live, so it was not operating 100% correctly the first few days. The 75-days came from a power-analysis, so I would like to have a pristine 75-days of data to analyse. If that's not possible I understand and Jtmorgan and I can put things back like they were until we can analyse the results. The results of the trial so far are that our two-bot-version co-ordination plan worked in practice. As far as whether the AI-powered HostBot had a higher efficacy at inviting question-asking users, or user that survived longer, I still have to crunch those numbers. Maximilianklein (talk) 23:16, 10 April 2019 (UTC)[reply]

I'd also appreciate a 25-day extension, if possible. If we have a clean 75 day sample, we can make stronger claims. Better Teahouse invite targeting could have a substantial positive impact on the Teahouse, and on retaining good faith newcomers in general. Cheers, J-Mo 18:39, 11 April 2019 (UTC)[reply]

@Jtmorgan, Maximilianklein, and Xaosflux: It's now 10 days since an extended trial was requested. I find this task quite interesting and helpful. Any update on whether it can continue? RhinosF1^(chat)_{(status)(contribs)} 10:41, 21 April 2019 (UTC)[reply]

Approved for extended trial (200 edits or 30 days). Please provide a link to the relevant contributions and/or diffs when the trial is complete. — xaosflux ^Talk 15:05, 21 April 2019 (UTC)[reply]

Thanks Xaosflux and RhinosF1. Maximilianklein. 25 days after April 7 is May 2. Let's shut off the experiment then. Once you turn off your script, post here to confirm and I'll resume inviting 100% of invitees the 'old' way while you analyze the pilot results. Cheers all, J-Mo 20:39, 22 April 2019 (UTC)[reply]

@Jtmorgan and Maximilianklein: Good Luck with the rest it the trial, I hope it works!!

Jtmorgan In slightly unrelated terms, I hope my research about new users will be have stage 1 done by mid next week. Ping me on IRC if you want to discuss that. RhinosF1^(chat)_{(status)(contribs)} 20:44, 22 April 2019 (UTC)[reply]

@Jtmorgan:. Will comply. I'll be pinging you soon. Maximilianklein (talk)

Xaosflux and Maximilianklein I have disabled my "odd number check" so HostBot will start inviting all eligible users on a once-per-24-hours cadence (default setting) starting tomorrow, May 3. Cheers, J-Mo 20:44, 2 May 2019 (UTC)[reply]

 Approved. no issues were reported. — xaosflux ^Talk 15:23, 27 May 2019 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.