Hi all - jotting down some quick answers to hop

In T378115#10333427, @SToyofuku-WMF wrote:

Questions to guide our implementation/understand scope:

1. Will there ever be more than two groups (control/treatment)?

Not for this A/B test

2. Do we need the ability to do a percentage based rollout?

Yes, if possible since the size of the audience is so large. (not a strict requirement if this would significantly increase the timeline). If not possible, we can rethink the overall wiki distribution and only run the test on a few select wikis.

3. Is there ever a situation where control and treatment are unevenly distributed? (note this makes analysis near impossible)

No

4. Do we want to opt in individual users (us) to the experiment?

If possible, something like a url parameter would be helpful for QA purposes.

5. What is the exact target audience of the experiment (logged out/which wikis/etc)? Will it change?

Logged-out users across Wikipedias.

• If possible, we might want to change the percentage of users exposed to the feature with large wikis (enwiki, eswiki, dewiki, jawiki, etc) showing the feature to a smaller percentage and remaining wikis 50/50.
• If easier, we could also restrict the A/B test to a subset of wikis

All done! Resolving

In T378098#10324207, @bwang wrote:

@ovasileva is there a timeframe we want to have summaries active for?

Just noting that this does the same thing as "search pages containing" on mobile

@RHo, @kostajh - this is now ready from our side so I'll go ahead and resolve the ticket. Let us know if there's any additional questions or follow-ups needed!

In T375152#10260991, @Jdlrobson wrote:

One word of note: Italian Wikipedia is using legacy Vector here - as a result it is not getting the same level of testing and performance monitoring then the rest of Wikimedia projects which uses Vector skin.

@ovasileva @sgrabarczuk are we any closer to enabling Vector 2022 on Italian Wikipedia? That might perhaps turn this task from a high priority to a curiosity.

In T376903#10298241, @SToyofuku-WMF wrote:

Created T379214 - this should now be ready for signoff!

Moving to sprint 3. First we need to get the API results, tracked in T379240: Generate API list for recommended content, will also mark as subtask

@KSarabia-WMF, @NBaca-WMF - just wanted to flag that this is somewhat blocked by the spec in T378103: Search recommendations A/B test: Write recommendations instrumentation spec with the order there being:

@jwang - what do you think about doing the following:

1. We proceed with instrumentation in statsd for:
2. out of all pageviews where summaries are displayed, X% of pageviews had a summary opened?
3. Total number of clicks to “yes” and “no”
4. For any additional nice to have instrumentation, we have a separate conversation on whether we want to introduce it after the experiment launch in a separate ticket.

I think we're all done here! Additional filtering can happen in a new ticket if necessary, or directly in the doc

Blocked on T374883: Section collapsing should not apply to certain pages so deploy will happen in next sprint.

@KSarabia-WMF to create an epic as a parent of this ticket and other recent metrics-platform-related work.