Mobile search A/B test instrumentation - technical enablers
Open, HighPublic2 Estimated Story Points
Actions

Description

Background

Our operative metric for the mobile search A/B test is:

"Users who interact with this feature will have 5% longer sessions than those who do not".

In order to measure this, we need to instrument the search results to allow for this. This task focuses on identifying and completing any technical work that is a prerequisite for the A/B test instrumentation.

User story

As an analyst on the Web team, I want to be able to measure the impact of recommended articles on session length, so I can determine success/failure of this experiment.

Requirements

Create a standard definition of session length (Consult with @phuedx on session length-related mix-ins) T378072
Create an instrumentation specification in consultation with @jwang T378103
Create a new Metrics platform schema to capture data for this A/B test

BDD

For QA engineer to fill out

Test Steps

For QA engineer to fill out

Design

Acceptance criteria

An instrument following the spec defined in https://phabricator.wikimedia.org/T378103 has been created

Communication criteria - does this need an announcement or discussion?

See https://phabricator.wikimedia.org/T378106

Rollback plan

This test should apply only to the treatment group / should not impact the control
We should be able to remove this instrumentation in the event of test failure

This task was created by Version 1.2.0 of the Web team task template using phabulous

Details

Other Assignee: Jdrewniak

	Subject	Repo	Branch	Lines +/-
	Search Overlay: Fires events at certain points in workflow	mediawiki/extensions/MobileFrontend	master	+27 -5
	Introduce sessionLength mixin	mediawiki/extensions/WikimediaEvents	master	+214 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T379707 [Hypothesis] [3.1.8] Content Discovery: Article-Level Recommendations
Open	None	T378094 [Epic] A/B test search suggestions on mobile
Open	KSarabia-WMF	T378109 Mobile search A/B test instrumentation - technical enablers
Open	Jdrewniak	T378103 Search recommendations A/B test: Write recommendations instrumentation spec

Event Timeline

NBaca-WMF created this task.Thu, Oct 24, 4:29 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Oct 24, 4:29 PM

NBaca-WMF added a parent task: T378094: [Epic] A/B test search suggestions on mobile.Thu, Oct 24, 4:30 PM

NBaca-WMF mentioned this in T378115: Implement A/B test bucketing for mobile search recommendation.Thu, Oct 24, 4:58 PM

NBaca-WMF mentioned this in T378117: Search A/B test: Add MoreLike-based article suggestions when activating search bar in mobile.Thu, Oct 24, 5:17 PM

ovasileva added a subtask: T378103: Search recommendations A/B test: Write recommendations instrumentation spec.Thu, Oct 24, 5:54 PM

Jdlrobson moved this task from Incoming to FY2024-25 Q2 Sprint 3 on the Web-Team-Backlog board.Thu, Oct 24, 9:41 PM

Jdlrobson edited projects, added Web-Team-Backlog (FY2024-25 Q2 Sprint 3); removed Web-Team-Backlog.

ovasileva added a project: FY2024-25 KR 3.1 Content Discovery.Mon, Oct 28, 5:50 PM

Jdlrobson edited projects, added Web-Team-Backlog (FY2024-25 Q2 Sprint 4); removed Web-Team-Backlog (FY2024-25 Q2 Sprint 3).Mon, Oct 28, 5:51 PM

KSarabia-WMF updated the task description. (Show Details)Thu, Oct 31, 7:47 PM

Change #1087253 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/WikimediaEvents@master] [WIP] Session length mixin POC

https://gerrit.wikimedia.org/r/1087253

gerritbot added a project: Patch-For-Review.Mon, Nov 4, 9:40 PM

@KSarabia-WMF, @NBaca-WMF - just wanted to flag that this is somewhat blocked by the spec in T378103: Search recommendations A/B test: Write recommendations instrumentation spec with the order there being:

Talk to @phuedx about possibilities with session length
Schedule discussion with @jwang on limitations of the spec
@jwang creates spec
Review spec for feasibility
Set up a ticket for implementing spec

Change #1088629 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/MobileFrontend@master] [WIP] Search Overlay Event Tracking

https://gerrit.wikimedia.org/r/1088629

Jdlrobson moved this task from Needs analysis to Ready for estimation on the Web-Team-Backlog (FY2024-25 Q2 Sprint 4) board.Tue, Nov 12, 5:17 PM

NBaca-WMF added a project: Web-Team.Wed, Nov 13, 2:49 PM

NBaca-WMF set the point value for this task to 2.Wed, Nov 13, 5:50 PM

NBaca-WMF moved this task from Product Backlog to FY2024-25 Q2 Sprint 4 (Nov 13 - Nov 26, 2024) on the Web-Team board.Wed, Nov 13, 6:32 PM

NBaca-WMF edited projects, added Web-Team (FY2024-25 Q2 Sprint 4 (Nov 13 - Nov 26, 2024)); removed Web-Team.

KSarabia-WMF claimed this task.Thu, Nov 14, 5:12 PM

KSarabia-WMF moved this task from Ready for Development to In Progress on the Web-Team (FY2024-25 Q2 Sprint 4 (Nov 13 - Nov 26, 2024)) board.

KSarabia-WMF updated Other Assignee, added: Jdrewniak.

Jdrewniak renamed this task from Instrument impact of recommended articles on session length to Mobile search A/B test instrumentation - technical enablers.Fri, Nov 15, 3:44 PM

Jdrewniak updated the task description. (Show Details)

Jdrewniak updated the task description. (Show Details)Fri, Nov 15, 3:53 PM

Change #1088629 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@master] Search Overlay: Fires events at certain points in workflow