Upgrade GNU Mailman from 2.1 to Mailman3
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	AzaToth
	Jul 6 2013, 2:48 PM

Description

As a Wikimedian I want to be able to follow discussions (read, reply, create) per project or theme in a convenient way, whether through email client on a (mobile) device, webmail on a (mobile) device, or through a web interface of the discussion system itself, so I'm up to date informed about what is going on and can join the conversation anytime as I like.

Alternatives to consider:

Keep Mailman 2.1
Migrate to Mailman 3.0 which has a new Django-based web user interface for end users and list administrators named Postorius (not yet officially sanctioned by GNU)
Consider Discourse as web interface for Mailman mailing list (requires development of synchronization)
~~Consider (flow enabled) talk pages on wiki - add support for reply by email to topics on a talk page for example~~

Not a user story (original task description)
We should update Mailman to version 3.

The new version, among others, stores hashed passwords, which could have minimized the impact of last weeks security incident

Details

Reference: bz50864

	Subject	Repo	Branch	Lines +/-
	lists: Stop mailman2 service	operations/puppet	production	+4 -1
	lists: Stop routing mail to mailman2	operations/puppet	production	+0 -41

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved	Security	None	T181803 Stop storing Mailman passwords in plain text
Resolved		None	T118641 Implement proper AAA for lists.wikimedia.org (mailman)
Resolved		None	T190054 List archives on lists.wikimedia.org is not mobile friendly
Resolved		None	T115329 "From" at start of line becomes ">From" in pipermail
Resolved		None	T52864 Upgrade GNU Mailman from 2.1 to Mailman3
Resolved		None	T130554 Official support for upgrade from existing Mailman 2.1 lists to Mailman 3
Resolved		None	T256536 Puppetize mailman3
Resolved		Ladsgroup	T256542 Puppetize mailman3 web and hyperkitty (mailman archiver)
Resolved		Marostegui	T256538 Create test databases for mailman3
Resolved		None	T256539 Figure out a way to sync old and new mailman
Resolved		Legoktm	T256541 Fix the problem with gravatar and mailman3
Declined		Ladsgroup	T276687 Backport hyperkitty 1.3.4 for buster
Resolved		bd808	T257270 Request creation of mailman VPS project
Resolved		Ladsgroup	T258365 Setup Mailman3 in Cloud VPS
Resolved		bd808	T259444 Request for creating a DNS record for lists.wmcloud.org to 185.15.56.28
Resolved		Legoktm	T276686 Requesting a test VM in production for mailman3
Declined		None	T276697 Implement central logging for mailman3
Resolved		Dzahn	T276712 Request for creation of mailman3-roots group
Resolved		Legoktm	T277263 Enable CSP for mailman3
Resolved		Legoktm	T277286 Make puppet for mailman3 ready for production
Resolved		Legoktm	T278280 Setup monitoring for mailman3
Resolved		Legoktm	T278352 Figure out dkim for mailman3
Resolved		Ladsgroup	T278609 Import several public mailing lists archives from mailman2 to lists-next to measure database size
Resolved		Legoktm	T278610 Install mailman3 on lists1001.wikimedia.org
Resolved		Ladsgroup	T278612 Install mailman3 and mailman2 at the same time on the cloud
Resolved		Marostegui	T278614 Create production databases for mailman3
Resolved		Ladsgroup	T280322 Upgrade mailing lists from mailman2 to 3 in batches
Duplicate		None	T284045 Still getting "lists.wikimedia.org mailing list memberships reminder" email

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

It works now, you can try it in https://lists-beta.wmflabs.org

I haven't managed to get the archive working but you can now join mailing lists and send mail!

I made the archiver work and you can now see it: https://lists-beta.wmflabs.org/hyperkitty/list/test-high-volume@lists.beta.wmflabs.org/thread/TOFSYCOMTGUWZPXNZGGIK3TBRCYAKAQJ/

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

In T52864#6242222, @Ladsgroup wrote:

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

It might be simpler (not to mention more user-friendly) to set up something like https://github.com/ThomasLeister/gravatar-privacy-proxy.

In T52864#6242301, @Tgr wrote:

In T52864#6242222, @Ladsgroup wrote:

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

It might be simpler (not to mention more user-friendly) to set up something like https://github.com/ThomasLeister/gravatar-privacy-proxy.

Having that in production probably requires a bigger discussion (like with security and SRE and legal). For the beta cluster instance, that's a piece of cake.

Ladsgroup renamed this task from Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 to Upgrade GNU Mailman from 2.1 to 3.3.Jun 27 2020, 3:35 PM

After talking to @herron we decided that we start the upgrade (slowly) and hopefully we will get it deployed and upgrade in a couple of months (maybe a year). I do this in my volunteer capacity so please be kind to me. I start by creating subtasks.

Daimona awarded a token.Jun 27 2020, 4:20 PM

Izno subscribed.Jun 27 2020, 4:28 PM

Kizule awarded a token.Jun 28 2020, 9:57 AM

MarcoAurelio mentioned this in T257118: Beta cluster has reached its quota.Jul 5 2020, 9:53 AM

Ladsgroup mentioned this in T257270: Request creation of mailman VPS project.Jul 6 2020, 10:28 PM

Ladsgroup added a subtask: T257270: Request creation of mailman VPS project.

jcrespo mentioned this in T257241: "Uncaught bounce notification" from Yahoo and AOL.Jul 8 2020, 10:03 AM

bd808 closed subtask T257270: Request creation of mailman VPS project as Resolved.Jul 9 2020, 9:02 PM

Ladsgroup renamed this task from Upgrade GNU Mailman from 2.1 to 3.3 to Upgrade GNU Mailman from 2.1 to Mailman3.Aug 8 2020, 11:51 PM

Jony subscribed.Aug 9 2020, 6:12 AM

Krd subscribed.Aug 9 2020, 7:21 AM

Kaartic awarded a token.Aug 9 2020, 12:03 PM

Gaurav awarded a token.Aug 9 2020, 5:26 PM

Gaurav subscribed.

AntiCompositeNumber subscribed.Aug 9 2020, 6:53 PM

MusikAnimal awarded a token.Aug 9 2020, 11:55 PM

Titodutta awarded a token.Aug 11 2020, 5:51 PM

Asaf awarded a token.Sep 16 2020, 8:08 PM

jhsoby mentioned this in T263248: Mojibake on Mailman.Sep 18 2020, 12:26 PM

CptViraj subscribed.Oct 15 2020, 7:04 AM

Ladsgroup mentioned this in T261031: Several unreadable mailing list descriptions (Mojibake) due to wrong charset encodings, should be Unicode.Oct 18 2020, 3:25 PM

Meno25 subscribed.Oct 18 2020, 10:20 PM

Ladsgroup mentioned this in T271123: Mailman password reminder mail (and other texts) has broken encoding in Czech.Jan 6 2021, 7:15 PM

Ladsgroup closed subtask T258365: Setup Mailman3 in Cloud VPS as Resolved.Jan 23 2021, 5:34 AM

Ladsgroup mentioned this in T274953: Access group for Gitlab contractors.Feb 23 2021, 9:28 PM

Akoopal subscribed.Feb 28 2021, 8:17 PM

R4356th subscribed.Mar 3 2021, 3:16 PM

Dzahn closed subtask T276712: Request for creation of mailman3-roots group as Resolved.Mar 8 2021, 10:56 PM

Tks4Fish subscribed.Mar 9 2021, 4:54 PM

Legoktm closed subtask T276686: Requesting a test VM in production for mailman3 as Resolved.Mar 19 2021, 8:19 PM

Ladsgroup mentioned this in T278078: Request for access to mailman3-roots role for Ladsgroup.Mar 22 2021, 7:18 AM

Marostegui closed subtask T256538: Create test databases for mailman3 as Resolved.Mar 22 2021, 7:25 AM

Ladsgroup closed subtask T256536: Puppetize mailman3 as Resolved.Mar 24 2021, 6:34 AM

Legoktm closed subtask T278352: Figure out dkim for mailman3 as Resolved.Mar 24 2021, 8:57 PM

Legoktm closed subtask T277286: Make puppet for mailman3 ready for production as Resolved.Mar 25 2021, 1:49 AM

Ladsgroup closed subtask T276687: Backport hyperkitty 1.3.4 for buster as Declined.Mar 25 2021, 7:32 AM

Legoktm closed subtask T256541: Fix the problem with gravatar and mailman3 as Resolved.Mar 25 2021, 7:44 AM

AdHuikeshoven awarded a token.Mar 25 2021, 8:17 AM

danshick-wmde awarded a token.Mar 25 2021, 11:57 AM

tobiaswiese subscribed.Mar 25 2021, 1:53 PM

Legoktm added a parent task: T118641: Implement proper AAA for lists.wikimedia.org (mailman).Mar 25 2021, 9:55 PM

In case you missed the announcement, a test server with mailman3 is now available: https://lists.wikimedia.org/pipermail/wikitech-l/2021-March/094382.html

Legoktm added a parent task: T190054: List archives on lists.wikimedia.org is not mobile friendly.Mar 26 2021, 7:48 AM

Legoktm closed subtask T277263: Enable CSP for mailman3 as Resolved.Mar 26 2021, 8:25 PM

RoySmith unsubscribed.Mar 26 2021, 8:27 PM

Legoktm added a parent task: T115329: "From" at start of line becomes ">From" in pipermail.Mar 26 2021, 9:12 PM

Gaurav unsubscribed.Mar 27 2021, 1:08 AM

Ladsgroup closed subtask T256539: Figure out a way to sync old and new mailman as Resolved.Mar 27 2021, 11:02 AM

Ladsgroup mentioned this in T278616: Request for adding Ladsgroup to mailman-admins group.Mar 27 2021, 11:19 AM

Thanks for taking care of this hard task! I hope we can then get rid of all Google Groups. ;-) (Is there a task for importing these already?)

Swap URLs of lists-next.wikimedia.org to lists.wikimedia.org and lists.wikimedia.org to lists-old.wikimedia.org (make sure archive URLs don't break)

So what's the plan here? Redirect all requests to (public) pipermail-style URLs from lists.wikimedia.org to a static copy on lists-old.wikimedia.org after the switch? (Remember not to regenerate the archives from the mbox, or the new archives will have different IDs.)

Redirecting requests to /pipermail should do most of the job but I'm not sure what to do with the mbox files and the private list archives currently served under the /private path. Preserving links to private archives would be nice, as it will otherwise be a pain to find the equivalents in the new archives. However a simple static mirror cannot handle authentication and translating from old URLs to hyperkitty URLs seems hopeless (unless we have some idea how those thread and message IDs are created).

In T52864#6970300, @Nemo_bis wrote:

Swap URLs of lists-next.wikimedia.org to lists.wikimedia.org and lists.wikimedia.org to lists-old.wikimedia.org (make sure archive URLs don't break)

Note that this is no longer the plan. In discussion on T256539: Figure out a way to sync old and new mailman the plan is to serve both mailman2 and mailman3 from the same server, so we can gradually migrate lists individually. This is tracked as T278610: Install mailman3 on lists1001.wikimedia.org. As mentioned in the announcement, lists-next is a temporary testing ground and will be fully deleted once we're satisfied with the process.

Redirect all requests to (public) pipermail-style URLs from lists.wikimedia.org to a static copy on lists-old.wikimedia.org after the switch? (Remember not to regenerate the archives from the mbox, or the new archives will have different IDs.)

In theory hyperkitty actually supports redirecting pipermail URLs (https://gitlab.com/mailman/hyperkitty/-/blob/master/hyperkitty/views/compat.py) but we need to continue testing the migration process to see whether it's good enough. FWIW the mailman mailing lists preserved their old pipermail archives and just left a note on top saying that the archives are no longer updated.

Redirecting requests to /pipermail should do most of the job but I'm not sure what to do with the mbox files and the private list archives currently served under the /private path. Preserving links to private archives would be nice, as it will otherwise be a pain to find the equivalents in the new archives. However a simple static mirror cannot handle authentication and translating from old URLs to hyperkitty URLs seems hopeless (unless we have some idea how those thread and message IDs are created).

We cannot continue to support /private URLs because as we're getting rid of the mailman2 authentication system. So either a redirector will need to be in place or we break links. Note that mailman3 has fulltext search, which hopefully will help finding old mails. If you have the Message-ID, you can use the hyperkitty API to get the new archive URL: T256539#6456966.

Qgil unsubscribed.Apr 6 2021, 8:05 AM

Ladsgroup closed subtask T278609: Import several public mailing lists archives from mailman2 to lists-next to measure database size as Resolved.Apr 8 2021, 12:32 AM

Legoktm closed subtask T278280: Setup monitoring for mailman3 as Resolved.Apr 20 2021, 7:10 PM

I just sent an announcement to all list administrators: https://lists.wikimedia.org/pipermail/listadmins/2021-April/000344.html

You can now access mailman3: https://lists.wikimedia.org/postorius/lists/

LGBT is now upgraded. More mailing lists will follow soon.

More info: https://lists.wikimedia.org/pipermail/wikitech-l/2021-April/094459.html

Legoktm closed subtask T278610: Install mailman3 on lists1001.wikimedia.org as Resolved.Apr 28 2021, 7:58 AM

the plan is to serve both mailman2 and mailman3 from the same server [...] FWIW the mailman mailing lists preserved their old pipermail archives and just left a note on top saying that the archives are no longer update

So the public HTML and txt.gz archives will keep being served indefinitely from the same URLs as before? That will reduce the pain considerably.

In T52864#7068604, @Nemo_bis wrote:

the plan is to serve both mailman2 and mailman3 from the same server [...] FWIW the mailman mailing lists preserved their old pipermail archives and just left a note on top saying that the archives are no longer update

So the public HTML and txt.gz archives will keep being served indefinitely from the same URLs as before? That will reduce the pain considerably.

I mostly figured out how to redirect public HTML URLs, see T280731: Implement static redirects from pipermail archives to hyperkitty archives. Should get deployed tomorrow or early next week.

I don't think hyperkitty has an equivalent for the txt.gz files, so we can keep them around (by not deleting them) if there's value.

hyperkitty allows you to download gzip files (click on "Download" in https://lists.wikimedia.org/hyperkitty/list/listadmins@lists.wikimedia.org/2021/5/) Thankfully, building a redirect for it would be pretty easy. Doesn't need any hash.