Change #1067283 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] testwikis to 1.43.0-wmf.20

https://gerrit.wikimedia.org/r/1067283

Change #1067283 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis to 1.43.0-wmf.20

https://gerrit.wikimedia.org/r/1067283

Mentioned in SAL (#wikimedia-operations) [2024-08-27T09:25:32Z] <hashar> train: fast forwarded mediawiki/core wmf/1.43.0-wmf.20 from 1faf18d6570 to ef87455d7c3 # T366965

Change #1067300 had a related patch set uploaded (by TrainBranchBot; author: trainbranchbot):

[mediawiki/core@wmf/1.43.0-wmf.20] Branch commit for wmf/1.43.0-wmf.20

https://gerrit.wikimedia.org/r/1067300

Change #1067300 merged by jenkins-bot:

[mediawiki/core@wmf/1.43.0-wmf.20] Branch commit for wmf/1.43.0-wmf.20

https://gerrit.wikimedia.org/r/1067300

The branches were cut last night but not pushed to Gerrit due to a change made in the cut script: https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/commit/663c84371a60e1232a501e627c266899d4f5298f . That renamed an environment variable which as a result was no more recognized by the branching script. That change got rolled back and the release Jenkins has been redeployed. I have left a comment on that commit.

Another issue was that mediawiki/core had patches merged in between and the branch already got created on the local disk:

11:03:21 Branching mediawiki/core to wmf/1.43.0-wmf.20 from HEAD
11:03:21 Warning: Branch wmf/1.43.0-wmf.20 already exists in repository mediawiki/core

But Gerrit then rejects it:

Exception: Command failed: ('/usr/bin/git', 'push', 'origin', 'HEAD:refs/for/wmf/1.43.0-wmf.20', '--push-option=l=Code-Review+2')
Output: remote: 
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done            
remote: ERROR: Implicit Merge of cde00b5 auth: Add AuthManagerFilterProviders hook        
remote: ERROR: Implicit Merge of e3cea8e auth: Add AuthManagerVerifyAuthentication hook        
remote: ERROR: Implicit Merge of 8cf23d3 Merge "auth: Add AuthManagerFilterProviders hook"        
remote: ERROR: Implicit Merge of 3abfedb Merge "auth: Add AuthManagerVerifyAuthentication hook"        
remote: ERROR: Implicit Merge of be9bb3e Add Nupe (nup) namespace translations        
remote: ERROR: Implicit Merge of ef87455 Localisation updates from https://translatewiki.net.        
To https://gerrit.wikimedia.org/r/mediawiki/core
 ! [remote rejected]         HEAD -> refs/for/wmf/1.43.0-wmf.20 (implicit merges detected)
error: failed to push some refs to 'https://gerrit.wikimedia.org/r/mediawiki/core'

My understanding is those patches are merged in master but were not known to Gerrit in the wmf/1.43.0-wmf.20 branch. Attempting to push them (which is fast forward) is rejected because Gerrit never seen them as changes to the wmf branch and it ends up rejecting them. In Gerrit that is the setting receive.rejectImplicitMerges):

*receive.rejectImplicitMerges*

Controls whether a check for implicit merges will be performed when changes are pushed for review. An implicit merge is a case where merging an open change would implicitly merge another branch into the target branch. Typically, this happens when a change is done on master and, by mistake, pushed to a stable branch for review. When submitting such change, master would be implicitly merged into stable without anyone noticing that. When this option is set to 'true' Gerrit will reject the push if an implicit merge is detected.

This check is only done for non-merge commits, merge commits are not subject of the implicit merge check.

Default is INHERIT, which means that this property is inherited from the parent project.

That is set to True in our root project All-Projects and that is the right thing.

Mentioned in SAL (#wikimedia-operations) [2024-08-27T10:06:52Z] <hashar@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.20 refs T366965

In T366965#10095068, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2024-08-27T09:25:32Z] <hashar> train: fast forwarded mediawiki/core wmf/1.43.0-wmf.20 from 1faf18d6570 to ef87455d7c3 # T366965

I did the push to explicitly fast forward the wmf branch to master. I could have send a merge commit for review and have it passes through CI. Alternative is to fix our make-release/automatic-branch-cut so that it does not update its local copy of the branch when it already has been cut.

Mentioned in SAL (#wikimedia-operations) [2024-08-27T10:13:00Z] <hashar@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.20 refs T366965

Mentioned in SAL (#wikimedia-operations) [2024-08-27T10:16:55Z] <hashar@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.20 refs T366965

Mentioned in SAL (#wikimedia-operations) [2024-08-27T10:33:37Z] <hashar@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.20 refs T366965

Mentioned in SAL (#wikimedia-operations) [2024-08-27T11:20:53Z] <hashar@deploy1003> Finished scap sync-world: testwikis to 1.43.0-wmf.20 refs T366965 (duration: 47m 15s)

In T366965#10095227, @hashar wrote:

I could have send a merge commit for review and have it passes through CI. Alternative is to fix our make-release/automatic-branch-cut so that it does not update its local copy of the branch when it already has been cut.

I think not updating the branch unexpectedly is probably the better choice?

Change #1067330 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] group0 to 1.43.0-wmf.20

https://gerrit.wikimedia.org/r/1067330

Change #1067330 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 to 1.43.0-wmf.20

https://gerrit.wikimedia.org/r/1067330

Mentioned in SAL (#wikimedia-operations) [2024-08-27T12:18:16Z] <hashar@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.20 refs T366965

In T366965#10095227, @hashar wrote:

I did the push to explicitly fast forward the wmf branch to master. I could have send a merge commit for review and have it passes through CI. Alternative is to fix our make-release/automatic-branch-cut so that it does not update its local copy of the branch when it already has been cut.

Filed as T373501: Automatic branch cut should skip already cut branch

Change #1067909 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] group1 to 1.43.0-wmf.20

https://gerrit.wikimedia.org/r/1067909

Change #1067909 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 to 1.43.0-wmf.20

https://gerrit.wikimedia.org/r/1067909

Mentioned in SAL (#wikimedia-operations) [2024-08-28T08:37:12Z] <hashar@deploy1003> rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.20 refs T366965

I have filed two errors related to authentication:

• T373504: Wikimedia\NormalizedException\NormalizedException: Authentication failed because of inconsistent provider array
• T373507: TypeError: Argument 1 passed to MediaWiki\Extension\CentralAuth\CentralAuthTokenManager::consume() must be of the type string, null given

I haven't made train blockers because the authentication metrics look good on https://grafana.wikimedia.org/d/000000004/authentication-metrics?orgId=1 so I assume there is very little or no impact. I have notified them to the MediaWiki platform team.

Change #1068681 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] group2 to 1.43.0-wmf.20

https://gerrit.wikimedia.org/r/1068681

Change #1068681 merged by jenkins-bot:

[operations/mediawiki-config@master] group2 to 1.43.0-wmf.20

https://gerrit.wikimedia.org/r/1068681

Mentioned in SAL (#wikimedia-operations) [2024-08-29T08:41:30Z] <hashar@deploy1003> rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.20 refs T366965

Tyler has done the log triage and backend logs look quiet so this version looks quiet!

I am keeping the task open since issues might be found over the next few hours. I will close it during my morning tomorrow if everything looks fine.