To display the reference links, CSS counters are used in the existing code. To add the refines, we would probably add another counter and data attribute, like so:
https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Cite/+/532682/5/modules/ext.cite.style.css

Regarding the rollback:

Possibly unrelated, but curious: I also get a timeout for the the page in the first test case (the page itself, not the diff): https://de.wikipedia.org/w/index.php?title=Portal:Biologie/Fehlende_Artikel/Missing_Topics

gedanken von mir zu dem thema:

I tried to reproduce this, without success so far. Is there anything specific I need to do or do we have a page where it reliably happens? Or does it seem to happen only randomly?

done.

Yes, I can confirm it needs the NFS mounts.

I don't have Chrome installed, but I found "interesting" results with other browsers:

• In Chromium 69.0.3497.81, nothing is greyed out. Everything looks fine to me.
• In Opera 54.0.2952.64 (yes that still exists!) everything looks fine as well
• In Firefox 62.0, the text boxes keep their color but the text itself is greyed out after clicking the edit button. In the attached screenshot the edit button was clicked in the first two rows, but not the last row.
  

In T181454#4570572, @tstarling wrote:

https://wmde-wikidiff2-patched.wmflabs.org/core/index.php?title=Diffcompare-ja
https://wmde-wikidiff2-patched.wmflabs.org/core/index.php?title=Diffcompare-zh
https://wmde-wikidiff2-patched.wmflabs.org/core/index.php?title=Diffcompare-th

Test cases are here: https://wmde-wikidiff2-debug.wmflabs.org/core/index.php?title=DiffcompareDebug

Hi @Jdlrobson! The moved-lines functionality for mobile output in wikidiff2 is disabled by an ini variable. That's why you don't see any classes in the mobile output. It can be enabled by ini_set'ing wikidiff2.moved_paragraph_detection_cutoff_mobile to some positive integer, eg 30. A mobile diff with a moved paragraph will then look like this: https://wmde-wikidiff2-mobile.wmflabs.org/core/index.php/Special:MobileDiff/1455 -- it shows something was moved, but there's no proper styling yet.

There is special code for the case where one destination paragraph seems to match several source paragraphs or vice versa. In that case, only one source (or destination) paragraph is then selected. In this particular diff, this detection doesn't work correctly. I'm working on a solution and feel I'm pretty close. Next step will be a patch, a new test VM and a mod to the compare script which will check what the fix changes. I will then check this list for possible regressions.

evaluation of mobile diff is here: http://wmde-wikidiff2-mobile.wmflabs.org/core/index.php?title=Diffcompare_Mobile

test server for the mobile stuff is at https://wmde-wikidiff2-mobile.wmflabs.org/core/index.php/Main_Page

here's the list:
https://wmde-wikidiff2-patched.wmflabs.org/core/index.php?title=Diffcompare

unfortunately, the diffs generated using this special-case code turned about to be incorrect so i had to revert this. looking at the list of random test diffs, these cases turn out to be rare though.

https://wmde-wikidiff2-unpatched.wmflabs.org/core/index.php?title=Honeyguide&type=revision&diff=45&oldid=44
https://wmde-wikidiff2-patched.wmflabs.org/core/index.php?title=Honeyguide&type=revision&diff=45&oldid=44

sometime a change DiffOp would contain add and delete ops, which would later not get picked up ba the moved paragraph stuff. fixed (see examples above).

examples: https://wmde-wikidiff2-patched.wmflabs.org/core/index.php?title=Irregul%C3%A4rer_Satellit&type=revision&diff=3239&oldid=3238 http://wmde-wikidiff2-patched.wmflabs.org/core/index.php?title=Autoimport/G.K.Singh&diff=2211&oldid=2210

I would say a default value of 0.2 looks pretty good for English. Some slightly annoying edge cases exist for any value, which could be fixed by special-case code. I also investigated "character runs" as an alternative to the character-based similarity but found no improvement.

the list of pages is on an automatically generated page on the wiki: http://wmde-wikidiff2-patched.wmflabs.org/core/index.php?title=Diffcompare

what will be done
Similar to the desktop diff, moved paragraphs will be detected, links inserted between them, and word diffs between the moved paragraphs will be displayed.

In T176637#3672066, @MoritzMuehlenhoff wrote:

In T176637#3672019, @jkroll wrote:

It should be restarted after a native PHP extension is upgraded.

HHVM was restarted after installing the new package (otherwise it would not load the revised modules anyway). The cache is persistent on disk, though (but we can prune it manually).

The wikidiff2 patch introduced new parameters with default values to wikidiff2_do_diff() and wikidiff2_inline_diff(). Possible explanation for the crashes: HHVM caches data for performance reasons, including function signatures provided by extensions. Between deploying the new wikidiff2 version on the 15th and the HHVM unattended-upgrade on the 28th, HHVM would cache the old signatures in memory. The new wikidiff2 would have the new signatures though, and the PHP code is already patched to use them, causing a mismatch.

In T176637#3669690, @MoritzMuehlenhoff wrote:

On the 28th, the update to HHVM 3.18.5 was installed via unattended-upgrades. That might have fixed it (or could be entirely unrelated/coincidental).

Also, getting a core file when this happens would be helpful. The core pattern was set to /data/project/cores/ but /data doesn't seem to be mounted. I've temporarily set it to /tmp but I'm not sure if it will be overwritten by the next puppet run.

Thanks @MaxSem. I've tried to reproduce it on deployment-mediawiki04.eqiad.wmflabs without success. According to logs, the last crash was on Sep 28.

@MaxSem: Jkroll

This seems to be related to the new wikidiff2. I wrote the moved-paragraphs patch, and I have been trying to reproduce this problem for the last couple days. It's not reproducible for me on apache/zend. I can't reproduce it with my distro-provided hhvm 3.22.0 either. Next, I tried to install mw-vagrant as per MoritzMuehlenhoff's suggestion in T176217. After working around T152801 and provisioning the box, I found that it wasn't running hhvm at all. Unsurprisingly, I couldn't reproduce the problem there, so I gave up on mw-vagrant. Instead I built HHVM-3.18.5 from source. I cannot reproduce the problem with that version either. I can display diffs just fine, including the mobile diff, and haven't seen any stack traces or suspiscious log entries.

Recompiling wikidiff2 is not the issue here, I wrote the moved-paragraphs patch.

I'm trying to reproduce it on my local machine. With my distro-supplied hhvm 3.22.0 (rel) I don't see anything suspicious in the logs at all so far. Can you pin down when the asserts happen? Probably when generating a mobile diff page?

can you (or anyone) help interpret this output? I do x86 assembly, but I'm not familiar with php jit bytecode :)

After some investigation, I found that the data in CatGraph is actually OK. The problem was in the catgraph-jsonp tool which makes CatGraph queries available to JavaScript code. catgraph-jsonp has some logic to catch case errors in category title input. In an edge case involving spaces in titles, catgraph-jsonp erroneously fetches the page_id of a redirection page. This commit fixes it. It's already deployed.

@MaxSem I added some code to your fuzz test which hits the other code paths in the patch. It just creates shuffled lines basically.

@MaxSem updated again :) Anything else you want changed? Or can you give a +1? Thanks

@MaxSem Thanks for looking into https://gerrit.wikimedia.org/r/#/c/356582/. I updated it. Could you review this and https://gerrit.wikimedia.org/r/#/c/319866/ so we can get it merged?
@tstarling You could also review this, thanks!

We benchmarked this and set the default to 25. The threshold is configurable in LocalSettings.

This, of course, cannot work:

Well the solution would be to make MediaWiki work with both the old and the new version of Wikidiff2, right?

In T156439#3155361, @srishakatux wrote:

@jkroll Is it fair to say that T146781: Show changes in moved text chunks (C++) is similar to this task?

Actually, this is already done :) The hostmap has been using lighttpd for some time now.

Taking Tim Starling's feedback on Gerrit into account, I've inserted named anchors in the diff output for moved paragraphs. The + and - symbols are replaced with "left shift" (<<) and "right shift" (>>) symbols. Hyperlinks allow to move between source and destination paragraphs. This is a good solution because

• it shows which paragraphs were moved, even if they weren't changed
• it's plain text, so it can be searched for in the browser
• it doesn't require JavaScript or visual changes which have to be adjusted to each Skin, such as different background colors for moved blocks

The code is straightforward and doesn't have performance implications, so it doesn't require a separate patch set.

It turns out that Tool Labs doesn't have the Wikidiff2 extension installed. Mediawiki falls back to some default diff implementation in that case (probably PHP) which produces slightly different output than Wikidiff2. I have set up two Labs instances with *unpatched* and with *patched* wikidiff2, so we can show a side-by-side diff of the diff code with and without our patch, yes that sentence makes sense.

You're right, sorry for the confusion. Thanks!

Great. Thanks @Andrew !

Here are some examples before/after the patch.

The case where a paragraph appears to have been moved to several places, and words deleted from the inserted copies, can't be displayed correctly. The workaround which is currently implemented shows only the first inserted paragraph as "moved". Users will rarely, if ever, see this edge case, so it is probably sufficient. Other suggestions to solve this are welcome.

Created an instance in the catgraph project for testing. Installed mediawiki with wikidiff2 extension. Setup webproxy. Wiki Link: https://mwdiffstuff.wmflabs.org/core/index.php/Main_Page