On the topic of using IndexedDB for module store, beware that there is an important difference with localStorage.

localStorage, while having a lower quota by default (~ 5MB), doesn't have a way to expand that quota. Which means when it's full, the API returns a failure code but nothing is shown the user.

IndexedDB, while having a higher quota by default (~ 10-50MB), has a way of extending the quota. It's not clear to me what happens when you reach the quota. It appears that browsers default to asking the user to expand the quota.

For the purpose of caching modules, and for Wikipedia in general, I don't think we should ever allow that to happen. So we may have to somehow manually poll the aggregate size of database to prevent the browser from asking the user any such question.

See http://www.html5rocks.com/en/tutorials/offline/quota-research/

In T66721#1340106, @matmarex wrote:

While the last patch may be helpful, it still does not prevent the cache from taking up all available localStorage space.

Especially on Firefox, where e.g. all of the Wikipedias are sharing quota. (There are a lot of Wikipedias, and there are reasons someone might look at quite a lot of them.)

Also, to clean up after this, you'll need to do something like what's described at https://bugzilla.mozilla.org/show_bug.cgi?id=1064466#c11: have a path on each wiki that will clean up the storage, and also a way to open an iframe to that path on *every* wiki for a given domain ...

So, um, any tips on how to manually clean up the MediaWikiModuleStore keys from all of the wikipedias on Firefox/Iceweasel, including 31.7.0esr as included in Debian Jessie?

In T66721#1411731, @SamB wrote:

So, um, any tips on how to manually clean up the MediaWikiModuleStore keys from all of the wikipedias on Firefox/Iceweasel, including 31.7.0esr as included in Debian Jessie?

It should not be possible to fill it up now that the patch above has been pushed. You must have gotten into some weird state. Clear localStorage (either by nuking your browser history or by using https://addons.mozilla.org/en-US/firefox/addon/volatile-storage/?src=cb-dl-updated) and see if the problem recurs.

That extension does not seems to be enough, since I still get this on every page on Portuguese Wikipedia using Firefox 38.0:

"Exception in store-localstorage-update:" load.php:177:752
"NS_ERROR_DOM_QUOTA_REACHED: Persistent storage maximum size reached" DOMException [NS_ERROR_DOM_QUOTA_REACHED: "Persistent storage maximum size reached"
code: 1014
nsresult: 0x805303f6
location: https://pt.wikipedia.org/w/load.php?debug=false&lang=pt&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=CZ6SC%2Fzc:175] load.php:177:790
log() load.php:177
handler() load.php:155
jQuery.Callbacks/fire() load.php:45
jQuery.Callbacks/self.fireWith() load.php:46
jQuery.Callbacks/self.fire() load.php:46
mw.track() load.php:155
flush() load.php:175

and running

localStorage.clear();

in the console doesn't help either.

In T66721#1411737, @ori wrote:

In T66721#1411731, @SamB wrote:

So, um, any tips on how to manually clean up the MediaWikiModuleStore keys from all of the wikipedias on Firefox/Iceweasel, including 31.7.0esr as included in Debian Jessie?

It should not be possible to fill it up now that the patch above has been pushed. You must have gotten into some weird state. Clear localStorage (either by nuking your browser history or by using https://addons.mozilla.org/en-US/firefox/addon/volatile-storage/?src=cb-dl-updated) and see if the problem recurs.

Hmm, I'm still a bit nervous about nuking localStorage entirely, though that certainly would be less tedious than visiting every Wikipedia I've ever loaded in this bowser (including mobile versions) one-by-one and clearing the relevant key from the JS console... (though, that might not be so bad if I used a greasemonkey script to do the actual cleanup for each wiki).

Anyway, this:

In T66721#1340094, @gerritbot wrote:

Change 216014 merged by jenkins-bot:
mw.loader.store: decline to store items > 100 kB

https://gerrit.wikimedia.org/r/216014

... doesn't sound very promising to me. Just because it only stores items < 100 kB does *not* prevent it from storing many such items, which could add up to a lot of usage on a single Wiki; factor in the fact that subdomains share quota on Firefox, and you could still easily hit 5 MiB, no?

Could we use ServiceWorker for this now (see T101731) ?

In T66721#1418743, @Jdlrobson wrote:

Could we use ServiceWorker for this now (see T101731) ?

Yes. We created T101732 last month.

In T96155#1539605, @matmarex wrote:

In T96155#1528593, @Jdlrobson wrote:

Yup. I recognise this is an issue but I am saying that this issue mostly impacts power users (especially on Firefox). I'm yet to see this on the mobile site, where gadgets and user scripts are currently not available nor used.

Eh, I don't think that it mostly impact power users, it can impact even occasional editors too.

As an example:

• Visit https://pl.wikipedia.org/wiki/Ostatnia_droga_Temeraire%27a, which is a pretty generic article on the Polish Wikipedia, as anonymous user. Run JSON.stringify( mw.loader.store ).length to see how much data ResourceLoader puts in localStorage (this is 707573 for me on first page load).
• Open the MediaViewer, which is not a power-user feature. The length increases to 923750.
• Open the VisualEditor, which is not a power-user feature either. The length increases to 2008078.

This is already 2 MB, and this is just one wiki and an anonymous user (I'm not sure how the browsers calculate the limit, but if anything, this is less than the number of bytes actually stored). Visit another wiki (say, the same article in English, if you're not satisfied with the content in Polish and speak two languages) and this doubles. Log in (which gets you features like Echo and lets you enable some gadgets or beta features), or visit a third wiki, and you're quickly over the 5 MB limit.

Interesting and thanks for the counter-example.
I've taked to @ori about T96155 and the plan is for me to make this code use the new frontend api. When that's in place we could potentially be a bit more clever about what we are allowed to store e.g. limit number of bytes we allow saving for.

Given that MobileFrontend/Gather are only using this storage mechanism to decide whether to show onboarding tutorials for new features, mostly targeting readers who are not editing or newbies, if localStorage is exhausted not showing these things is not the end of the world.

I assume anyone who wants to use localStorage is supposed to use try/catch on every attempt to save a value so that unrelated scripts are still executed? E.g.:
https://pt.wikipedia.org/wiki/MediaWiki:Gadget-fastbuttons.js?diff=43318591
https://meta.wikimedia.org/w/index.php?diff=13512749&oldid=13462279&diffonly=1
https://github.com/brion/MediaWiki-MobileSidebar/pull/5/files?w=1

Generally yes, regardless of this bug, even. localStorage likes to throw
exceptions. (For example, apparently it will be defined but throw exception
on access on Firefox if you disable it in browser settings.)

FYI, i had an issue fixed in WikEd, which was throwing exceptions all over the place, because it wasn't expecting a full localStorage.

That's the problem with this issue. it's infectious....

Change 263417 had a related patch set uploaded (by Ori.livneh):
Module storage: set stricter storage quotas on Firefox

https://gerrit.wikimedia.org/r/263417

Change 263417 merged by jenkins-bot:
Module storage: set stricter storage quotas on Firefox

https://gerrit.wikimedia.org/r/263417

Nothing has changed here since 2014 -- it was a mistake to enable this feature on Firefox at the time, and it is a mistake to enable it now with a 30KB per-module limit. On my browser at the moment, MediaWikiModuleStore:enwiki is using 655KB, which is hardly an improvement from when the bug was first reported. You only need to visit 7 or 8 wikis to exhaust the quota.

I don't think a <250ms performance improvement justifies randomly breaking the many extensions which use localStorage.

mediawiki.js will keep writing module caches until it hits the quota and fails. The functionality of the site after such a failure depends on how much of the quota is left -- effectively the remainder of the division, 5MB modulo the per-wiki module store size. An unlucky combination could leave a user unable to dismiss dialog boxes or to choose a preferred language.

Some extensions try localStorage first, and fall back to a cookie if it fails. But the MW core does not provide a convenient interface for this, leaving each extension to implement it separately.

Has anyone encountered any bugs since rMW7787a85d5c6f: Module storage: set stricter storage quotas on Firefox went out, just under a month ago?

I continue to get this error on every page view using Opera 12 (obviously, since you only blacklisted Firefox).

I still get this on every page view using Firefox 44.

Ok. I have been in denial about this long enough, I suppose. Let's start by disabling it completely in Firefox and Opera. I am still interested in more reports, because it will help determine where we go from there.

Change 269625 had a related patch set uploaded (by Ori.livneh):
ResourceLoader: disable localStorage cache on FF, Opera

https://gerrit.wikimedia.org/r/269625

Change 269625 merged by jenkins-bot:
resourceloader: Disable localStorage cache on FF, Opera

https://gerrit.wikimedia.org/r/269625

From T85923: Anonymous user's decision to disable Media Viewer is forgotten after two weeks: wgMediaViewerOnClick apparently cannot be reliably saved to local storage; used space (as per encodeURIComponent(JSON.stringify(localStorage)).length) is 2600025, quota is (per this test) somewhere between 2600000 and 2700000. The browser is Safari. May be related, although 2600025 is 0b1001111010110001011001 which does not seem like something that would be very close to a quota limit.

Replied at T85923#2471543 as it seems about a key disappearing, not a key being unable to be set.

As for the issue reported in this task, it's been resolved. In theory, it will always be possible for the storage quota to be reached, at which points random keys will fail to be set. (This is true for cookies as well.)

Task T101732 tracks moving the module cache to a different interface for that reason.

As for the issue reported in this task, it's been resolved. In theory, it will always be

So, what is the appropriate course of action? Given that I'm seeing an elevated level (for FF and Chrome) of:

Exception in store-localstorage-update:
:8080/mw-31-00-elastic/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=0xnodom:177 QuotaExceededError: Failed to execute 'setItem' on 'Storage': Setting the value of 'MediaWikiModuleStore:mw-31-00-elastic' exceeded the quota. DOMException: Failed to execute 'setItem' on 'Storage': Setting the value of 'MediaWikiModuleStore:mw-31-00-elastic' exceeded the quota.
    at flushWrites (http://127.0.0.1:8080/mw-31-00-elastic/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=0xnodom:175:212)
logError @ :8080/mw-31-00-elastic/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=0xnodom:177

Are you seeing that error in Firefox? That shouldn't be possible because MediaWiki self-disables the mw.loader.store functionality on all Firefox versions. And upon viewing a page, any previous use of it is also proactively cleared. The flushWrites code should not be reachable. If you find that it is being reached in Firefox, please file a separate task and we'll investigate to make sure it remains off.

As for Chrome, there shouldn't be a course of action needed. The system is meant to be self-correcting. The errors you're seeing are "caught", that means the MediaWiki code did not crash or stop unexpectedly. The code prints the caught error to the developer console as a "warning" so that when developers investigate other issues, they also know this particular condition was reached. Are you finding any loss of functionality in Chrome?

The main problem found in this task was due to multiple wikis storing their modules under the same quota in Firefox. That is not something we can control. More-over, the browser also does not provide a way to let module storage expire from wikis not recently visited. For this reason, we have decided to disable this performance feature in Firefox. In Chrome this should not be a problem given that Chrome sets the quota at the full domain level (not the parent domain).

I noticed in your example that you might be experiencing problems from serving multiple MediaWiki installations from the same domain under different directories. If that is the case and you intend to also offer multiple installations this way in production, I recommend $wgResourceLoaderStorageEnabled = false;.

Alternatively, if this is only for local development, run localStorage.clear(); from the console to clear the storage used by previous/other wiki installations.

This issue seems to be resolved in Firefox 69 and apparently the quota is now per origin. See: https://bugzilla.mozilla.org/show_bug.cgi?id=1064466#c18

So maybe we can start thinking about enabling local storage for Firefox again? At least dependent on the Firefox version number in the user agent?

Thank you, would you mind creating a new task for that?

Change 776261 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] resourceloader: Enable mw.loader.store in Firefox

https://gerrit.wikimedia.org/r/776261

Change 776261 merged by jenkins-bot:

[mediawiki/core@master] resourceloader: Enable mw.loader.store in Firefox

https://gerrit.wikimedia.org/r/776261

I get "Exception in store-localstorage-update" and "The quota has been exceeded." on https://test.wikipedia.org/wiki/. At the same time, enwiki doesn't seem to be affected.

JSON.stringify( mw.loader.store ).length is 938292. After clicking an image (opening it in MediaViewer) it's 1096751. After that I pressed "Your alerts" (the bell) it's 1375615. That's like, 1.4MB? For what exactly?

JSON.stringify( window.localStorage ).length is 3281.

Those values combined, while high, are nowhere near 5MB so the error doesn't quite make sense to me still. I must be missing a storage mechanism?

Sometimes I worry if my userscript is efficient enough. Then I see numbers like 1.4MB and I'm confident again that no matter how bad, I'm somehow vastly outperforming the paid WMF team when it comes to efficiency. Which is fairly insane.

Is this task why I'm seeing this error? Is this really resolved?

In T66721#7865568, @AlexisJazz wrote:

I get "Exception in store-localstorage-update" and "The quota has been exceeded." on https://test.wikipedia.org/wiki/.

At the same time, enwiki doesn't seem to be affected.

This means that the changes from this ticket, made some 5-7 years ago, are fortunately still working as expected. We do not use the defragmented module cache in environments where we know localStorage quotas would be shared.

While not a "cause", the reason you are seeing this console warning today and not before, is because something is storing large values in localStorage. That is most likely mw.loader.store indeed, which was recently re-enabled in Firefox as per T235852, after we learned and confirmed emperically that the shared quotas indeed are no longer a thing in Firefox. Thus allowing the same savings from the defragmented module cache, that people using Safari- and Chromium-based browsers have been enjoying all these years (documentation).

JSON.stringify( mw.loader.store ).length is 938292. After clicking an image (opening it in MediaViewer) it's 1096751. After that I pressed "Your alerts" (the bell) it's 1375615. That's like, 1.4MB? For what exactly?

This is measuring the uncompressed size of the (minified) CSS and JS code downloaded during your browsing session so far (including from previous pageviews; limited to 1 copy of any given module bundle by name; old versions or variants for other lang/skin contexts are automatically discarded).

1.4MB is not a particularly large number in that regard. Note that nothing about this feature is incurring additional cost in terms of what you download. Quite the opposite.

Features like MediaViewer are not new. They were there before and are there now. Previously, this was stored in the HTTP cache by URL key. They are now stored in a defragmented fashion in the module cache.

By doing this, your browser is able to make optimal use of this cache, allowing it to re-use part of something you downloaded on a previous pageview, and re-use it on this one. This has the impact that as a Firefox user, you are now downloading a drastically lower amount of bytes from the network on a typical pageview, and storing essentially only 1 copy. Saving bandwidth cost, saving disk space, and saving time.

JSON.stringify( window.localStorage ).length is 3281.

Those values combined, while high, are nowhere near 5MB so the error doesn't quite make sense to me still. I must be missing a storage mechanism?

No. As part of ensuring the cache plays nice with other features and gadgets that make use of cookies and localStorage, mw.loader automatically removes its portion from localStorage if it is full. You probably just missed the window during which this was briefly stored and then no longer.

I suspect you might have additional features enabled that I don't, or have built up caches over a longer period of time. For me in Firefox (my main browser) on test.wikipedia. That's okay, automatically clearing unused code is part of the cycle. It'll start paying off again starting on the next pageview.

JSON.stringify( mw.loader.store ).length
//> 811398 (0.8MB)
JSON.stringify( window.localStorage ).length
//> 847312 (0.8MB)

I'm somehow vastly outperforming the paid WMF team when it comes to efficiency. Which is fairly insane.

Extraordinary claims require extraordinary evidence.

Note, I don't think it's extraordinary in general that a volunteer would write good code. On the contrary, I'm sure your code is fine, and in fact many extensions we deploy or host as gadgets are maintained by volunteers and often quite to a standard that is as good or above average, including in following performance guidenance.

However your particular claim is obviously meant to suggest that our software somehow is inefficient, which is factually inaccurate by almost every imaginable metric. Wikipedia.org is and remains one of the best performing sites in the world, in any category. That's no small feat.

Is this task why I'm seeing this error? Is this really resolved?

I believe it remains resolved. The warning you saw (not an uncaught error) in the developer console, is a rare but entirely expected and harmless side-effect of the system working as intended, as has it has been for other browsers over the past 7 years.

This warning may appear on 1 single pageview once a month or so, possibly more regularly if you engage with many different and large frontend features in a short period of time. If you see it on every page view, or if it has any observable negative effect on functionality by a gadget or user script, or on network/bandwidth use; please create a task as that would be an important issue to address.