Just to document where we're at with this. Getting labpawspublic to work with the api changes was pretty straightforward and only really required https://github.com/toolforge/labpawspublic/commit/0dda04b09f5a0eda8a9a7054e5dc0e47d7ef9b27. Unfortunately I couldn't figure out why it doesn't build on our singleuser image from source with yarn 3.5 (the default for jupyterlab). Not quite sure how to proceed with debugging this, my current plan is trying to avoid rebuild and go for a prebuilt extension.

@Isaac mentioned paws public links are not working in irc/telegram, this should get the links back.

In T367599#9947518, @Don-vip wrote:

Sorry Andrew, still one request!

Can you please add me to video2commons-socketio and video2commons-test Toolforge projects?

In T365154#9818560, @Don-vip wrote:

First question that come to my mind is: why do we have our own Redis instance? If we could switch to Toolforge instance, that would make a component less for us to manage, and we would get support from WMCS team.

In T365154#9811604, @Don-vip wrote:

I would like to help, is there something we can do?
Given that frontend works nicely, I guess the problem comes from celery workers that don't pickup jobs or fail to report progress back to redis? (all my tasks are "pending" but I don't see what could be wrong using grafana only)

There are a few current problems. I have not had the time to look into the current issue, but Worker exited prematurely sounds like the task was picked up, but something killed the celery process (probably OOM).

Clearly V2C is not ready for use, except for beta-testers.

In T283839#9572430, @sbassett wrote:

Anything here that would keep us from making this task public? I'm not seeing anything obvious.

@Ita140188 I'd like to strongly pushback on your comment. It is meaningless to say hundreds of people work at WMF implying therefore solutions should be quick if we don't analyse the magnitude of the work to keep the websites of the wikimedia projects working.

In T236446#9428663, @Yann wrote:

FYI trying to download https://www.youtube.com/watch?v=ecQWZWpwZVw locally, Youtube Downloader HD ( https://www.youtubedownloaderhd.com/download.html ) doesn't work, but yt-dlp ( https://github.com/yt-dlp/yt-dlp ) does work. Does YT block V2C by IP or by software ID, or both, or something else?

FYI google has started to block downloads from video2commons again since around 2023-12-29 22:30.

In T341378#8998519, @SEgt-WMF wrote:

Thank you @Chicocvenancio for reading my question and adding your ideas! Yeah, I just run the code with another Wiki account I have and it went well, so I'm guessing every user gets a limited number of SQL runs?

Running your code I realized that for loop only runs once, so even if it is leaking connections it probably is not your main issue. But since the max_connections are per user they might be coming from another notebook/process.

This is not a PAWS issue as much as a connection limit on the replicas issue. I'm not quite sure what pymysql and Pandas are doing there but it seems connections are leaking beyond each iteration of

def query_wikis(query, df):
    wikis = df.database_code.unique().tolist()
    combined_result = pd.DataFrame()
    for wiki in wikis:
        result = pd_query(query, wiki)
        result['wiki_db'] = wiki
        combined_result = pd.concat([combined_result, result])
    return combined_result

This is a change in behavior for jupyterlab 4 impacting labpawspublic. The property we are using is having RTC: prepended to it. I cannot figure out where to get the proper path, in fact I cannot find the documentation for jupyterlab's 4.0 api... (3.0 docs).
A quick solution is to remove the RTC: there.

LGTM, just apply the same changes to the other tools.

Hopefully resolved with current updates.

Should be fixed now.

Local uploads should be working again.

@Andrew https://wikitech.wikimedia.org/wiki/Help:Trove_database_user_guide#Before_you_start

@taavi I thought that as well, but wikitech docs link to this template, and there is no need for a project beyond the db.

Sorry I missed the followup questions, but I think you have solved the mystery. Since I have ipblockexempt it didn't stop me from editing, but its likely my vpn provider was blocked.

Linkback from https://lists.wikimedia.org/hyperkitty/list/wikimedia-l@lists.wikimedia.org/message/DVA7I5TUHSNJAAYKOBD6O77FJT35E2MK/ where I cite this as an example of how WMF has delayed UCoC until the enforcemente guidelines are approved.

In T326164#8498955, @rook wrote:

In T326164#8498932, @Chicocvenancio wrote:

In T326164#8497724, @rook wrote:

It is possible that this will be resolved with T317787

Not really. :/, this is the new Jupyterhub rbac.

Interesting, I was making this guess as I can still delete my server in minikube, and the major difference is that we are not introducing Pod Security Policies in minikube. Begging the question of if not psp, why is it working in minikube. Perhaps we are configuring something else differently in prod that we could adjust?

Some sections of this, mostly host path parts, will become irrelevant with T308873 other parts remain unclear to me on why we need them. Aside from by default needing a psp, are there known benefits of our current setup?

In T326164#8497724, @rook wrote:

It is possible that this will be resolved with T317787

In T326160#8496160, @Bugreporter wrote:

In T326160#8496065, @Dominicbm wrote:

I tried again, and get a different error message:

400 : Bad Request
Invalid client_id parameter value.

I also get the same issue.

(my best guess is the authenticator is storing the client-id in the auth state and that has now changed in the upgrade)

Could you try one more time? I've deleted your user from the db, it should be recreated on login for you.

Its probably related to the user being running at the time of upgrade, let me try to manually clear the tokens in the db.

In T326167#8495997, @rook wrote:

If I'm understanding that network policy shouldn't 172.16.1.129 be permitted under 172.16.0.0/12?

This seems to be the new networkpolicy blocking private ip addresses.

Didn't manage to delete the server, just logged out and back in.

More I'm curious, added me where? I'm not in the list of admins in values.yaml, though I have access to the admin tab (well did until today)

Still never known how I had admin before though. I always assumed it was pulling from cloud vps, but that seems unexpected?

https://jupyterhub.readthedocs.io/en/stable/rbac/roles.html# seems to be the new way of defining admins. I see the admin tab but nothing is there for me right now.

Uhh, it now works... Maybe there was a cached logged in SA for me?

That seems fine to me. This is RBAC internal to jupyterhub to do oauth dances between the various components. I'm reading code/docs trying to figure out what needs to change.

In T326160#8495643, @rook wrote:

Oh, an account that was not logged in at the time of the upgrade can auth. Maybe it is something about accounts that were active during the upgrade?

That may parallel why it seemed to work in minikube.

Seems the oauth tokens for the internal service accounts in jupyterhub don't have the right scopes, weird that this doesn't seem to be mentioned in the upgrade guide.

Why was a rollback necessary? Is trying to fix the new version an option?

Probably a migration was applied with the upgrade and alembic sees it in the table but not in the migration code of the rolled back version? Nuking the db is always an option, will only disrupt currently running pods and log everyone out. There might be a rollback migration code for the new migration, or it might even be compatible needing only the a DELETE in the revision, that would need to be figured it out if keeping the db is desired.

Things are working again for me. I see no 400 errors in the logs for the past 10 minutes.

I confirm its down for me as well.

Reading the github issue I guess moving to notebook-based culling is the way to improve things, I can try to take a look at that.

the hub pod has some in-memory state, and (worse) some shared state with the proxy pod. https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1364 is main upstream issue on this, I think.

https://github.com/toolforge/paws/pull/173 lgtm, Will make the PR upstream.
https://github.com/jupyterhub/jupyter-rsession-proxy/pull/130 is the upstream PR to track, we can use a fork if it takes a while to get merged.

Probably https://github.com/jupyterhub/jupyter-rsession-proxy/blob/master/jupyter_rsession_proxy/__init__.py#L139

I created https://github.com/jupyterhub/jupyter-rsession-proxy/issues/129 upstream for this. I think should be a pretty straightforward change there to get things working. As I see it, NB_USER should be the source of truth for users in jupyterhub land.

I think volume persistence is the biggest blocker here, right?

Feels like your firefox is keeping jupyterlab open after it has a disconnect (Very possibly only a disconnect from the nfs), without telling you that it isn't connected.

I think there is something a bit more esoteric going on. If the the pod wasn't running or didn't have access to the filesystem creating a new file would not be possible. Might be browser session, pod v jupyterhub authentication or websocket connection failing. I would look into the hub and proxy logs for any error around the routes to the user pod and in the user pod itself for any error message the next time this happens.

Have you tried using the absolute path?

Sounds about right to me.

yeah, we haven't used in a few years.

The lack of an uptick in new weekly editors on fawiki during the ip block indicates to me a big difference from ptwiki's implementation. My lack of farsi prevents a deeper investigation, but in ptwiki the interface was hacked to point logged out users at the point of edit. The edit button was changed to point directly to the log-in page and banner explaining the need to create an account linking to the reasoning was put there.

@Isaac indeed. Fixed the link to a permalink to avoid future issues.

6.6.5 was merged.

Other bank transfer methods are also erroring for me.

@EMartin its before we actually make the transaction.

All for the upgrade but just for the record I don't see a path for exploitation here. All requests to openrefine go through the proxy and only authenticated requests pass. And if you're authenticated into a user server, well RCE is the main feature for jupyterhub.

LGTM. Opinionated ways to run the more basic tasks is a good way to reduce the burden on new users.

I guess portaudio19-dev is the missing apt package here.

Thanks to @yuvipanda on the assistance on this. Would've taken me hours to understand it.

Just noting we received a report of this bug in the Portuguese wikipedia telegram group today.

I had to rollback as the new Host validation in openrefine 3.5 broke it with the current settings. I'm not quite sure what best to do here, since we're already proxying it through the jupyter server we could disable the validation. But I think I'll set it up to have the proper value in the next few days.

PR in https://github.com/toolforge/paws/pull/110

Merged and deployed