Topic on User talk:Magnus Manske

Hi, could you please remove autoscrape settings from these catalogues? I've been monitoring them for a long time and they stopped having autoscrape working a long time ago, but people are constantly restarting autoscrape job which hangs up in the queue for quite a long without any results.

• https://mix-n-match.toolforge.org/#/catalog/1 odnb (at one point it gave wrong id's which had to be manually marked as not applicable, but now it has stopped creating anything, although new id's appear monthly on the site)
• https://mix-n-match.toolforge.org/#/catalog/1011 WNS number (stucks in the queue for a few days and gives nothing for a couple of years)
• https://mix-n-match.toolforge.org/#/catalog/1486 mb artists (musicbrainz blocks request with non-browser UserAgent - so mix-n-match now always sees white page at every request)
• https://mix-n-match.toolforge.org/#/catalog/3476 mb genre (same as above)
• https://mix-n-match.toolforge.org/#/catalog/2043 letterboxd (gives no results for quite some time, although new ids appear regularly)
• https://mix-n-match.toolforge.org/#/catalog/3658 pitchfork (the scrapped page has been heavily redesigned and it no longer working)

There is also still the problem I mentioned in this post. I've launched through api autoscrape start for a set of catalogues. And the ones where autoscrape doesn't exist hang in the queue with a constant restarts via schedule. The catalogues are:

• https://mix-n-match.toolforge.org/#/jobs/5923 Deezer music genre
• https://mix-n-match.toolforge.org/#/jobs/5161 Napster music genre [archived]
• https://mix-n-match.toolforge.org/#/jobs/4905 MasterClass music genre guide
• https://mix-n-match.toolforge.org/#/jobs/4939 Free Music Archive music genre
• https://mix-n-match.toolforge.org/#/jobs/5000 PromoDJ music genre

This probably needs handling with a response and stopping the task if autoscrape is requested when it can't be executed.

(I also tried running a "pause" task in https://mix-n-match.toolforge.org/#/jobs/5923 , which is also saved in the tasks. This probably needs handling too to avoid potential vandalism)

Done, and thanks for the list. I made a new job status "BLOCKED" that can not be started from the web interface.

I also filter for existing job action types now

Thanks for the quick resolution. Btw I also saw PAUSE status in the queue a few days ago. I think it would be useful to have it in the web interface (or ABORT button) to stop at least your own tasks to avoid misclicks or unnecessary schedules as in the case of music genres above.

Hi. Could you please also block autoscrape on these catalogs. Most of them have either changed their layout and no longer work, or are stuck permanently with no new results for over a year.

• https://mix-n-match.toolforge.org/#/jobs/1
• https://mix-n-match.toolforge.org/#/jobs/1011
• https://mix-n-match.toolforge.org/#/jobs/1486
• https://mix-n-match.toolforge.org/#/jobs/2043
• https://mix-n-match.toolforge.org/#/jobs/3658

UPD: Oops, it appears I've reported them all above. They've probably been unblocked and are back in the queue?

And by the way, is it possible to remove the "automatch by search" task from the regular repetition here https://mix-n-match.toolforge.org/#/catalog/3789 and https://mix-n-match.toolforge.org/#/jobs/5195 ? I hit "purge automatches" every quarter, as without types syncing is more harmful due to clogging than helpful. It would probably be useful to have a button to remove regular tasks from the schedule in such cases (at least for those who have admin rights).

Hi, I can't see any problem with the https://mix-n-match.toolforge.org/#/catalog/1011 WNS number . The URL and IDs seems to be the same. The web site is updated periodically, last time 9 October 2024. The web has now 116 680 postage stamps registered but in the catalog there are only 90 241. What can be the problem? Thank you.

Hi, Gerwoman. Autoscrape doesn't work in this catalogue. Given that the catalogue is relatively small, autoscrapping should take no more than half an hour, by my experience. But in this catalogue autoscrape regularly falls into the queue and hangs there for several days/weeks without adding anything new for years. This can be verified in this way: The latest mix-n-match ID in this catalogue is https://mix-n-match.toolforge.org/#/entry/91669282. The latest mix-n-match IDs in recent catalogues have 172699175 ID. # 91m vs # 172m means that none of the autoscrape jobs for 2-3 years has not added a single new ID in this catalog. Thus, it does not work and constantly hangs in the queue.

If you know that IDs can still be autocollected from the site, you just need to reconfigure autoscrape from scratch by specifying 1011 in Catalog ID here https://mix-n-match.toolforge.org/#/scraper/new and configuring everything as you did the first time. If you can't reconfigure it again, it's better to just disable the autoscraper in it.

Thank you Solidest, but I can't remember how configured it the first time. I don't have access to the URL or regex...

I've reconfigured the autoscrape. Let's see.

Yeah, it worked, the catalogue entries are now at 116k. Thanks! I've crossed that catalogue off the list.