Allow CHUNK value to be passed in as an option for munge.sh
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	Addshore
	Aug 10 2019, 11:19 PM

Description

When importing munged data into a fresh query service some timeouts happen:

Processing wikidump-000000274.ttl.gz
SPARQL-UPDATE: updateStr=LOAD <file:///mnt/disks/ssddata/mungeOut/wikidump-000000274.ttl.gz>
java.util.concurrent.TimeoutException

Reducing the chunk size of the munge step seems to resolve this.

Right now the chunk size is not customizable without changing the file.
It would be great to be able to pass the chunk size in as an option.

Details

	Subject	Repo	Branch	Lines +/-
	Allow specifying chunk size in the script	wikidata/query/rdf	master	+3 -2

Customize query in gerrit

Event Timeline

Addshore created this task.Aug 10 2019, 11:19 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 10 2019, 11:19 PM

Addshore added a project: User-Addshore.Aug 10 2019, 11:20 PM

Addshore moved this task from Unsorted 💣 to Back Burner 🏛️ on the User-Addshore board.

This should not be hard to do.

Change 530634 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/rdf@master] Allow specifying chunk size in the script

https://gerrit.wikimedia.org/r/530634

gerritbot added a project: Patch-For-Review.Aug 16 2019, 7:49 PM

Smalyshev moved this task from Backlog to Done on the Discovery-Wikidata-Query-Service-Sprint board.Aug 19 2019, 3:24 AM

Change 530634 merged by jenkins-bot:
[wikidata/query/rdf@master] Allow specifying chunk size in the script

https://gerrit.wikimedia.org/r/530634

Maintenance_bot removed a project: Patch-For-Review.Aug 19 2019, 1:10 PM

Amazing, would it be possible to get this bakcported to 0.3.1?
Or should I just back it into the docker images ? :)

@Addshore I was planning to do 0.3.2 pretty soon, if that is easier for you you could use that.

Smalyshev closed this task as Resolved.Aug 26 2019, 6:35 AM

In T230288#5423434, @Smalyshev wrote:

@Addshore I was planning to do 0.3.2 pretty soon, if that is easier for you you could use that.

Sounds great to me/ I'll keep an eye out.

@Addshore 0.3.2 should be up already.

Great!

Allow CHUNK value to be passed in as an option for munge.shClosed, ResolvedPublicActions

Description

Details

Event Timeline

Allow CHUNK value to be passed in as an option for munge.sh
Closed, ResolvedPublic
Actions