OutOfMemoryError: GC overhead limit exceeded

[visenger]

Hello,

currently, I am running a set of experiments on the HOSP dataset:
1k - 100k tuples with 2%-10% noise introduced by myself.
HOSP is provided within NADEEF github repository.

At some point, I am getting OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Thread-762" java.lang.OutOfMemoryError: GC overhead limit exceeded

I increased the memory usage, and run NADEEF on Linux machine with the following configuration:

java -Xmx14G -cp out/bin/*:examples/:out/test qa.qcri.nadeef.console.Console
Any idea, what else I could configure to get NADEEF running without OOM errors?

Thank you for your help.

[hammady]

Hi @visenger
Can you please share the dataset after introducing the noise?
It is also helpful to share the full error log.
Thank you.

[visenger]

Hi @hammady , I created a repo for this https://github.com/visenger/noiselog.git

Let me know if you have any questions. Thank you!

[hammady]

Thanks @visenger, do you remember which $i/$j combinations gave you the memory issue?

[visenger]

Hi Hossam, I run this experimens several times, and the OOM issue is not deterministic. E.g. i=4%; j=40k or i=8%; j=100k
Different combination and different order execution can lead to the different points of OOM.
Here is an example of the current setting: https://github.com/visenger/noiselog/blob/master/args.txt

Thank you a lot @hammady !

[hammady]

For documentation purposes, here is the error trace:

Exception in thread "Thread-3" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3236)
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
        at qa.qcri.nadeef.core.pipeline.ViolationExportToCSV.execute(ViolationExportToCSV.java:90)
        at qa.qcri.nadeef.core.pipeline.ViolationExportToCSV.execute(ViolationExportToCSV.java:34)
        at qa.qcri.nadeef.core.pipeline.Node.execute(Node.java:68)
        at qa.qcri.nadeef.core.pipeline.Flow$1.run(Flow.java:174)

[hammady]

@visenger I have changed the way violations table is written to CSV, this should fix the memory problem (at least at this point of code). However, please note that currently NADEEF is not scalable and it may not be able to handle big data. We are working on a scalable NADEEF on a private repo if you are interested.

[visenger]

@hammady Thank you a lot! I will check this out.