Source MongoDB: customize DISCOVER_LIMIT

[alafanechere]

Tell us about the problem you're trying to solve

To make schema discovery faster #8491 limited the schema discovery step to the analysis of 10k records.
Some users might have heterogenous schema spanning on more than this hardcoded limit.

Describe the solution you’d like

We could expose DISCOVER_LIMIT as a parameter in the source configuration to allow users to customize this value, or choose to read all documents in the discovery step.

Describe the alternative you’ve considered or used

Downgrade to version 0.1.8

Additional context

Requested by a user in this Slack conversation

[phammer]

Vote from my side for this suggestion, but for different reasons -> make discovery faster.

Our database scheme is quite clean but we have very big documents in it. We have been in the trial with Airbyte Cloud and the MongoDb source connector always timed out during discovery, we were not able to get it running. Support of Airbyte Cloud couldn't help with the topic (thus I don't have logs).

Hint: having the parameter even configurable per MongoDb collection would be the optimal solution because most collection have simple documents, just a few have very complex, heavy documents.

[mickaelandrieu]

Do you know how can I do to override the current connector and change the DISCOVER_LIMIT ? Sadly, I'm using the docker image in production so I guess I have to build the update again inside my container, but my container probably doesnt have what is required to build a new Java connector.

WDYT ?

[mickaelandrieu]

Also

Hint: having the parameter even configurable per MongoDb collection would be the optimal solution because most collection have simple documents, just a few have very complex, heavy documents.

totally agree