Airbyte command line - select streams to sync via cmd argument

[vovavovavovavova]

Tell us about the problem you're trying to solve

Now, we have a connector with a lot of streams. In UI, the end user may choose which streams to sync. By commandline, we use all the streams.

Sometimes we need to debug issues and test only some of streams. To avoid long execution, we need to cut the part of configured catalog, paste to another json, and comment all the extra streams in the code.

Describe the solution you’d like

Add commandline argument --streams. Now it works like:
python main.py read --config ... --catalog ... --state ...
We can have an extra option, for example --streams {stream1, stream2, stream3} or --streams {not{ stream1, stream6}}
On execution, the configured catalog is cut to only the required streams. The return of MySource.streams is cut to only the required streams.

Describe the alternative you’ve considered or used

Manually/programabably cut the confugured catalog, make it's copy, comment out all the extra streams manually, then uncomment everything after sync is done.

Additional context

Add any other context or screenshots about the feature request here.

Are you willing to submit a PR?

Remove this with your answer :-)

[jrhizor]

Instead of adding this feature to the protocol (and require changing all connectors / supporting this in the future) can we instead create a simple shell script to extract a catalog for specific streams?

Something like:

$ ls .
config.json
catalog.json
$ some_script catalog.json catalog2json --streams stream1, stream2 > catalog2.json
$ python main.py read --config config.json --catalog catalog2.json
...