Fixing Job class (attempts) + Waiting for workers to kill BigQuery streams + Fixing workers streams/offsets + Regulate global queue #63
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
1 - Fixing Job class (attempts)
The Job class wasn't accessible for incrementing "attempts" variable and released an uncaught Exception in the code. Now it is.
2 - Waiting for workers to finish their jobs to kill BigQuery streams (#43 )
Sinks used to kill BigQuery streams sometimes before Workers finished their jobs. It caused BigQuery connections issues and the hole Meltano job used to go on an infinite run without giving the possibility to workers to stop themselves.
3 - Each worker has his own offsets (#54 )
Workers could process several tables and sometimes the same that an other. So we give to each worker a Dict variable that contains keys (the streams he's processing) and values (his offset for this stream/key). This for avoiding BogQuery's offset issue between all connections.
4 - Adding a MAX_JOBS_QUEUED variable to regulate workers's global jobs queue
Sinks wait now that workers finish their queues to add more jobs to process if necessary.
Bonus :
We gave the possibility to user to override the constant variables in tests.