Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing Job class (attempts) + Waiting for workers to kill BigQuery streams + Fixing workers streams/offsets + Regulate global queue #63

Merged
merged 2 commits into from
Jul 3, 2023

Conversation

levavaq
Copy link
Contributor

@levavaq levavaq commented Jun 28, 2023

1 - Fixing Job class (attempts)

The Job class wasn't accessible for incrementing "attempts" variable and released an uncaught Exception in the code. Now it is.

2 - Waiting for workers to finish their jobs to kill BigQuery streams (#43 )

Sinks used to kill BigQuery streams sometimes before Workers finished their jobs. It caused BigQuery connections issues and the hole Meltano job used to go on an infinite run without giving the possibility to workers to stop themselves.

3 - Each worker has his own offsets (#54 )

Workers could process several tables and sometimes the same that an other. So we give to each worker a Dict variable that contains keys (the streams he's processing) and values (his offset for this stream/key). This for avoiding BogQuery's offset issue between all connections.

4 - Adding a MAX_JOBS_QUEUED variable to regulate workers's global jobs queue

Sinks wait now that workers finish their queues to add more jobs to process if necessary.

Bonus :

We gave the possibility to user to override the constant variables in tests.

@levavaq levavaq changed the title Fixing Job class (attempts) + Waiting for workers to finish their jobs to kill BigQuery streams Fixing Job class (attempts) + Waiting for workers to kill BigQuery streams + Fixing workers streams/offsets + Regulate global queue Jun 28, 2023
@levavaq levavaq closed this Jun 28, 2023
@levavaq levavaq reopened this Jun 28, 2023
@z3z1ma z3z1ma merged commit 9cfc820 into z3z1ma:main Jul 3, 2023
@firehist
Copy link
Contributor

Hey @z3z1ma

It seems that this PR revert the #53 PR merged prior.
Maybe a rebase problems!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants