Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: map "if_exists" value to LoadJobConfig.WriteDisposition #583

Merged
merged 61 commits into from
Nov 7, 2022
Merged

feat: map "if_exists" value to LoadJobConfig.WriteDisposition #583

merged 61 commits into from
Nov 7, 2022

Conversation

aribray
Copy link
Contributor

@aribray aribray commented Oct 27, 2022 •
edited
Loading

Use LoadJobConfig.WriteDisposition to replace if_exists='fail'/'replace'/'append' behavior in to_gbq()

The WriteDisposition value specifies the action that occurs if destination table already exists.

  • if_exists="fail" ----> write_disposition="WRITE_EMPTY"
  • if_exists="append" ----> write_disposition="WRITE_APPEND"
  • if_exists="replace" ----> write_disposition="WRITE_TRUNCATE"

By using WriteDisposition, we can move away from the delete_and_recreate_table() behavior

Fixes #118 #43 #453

aribray added 26 commits October 21, 2022 15:07
@aribray
feat: add WriteDispositon to to_gbq
70927c5
@aribray
pass test_load unit tests
15de9f3
@aribray
nox -s blacken
c44f1a4
@aribray
test write_disposition parameter
bc8faf5
@aribray
refactor tests for WriteDisposition
c3ed7a7
@aribray
remove unused import
936113e
@aribray
update minimum google-auth version to 2.13.0
8613458
@aribray
update constraints-3.7.txt
bc09929
@aribray
bump google-api-core version to 2.10.2 and google-auth-oauthlib to 0.7.0
f7394bb
@aribray
remove version constraints from google-api-core
0d45665
@aribray
bump google-cloud-bigquery version to 3.3.5, google-cloud-bigquery-st…
0e4be09 
…orage version to 2.16.2, and pydata-google-auth version to 1.4.0
@aribray
bump pandas version, bump db-dbtypes version
57c0bf9
@aribray
fix pandas version
16411c9
@aribray
resolve dependency conflicts
cb651d2
@aribray
bump dbtypes version
81736e2
@aribray
bump circleci pandas version
a006ac0
@aribray
rename conda requirements file
1fa654e
@aribray
resetcircleci config pandas version
0407ae5
@aribray
reset pandas version
3680702
@aribray
readjust constraints-3.7 versions
846a44d
@aribray
test adding to kokoro
f44704b
@aribray
test removing circleci
f3ced37
@aribray
test removing circleci
ad45c6d
@aribray
Revert "test removing circleci"
6c771d9 
This reverts commit ad45c6d.
@aribray
add system tests to github workflow
5f23bd0
@aribray
refactor to_gbq to map 'if_exists' to write_disposition
0a96c98
@product-auto-label product-auto-label bot added size: l Pull request size is large. api: bigquery Issues related to the googleapis/python-bigquery-pandas API. labels Oct 27, 2022
@aribray aribray changed the title Aribray to gbq feat: map "if_exists" value to LoadJobConfig.WriteDisposition Oct 27, 2022
@aribray aribray added the owlbot:run Add this label to trigger the Owlbot post processor. label Oct 30, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Oct 30, 2022
@aribray aribray added the owlbot:run Add this label to trigger the Owlbot post processor. label Oct 30, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Oct 30, 2022
@aribray aribray added the owlbot:run Add this label to trigger the Owlbot post processor. label Oct 30, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Oct 30, 2022
@aribray aribray marked this pull request as ready for review October 30, 2022 03:16
@aribray aribray requested a review from a team as a code owner October 30, 2022 03:16
@aribray aribray requested review from a team, prash-mi and tswast October 30, 2022 03:16
@aribray aribray added the owlbot:run Add this label to trigger the Owlbot post processor. label Oct 30, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Oct 30, 2022
@aribray
Copy link
Contributor Author

aribray commented Oct 31, 2022

@tswast are the version bumps/PR description added to the Changelog automatically?

@aribray aribray requested a review from shollyman October 31, 2022 17:16
@tswast
Copy link
Collaborator

tswast commented Oct 31, 2022

@tswast are the version bumps/PR description added to the Changelog automatically?

No, that needs to be included in the commit message when we 'squash and merge'. See: https://github.com/googleapis/release-please#what-if-my-pr-contains-multiple-fixes-or-features

@aribray aribray requested a review from rosiezou November 2, 2022 15:43
shollyman
shollyman approved these changes Nov 6, 2022
View reviewed changes
Copy link
Collaborator

@shollyman shollyman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together. Switching to leveraging write disposition cleans things up nicely.

pandas_gbq/gbq.py Show resolved Hide resolved
@aribray aribray merged commit 7389cd2 into main Nov 7, 2022
@aribray aribray deleted the aribray--to-gbq branch November 7, 2022 17:36
@release-please release-please bot mentioned this pull request Nov 7, 2022
Merged
@ssuhre ssuhre mentioned this pull request Jan 30, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shollyman shollyman shollyman approved these changes

@prash-mi prash-mi Awaiting requested review from prash-mi

@tswast tswast Awaiting requested review from tswast

@rosiezou rosiezou Awaiting requested review from rosiezou

Assignees
No one assigned
Labels
api: bigquery Issues related to the googleapis/python-bigquery-pandas API. size: m Pull request size is medium.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

CLN: Use WriteDisposition for if_exists=fail/replace/append behavior in to_gbq
4 participants
@aribray @tswast @shollyman @rosiezou