Skip to content

Commit

Permalink
fix: allow full replacement of bq schema with table_schema when if_ex…
Browse files Browse the repository at this point in the history
…ists="replace" (googleapis#670)

A previous issue (googleapis#315) noted that when using `if_exists="append"`, pandas should retrieve the existing schema from BQ and update the user-supplied `table_schema` to ensure that all column modes (REQUIRED/NULLABLE) match those of the existing table.

The commit that fixed this issue applies this policy to *all* write_dispositions, but it is only applicable when appending. When replacing, the user should be able to overwrite with new datatypes.
  • Loading branch information
Imperssonator authored Sep 21, 2023
1 parent b350452 commit 80c4609
Showing 1 changed file with 8 additions and 6 deletions.
14 changes: 8 additions & 6 deletions pandas_gbq/gbq.py
Original file line number Diff line number Diff line change
Expand Up @@ -1205,12 +1205,14 @@ def to_gbq(
)
table_connector.create(table_id, table_schema)
else:
# Convert original schema (the schema that already exists) to pandas-gbq API format
original_schema = pandas_gbq.schema.to_pandas_gbq(table.schema)

# Update the local `table_schema` so mode (NULLABLE/REQUIRED)
# matches. See: https://github.com/pydata/pandas-gbq/issues/315
table_schema = pandas_gbq.schema.update_schema(table_schema, original_schema)
if write_disposition == "WRITE_APPEND":
# If appending, ensure types and modes of existing columns match supplied schema
# Convert original schema (the schema that already exists) to pandas-gbq API format
original_schema = pandas_gbq.schema.to_pandas_gbq(table.schema)

# Update the local `table_schema` so mode (NULLABLE/REQUIRED)
# matches. See: https://github.com/pydata/pandas-gbq/issues/315
table_schema = pandas_gbq.schema.update_schema(table_schema, original_schema)

if dataframe.empty:
# Create the table (if needed), but don't try to run a load job with an
Expand Down

0 comments on commit 80c4609

Please sign in to comment.