-
Notifications
You must be signed in to change notification settings - Fork 122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
df.to_gbq(): InvalidSchema when using if_exists = 'append' #317
Comments
It makes sense that it works with
|
The schema dictionary I pass when successfully using 'replace' is the same schema dictionary that fails when calling 'append'. The schema dictionary reflects what is in BigQuery. The schema dictionary successfully generates the BQ table using 'replace', but fails when appending to that same BQ table when using 'append'. Also of note is that the dataframe's columns are strings, some of which I want to turn into dates in BQ, which is why I am supplying a schema. Correct column conversion occurs when using 'replace', but perhaps that fails when using 'append'. Using 'replace' + the schema dictionary, to_gbq() correctly turns some columns into dates and uploads to table. Using 'append' + the schema dictionary, the error occurs. |
Looks like #318 should fix. Thanks for looking into it! |
I have this exact issue and it does not appear to be resolved. output = [['Test', 'v1', 'cost_center', '123', date(2020, 1, 31), 30.0], headers = ['scenario', 'version', 'entity', 'account', 'period', 'amount'] dataset_table = 'my_dataset' It successfully creates the table in BQ the first time. Then I run the exact same query a second time and I get the Andrew Good |
@andrewlgood Are you using the version from GitHub? We haven't released |
I do not think so. I am using pandas-gbq 0.13.1. with Python 3.8.2 on a Window 10 machine
From: Tim Swast <[email protected]>
Sent: Thursday, May 14, 2020 10:18 AM
To: pydata/pandas-gbq <[email protected]>
Cc: andrewlgood <[email protected]>; Mention <[email protected]>
Subject: Re: [pydata/pandas-gbq] df.to_gbq(): InvalidSchema when using if_exists = 'append' (#317)
@andrewlgood <https://github.com/andrewlgood> Are you using the version from GitHub? We haven't released 0.13.2 yet with this fix.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#317 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYPII4JUZ3LXAY4D5TJWVTRRP4QNANCNFSM4LW433IQ> . <https://github.com/notifications/beacon/AEYPIIYUNC2HBU5GROSHPBTRRP4QNA5CNFSM4LW433I2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEV4K72I.gif>
|
@andrewlgood I just released https://pypi.org/project/pandas-gbq/0.13.2/, hopefully that solves your issue. |
Tim,
Thank you for working on this. I just upgraded to pandas-gbq 0.13.2 but still get the error.
Here is the code I am using. I simply drop the table and run the script twice to test.
output2 = [['Test', 'v1', 'cost_center', '123', date(2020, 1, 31), 30.0],
['Test', 'v1', 'cost_center', '345', date(2020, 1, 31), 72.0]]
headers = ['scenario', 'version', 'entity', 'account', 'period', 'amount']
df_output = pd.DataFrame(output2, columns=headers)
#dataset_table = 'my_dataset'
#project_id ='my_project_id'
table_schema = [{'name':'scenario', 'type':'string', 'mode':'required'},
{'name':'version', 'type':'string', 'mode':'required'},
{'name':'entity', 'type':'string', 'mode':'required'},
{'name':'account', 'type':'string', 'mode':'required'},
{'name':'period', 'type':'date', 'mode':'required'},
{'name':'amount', 'type':'float', 'mode':'required'}
]
df_output.to_gbq(destination_table=dataset_table,
project_id= project_id,
if_exists='append',
table_schema=table_schema)
Andrew
From: Tim Swast <[email protected]>
Sent: Thursday, May 14, 2020 11:49 AM
To: pydata/pandas-gbq <[email protected]>
Cc: andrewlgood <[email protected]>; Mention <[email protected]>
Subject: Re: [pydata/pandas-gbq] df.to_gbq(): InvalidSchema when using if_exists = 'append' (#317)
@andrewlgood <https://github.com/andrewlgood> I just released https://pypi.org/project/pandas-gbq/0.13.2/, hopefully that solves your issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#317 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYPII4IGWNEXB2UHU3JV3DRRQHG7ANCNFSM4LW433IQ> . <https://github.com/notifications/beacon/AEYPII47DMFQDD5S73ZEWD3RRQHG7A5CNFSM4LW433I2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEV4Y2SA.gif>
|
Tim,
Question was answered on StackOverflow. The ‘type’=’FLOAT’ must be capitalized (same for string and date).
Thank you for looking into this.
Andrew
From: Tim Swast <[email protected]>
Sent: Thursday, May 14, 2020 11:49 AM
To: pydata/pandas-gbq <[email protected]>
Cc: andrewlgood <[email protected]>; Mention <[email protected]>
Subject: Re: [pydata/pandas-gbq] df.to_gbq(): InvalidSchema when using if_exists = 'append' (#317)
@andrewlgood <https://github.com/andrewlgood> I just released https://pypi.org/project/pandas-gbq/0.13.2/, hopefully that solves your issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#317 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYPII4IGWNEXB2UHU3JV3DRRQHG7ANCNFSM4LW433IQ> . <https://github.com/notifications/beacon/AEYPII47DMFQDD5S73ZEWD3RRQHG7A5CNFSM4LW433I2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEV4Y2SA.gif>
|
I have a schema for my upload dataframe.
to_gbq() runs successfully if I am replacing the table (if_exists='replace')
to_gbq() fails if I am appending to the BQ table with matching schema (if_exists='append'):
InvalidSchema: Please verify that the structure and data types in the DataFrame match the schema of the destination table.
My schemas certainly should not be a problem, I was appending to the same table I just successfully wrote calling to_gbq() with if_exists ='replace' with the same schema dict.
The text was updated successfully, but these errors were encountered: