to_gbq fails to append to table because of alleged schema mismatch

[ghost]

DF's schema returned by pandas.io.gbq.generate_bq_schema(df):

{'fields': [
{'type': 'TIMESTAMP', 'name': 'field1'}, 
{'type': 'STRING', 'name': 'field2'}, 
{'type': 'FLOAT', 'name': 'field3'}, 
{'type': 'STRING', 'name': 'field4'},
{'type': 'FLOAT', 'name': 'field5'},
{'type': 'STRING', 'name': 'field6'}, 
{'type': 'FLOAT', 'name': 'field7'}, 
{'type': 'STRING', 'name': 'field8'},
{'type': 'STRING', 'name': 'field9'}]}

Schema from BQ CLI:

|- field1: timestamp
|- field2: string
|- field3: float
|- field4: string
|- field5: float
|- field6: string
|- field7: float
|- field8: string
|- field9: string

to_gbq(df,"dataset.table","project",if_exists="append",private_key=service_key_path)
Throws:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/lib/python2.7/site-packages/pandas/io/gbq.py", line 827, in to_gbq
    raise InvalidSchema("Please verify that the structure and "
pandas.io.gbq.InvalidSchema: Please verify that the structure and data types in the DataFrame match the schema of the destination table.

[ghost]

I'm working on the pull request.

[jreback]

transfered issue from pandas-dev/pandas#15387

[LiChangNY]

Is this pull request still being worked on? I encountered the problem as @medullaskyline described on May 5, 2016. The schema generated by _generate_bq_schema doesn't have mode key, which threw the InvalidSchema error when compared to bq schema. https://github.com/pydata/pandas-gbq/blob/master/pandas_gbq/gbq.py#L1059

[j450h1]

Same here. Python version 2.7 here.

pandas_gbq.gbq.InvalidSchema: Please verify that the structure and data types in the DataFrame match the schema of the destination table.

I compared existing BQ table and the schema of the dataframe and it looks identical. This was working fine for several months.

[j450h1]

Does the mode key have to be applied to my dataframe? How can I do that?

[j450h1]

Disregard. It looks like my new table has a few extra fields.

[avinashsp93]

Same here. Python version 2.7 here.

pandas_gbq.gbq.InvalidSchema: Please verify that the structure and data types in the DataFrame match the schema of the destination table.

I compared existing BQ table and the schema of the dataframe and it looks identical. This was working fine for several months.

I've the same problem, the first time, it creates all the schemas by itself and works fine. Second time if I try to upload identical dataframe, it doesn't work

[tswast]

@avinashsp93 This issue is quite old and the library has changed a lot since then. Could you open a separate issue and include a minimal code example that reproduces the issue?

[harzus]

had the same issues, i just didn't include the field "mode". haven't check whether it cares about "description".here is an example of a table schema with mode:
table_schema = [ {'name': 'dateValue', 'type': 'DATETIME', 'mode':"REQUIRED"}, {'name': 'wert1', 'type': 'FLOAT', 'mode':"NULLABLE"}, {'name': 'wert2', 'type': 'FLOAT', 'mode':"NULLABLE"}, {'name': 'projectNumber', 'type': 'STRING', 'mode':"REQUIRED"}, {'name': 'dateCreated', 'type': 'DATETIME', 'mode':"NULLABLE"} ]

[Benefit-Cometics]

n
@avinashsp93
did you find a solution for the issue? I'm getting the same issue and its 2022

[maria-grigorieva]

I reproduced the same issue today ((

[dharmendrakariya]

I reproduced the same issue today ((

did u figure out?