Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gbq.io.to_gbq raises InvalidSchema even thought schema matches #15387

Closed
ghost opened this issue Feb 13, 2017 · 4 comments
Closed

gbq.io.to_gbq raises InvalidSchema even thought schema matches #15387

ghost opened this issue Feb 13, 2017 · 4 comments

Comments

@ghost
Copy link

ghost commented Feb 13, 2017

DF's schema returned by pandas.io.gbq.generate_bq_schema(df):

{'fields': [
{'type': 'TIMESTAMP', 'name': 'field1'}, 
{'type': 'STRING', 'name': 'field2'}, 
{'type': 'FLOAT', 'name': 'field3'}, 
{'type': 'STRING', 'name': 'field4'},
{'type': 'FLOAT', 'name': 'field5'},
{'type': 'STRING', 'name': 'field6'}, 
{'type': 'FLOAT', 'name': 'field7'}, 
{'type': 'STRING', 'name': 'field8'},
{'type': 'STRING', 'name': 'field9'}]}

Schema from BQ CLI:

|- field1: timestamp
|- field2: string
|- field3: float
|- field4: string
|- field5: float
|- field6: string
|- field7: float
|- field8: string
|- field9: string

to_gbq(df,"dataset.table","project",if_exists="append",private_key=service_key_path)
Throws:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/lib/python2.7/site-packages/pandas/io/gbq.py", line 827, in to_gbq
    raise InvalidSchema("Please verify that the structure and "
pandas.io.gbq.InvalidSchema: Please verify that the structure and data types in the DataFrame match the schema of the destination table.
@jorisvandenbossche
Copy link
Member

@mremes Thanks for the report!
We are in the process of moving the gbq functionality into its own repo, would you mind to resubmit your issue over there? https://github.com/pydata/pandas-gbq

@ghost
Copy link
Author

ghost commented Feb 26, 2017

Sure. I'd like to make a pull request for this problem. I'm thinking about making it here because I guess pandas gbq functionality is still used mainly through this package?

@jreback
Copy link
Contributor

jreback commented Feb 26, 2017

closing in favor of pandas-gbq issue as above.

@jreback jreback closed this as completed Feb 26, 2017
@jorisvandenbossche jorisvandenbossche added this to the No action milestone Feb 26, 2017
@jorisvandenbossche
Copy link
Member

@mremes I see you already opened a PR at pandas-gqb, so indeed, that's where the code will be developed from now on, not in pandas anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants