BUG: Adapt to breaking change in google-cloud-bigquery 0.32.0.dev1 #152

tswast · 2018-03-22T23:06:31Z

There was a breaking change in 0.32.0 which changed the way
configuration for the query job gets loaded. Also, it added the
'description' field to the schema resource, so this change updates the
schema comparison logic to account for that.

Note: I've created this PR based on the code currently in master in the google-cloud-bigquery repo. The tests pass locally, but they won't pass anywhere else until google-cloud-bigquery 0.32.0 is released. I've added the DO NOT MERGE label to this PR for now.

Since pandas-gbq will be broken when google-cloud-bigquery does its next release, I think it would make sense to review this before than so it can be merged when everything is ready. Once it goes in, I can create a pandas-gbq release to minimize the breakage time.

max-sixty · 2018-03-22T23:59:58Z

pandas_gbq/gbq.py

-                del field['mode']
+        fields_remote = self._clean_schema_fields(
+            self.schema(dataset_id, table_id))
+        fields_local = self._clean_schema_fields(schema['fields'])


max-sixty · 2018-03-23T00:04:04Z

I think we should either make this backwards compatible or check the version on import and raise. Otherwise people are going to get very obscure error messages if their imports aren't correct (which is very possible with pinned installs)

tswast · 2018-03-23T00:07:53Z

Agreed. I had a hard time figuring out how to make this one backwards compatible since old versions would accept the config but then the API call to BigQuery fails later on.

I've updated the version number in _check_google_client_version(), so folks should get an understandable error message.

max-sixty · 2018-03-23T00:11:17Z

Though IIUC it's only backward incompatible for those passing in config dicts, so a very small share of users. Not worth a big lift, then

max-sixty · 2018-03-23T00:11:42Z

I've updated the version number in _check_google_client_version(), so folks should get an understandable error message.

Ah great! (some crossed wires with my last msg)

max-sixty · 2018-03-23T00:13:20Z

pandas_gbq/gbq.py

+                                     "inside config while it is specified "
+                                     "as parameter")
+                query = config['query']['query']
+                del config['query']['query']


no worth changing, but these two could be query = config['query'].pop('query')

max-sixty · 2018-03-23T00:17:32Z

We could update the final build to also use master from google.cloud.bigquery as well as pandas - would be generally helpful as well as for this

tswast · 2018-03-23T20:54:02Z

@maxim-lian Good idea. I've made the "MASTER" build use the version of google-cloud-python hosted on GitHub. 5f427a0

I'll keep an eye on the MASTER build on Travis to see if it works. ~~https://travis-ci.org/tswast/pandas-gbq/builds/357586225~~ ~~https://travis-ci.org/tswast/pandas-gbq/builds/357588439~~ https://travis-ci.org/tswast/pandas-gbq/builds/357606304

max-sixty · 2018-03-23T21:25:25Z

.travis.yml

@@ -28,6 +28,7 @@ install:
      conda install -q numpy pytz python-dateutil;
      PRE_WHEELS="https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com";
      pip install --pre --upgrade --timeout=60 -f $PRE_WHEELS pandas;
+      pip install -e 'git+https://github.com/GoogleCloudPlatform/google-cloud-python.git#egg=version_subpkg&subdirectory=bigquery';


I've struggled with these from time to time. I think the egg=version_subpkg is like foo and can be excluded. But if this works then great

Probably. I don't really want to mess with it, though. I got it from this answer on installing with pip from a git subdirectory.

tswast · 2018-03-23T22:04:40Z

Finally got a green build on MASTER. https://travis-ci.org/tswast/pandas-gbq/jobs/357606308 Got failures on the other builds as expected, since google-cloud-bigquery hasn't released 0.32.0 yet.

There was a breaking change in 0.32.0.dev1 which changed the way configuration for the query job gets loaded. Also, it added the 'description' field to the schema resource, so this change updates the schema comparison logic to account for that. Updates the MASTER build in CI to also build with google-cloud-bigquery at MASTER.

codecov-io · 2018-04-03T18:24:57Z

Codecov Report

Merging #152 into master will decrease coverage by 42.49%.
The diff coverage is 64.7%.

@@            Coverage Diff             @@
##           master     #152      +/-   ##
==========================================
- Coverage   78.87%   36.37%   -42.5%     
==========================================
  Files           8       10       +2     
  Lines        1529     1545      +16     
==========================================
- Hits         1206      562     -644     
- Misses        323      983     +660

Impacted Files	Coverage Δ
pandas_gbq/tests/test_gbq.py	`28.08% <0%> (-63.21%)`	⬇️
pandas_gbq/tests/test__query.py	`100% <100%> (ø)`
pandas_gbq/gbq.py	`31.97% <26.31%> (-44.96%)`	⬇️
pandas_gbq/_query.py	`84.61% <84.61%> (ø)`
pandas_gbq/_load.py	`62.5% <0%> (-32.5%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9666965...c636448. Read the comment docs.

tswast · 2018-04-03T18:26:25Z

@maxim-lian I figured out how to make it backwards compatible so that we don't have to force upgrade from 0.29.0 versions. Basically, I use the same version detection code we're already doing in the _check_google_client_version() function but in a new function to construct the query configuration.

Please take a look. With this new way of doing it, I think we can merge without waiting to sync with a release of google-cloud-bigquery.

tswast · 2018-04-03T18:27:05Z

New Travis build:

max-sixty · 2018-04-03T20:06:28Z

Very nice! Good job on the backward-compat.

max-sixty · 2018-04-03T20:08:28Z

Re _query.py - I haven't seen other libraries that use private module names. It's fine and not worth changing, but generally libraries put the public API in pandas_gbq.__init__, and then there's no need to make the module names private.

tswast · 2018-04-03T20:17:57Z

generally libraries put the public API in pandas_gbq.init, and then there's no need to make the module names private.

Makes sense. Filed #154 to follow-up with some module renames.

tswast added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 22, 2018

tswast self-assigned this Mar 22, 2018

tswast requested review from parthea and max-sixty March 22, 2018 23:06

tswast mentioned this pull request Mar 22, 2018

test read_gbq with dryRun in configuration parameter #88

Open

max-sixty reviewed Mar 22, 2018

View reviewed changes

max-sixty reviewed Mar 23, 2018

View reviewed changes

tswast mentioned this pull request Mar 23, 2018

BigQuery: Add dev version for 0.32.0 release googleapis/google-cloud-python#5105

Merged

max-sixty reviewed Mar 23, 2018

View reviewed changes

tswast force-pushed the gcb-0.32.0 branch from 356dadc to 9492275 Compare March 26, 2018 17:27

tswast mentioned this pull request Mar 26, 2018

Printing rather than logging? #12

Closed

tswast added 4 commits April 3, 2018 10:03

DOC: Changelog to indicate reason for deps upgrade.

244d5f8

DOC: Add verbose deprecation to changelog.

3536cf4

Detect google-cloud-bigquery version for backwards compatibility

3110f8d

tswast force-pushed the gcb-0.32.0 branch from b66e2d4 to 3110f8d Compare April 3, 2018 18:20

pop

3e34f58

tswast added 2 commits April 3, 2018 11:31

Add installed version arg

3919a1c

Add validation for old versions. Fix schema tests for old versions.

74952de

tswast mentioned this pull request Apr 3, 2018

CLN: No need to use private module names. #154

Closed

tswast changed the title ~~BUG: Update pandas-gbq to latest version of google-cloud-bigquery~~ BUG: Adapt to breaking change in google-cloud-bigquery 0.32.0.dev1 Apr 3, 2018

Update changelog.

c636448

tswast merged commit 6294f7a into googleapis:master Apr 3, 2018

tswast deleted the gcb-0.32.0 branch April 3, 2018 20:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Adapt to breaking change in google-cloud-bigquery 0.32.0.dev1 #152

BUG: Adapt to breaking change in google-cloud-bigquery 0.32.0.dev1 #152

tswast commented Mar 22, 2018

max-sixty Mar 22, 2018

max-sixty commented Mar 23, 2018

tswast commented Mar 23, 2018

max-sixty commented Mar 23, 2018

max-sixty commented Mar 23, 2018

max-sixty Mar 23, 2018

max-sixty commented Mar 23, 2018

tswast commented Mar 23, 2018 •

edited

Loading

max-sixty Mar 23, 2018

tswast Mar 23, 2018

max-sixty Mar 23, 2018

tswast commented Mar 23, 2018

codecov-io commented Apr 3, 2018 •

edited

Loading

tswast commented Apr 3, 2018

tswast commented Apr 3, 2018 •

edited

Loading

max-sixty commented Apr 3, 2018

max-sixty commented Apr 3, 2018

tswast commented Apr 3, 2018

BUG: Adapt to breaking change in google-cloud-bigquery 0.32.0.dev1 #152

BUG: Adapt to breaking change in google-cloud-bigquery 0.32.0.dev1 #152

Conversation

tswast commented Mar 22, 2018

max-sixty Mar 22, 2018

Choose a reason for hiding this comment

max-sixty commented Mar 23, 2018

tswast commented Mar 23, 2018

max-sixty commented Mar 23, 2018

max-sixty commented Mar 23, 2018

max-sixty Mar 23, 2018

Choose a reason for hiding this comment

max-sixty commented Mar 23, 2018

tswast commented Mar 23, 2018 • edited Loading

max-sixty Mar 23, 2018

Choose a reason for hiding this comment

tswast Mar 23, 2018

Choose a reason for hiding this comment

max-sixty Mar 23, 2018

Choose a reason for hiding this comment

tswast commented Mar 23, 2018

codecov-io commented Apr 3, 2018 • edited Loading

Codecov Report

tswast commented Apr 3, 2018

tswast commented Apr 3, 2018 • edited Loading

max-sixty commented Apr 3, 2018

max-sixty commented Apr 3, 2018

tswast commented Apr 3, 2018

tswast commented Mar 23, 2018 •

edited

Loading

codecov-io commented Apr 3, 2018 •

edited

Loading

tswast commented Apr 3, 2018 •

edited

Loading