ENH: show progress while fetching rows for query

[tswast]

As reported by @QuinRiva in #12 (comment)

Progress is written to logging while a query is running, but no progress is reported between when a query finishes and while the data is being downloaded to be added to a DataFrame. The problem is that we call list(rows_iter) to fetch all pages. Previously, progress was written as each page was downloaded.

https://github.com/pydata/pandas-gbq/blob/08166685d3305a57fbfd3bc4c41a1cf5df98ebcf/pandas_gbq/gbq.py#L294-L299

A possible solution is to loop over rows_iter.pages instead. After the first page is fetched, the rows_iter.total_rows property is available, so it would be possible to display a percent complete or even use tqdm as done in #166.

[max-sixty]

I found tqdm to be really nice; I would vote for that depending on whether we think it's OK to add that as a dependency.

A possible solution is to loop over rows_iter.pages instead.

Yes, and that gives us a bunch of potential performance improvements too!

[JohnPaton]

As a frequent downloader of large tables over a slow connection I'd like to implement this. Questions for @tswast @max-sixty :

• Are you guys okay with adding tqdm as a dependency?
  • If yes: Other logging currently goes via logging, are you okay with having tqdm write to stdout/stderr (the default)? Else it will require one of the solutions from here
  • If no: What format should the row download logging have?

[tswast]

I'd prefer to keep tqdm an optional dependency. (We already use it to show some progress during to_gbq.

But note now that we're using to_dataframe() from the google-cloud-bigquery library, I think adding this logic there would be most appropriate. I believe the RowIterator. _to_dataframe_tabledata_list would be the most appropriate place to use tqdm if it's installed.

tqdm write to stdout/stderr (the default)?

If tqdm is installed, I think following it's default behavior is the most appropriate thing to do.

[tswast]

Oh, and I'm a maintainer of google-cloud-bigquery, too, so it shouldn't be any more scary to contribute over there. We're receptive of PRs to the Google Cloud client libraries, too. :-)

[JohnPaton]

Now that googleapis/google-cloud-python#7552 is merged, should read_gbq get a progress_bar parameter to match to_gbq?

[tswast]

Yeah, once I make a new release of google-cloud-bigquery (I expect to next week), we can start populating the progress bar argument. I'd be okay always populating it as 'tqdm', but it is nice to have the special version for notebooks, so an argument to read_gbq makes sense.

[dakl]

Now that to_dataframe in google-cloud-bigquery supports progress_bar_type, can we make use of that to add a progress bar to read_gbq? @tswast