Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

read_gbq precision lost for floats above 10^4 #14305

Closed
tworec opened this issue Sep 27, 2016 · 0 comments
Closed

read_gbq precision lost for floats above 10^4 #14305

tworec opened this issue Sep 27, 2016 · 0 comments
Labels
Dtype Conversions Unexpected or buggy dtype conversions
Milestone

Comments

@tworec
Copy link
Contributor

tworec commented Sep 27, 2016

pandas.read_gbq will silently cast float values over 10k to integeres causing precision lost

example of the issue

In[1]: 
import pandas as pd
import numpy as np

df=pd.read_gbq('''select * from
                    (select pi(),9999.9,10000.1,200000.2),
                    (select 2.72,9999.9999999,10000.1,300000.3)''',
project_id='CHANGE-ME')

Actual Output

In[2]: print(df)
Out[2]: 
        f0_      f1_    f2_     f3_
0  3.141593   9999.9  10000  200000
1  2.720000  10000.0  10000  300000

In[3]: df.dtypes
Out[3]: 
f0_    float64
f1_    float64
f2_      int64
f3_      int64
dtype: object

Expected Output

In[2]: print(df)
Out[2]: 
        f0_      f1_      f2_       f3_
0  3.141593   9999.9  10000.1  200000.2
1  2.720000  10000.0  10000.1  300000.3

In[3]: df.dtypes
Out[3]: 
f0_    float64
f1_    float64
f2_    float64
f3_    float64
dtype: object

Output of pd.show_versions()

## INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-36-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.19.0rc1+33.g7dedbed
nose: 1.3.7
pip: 8.1.2
setuptools: 25.1.6
Cython: 0.24.1
numpy: 1.11.1
scipy: None
statsmodels: None
xarray: None
IPython: 5.1.0
sphinx: 1.4.6
patsy: None
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.3
openpyxl: 2.4.0
xlrd: 1.0.0
xlwt: None
xlsxwriter: None
lxml: 3.6.4
bs4: 4.5.1
html5lib: 0.999999999
httplib2: 0.9.2
apiclient: 1.5.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

@jreback jreback added this to the 0.20.0 milestone Feb 9, 2017
@jreback jreback added the Dtype Conversions Unexpected or buggy dtype conversions label Feb 9, 2017
@jreback jreback closed this as completed in c23b1a4 Feb 9, 2017
AnkurDedania pushed a commit to AnkurDedania/pandas that referenced this issue Mar 21, 2017
…e 10k

closes pandas-dev#14020
closes pandas-dev#14305

Author: Piotr Chromiec <[email protected]>

Closes pandas-dev#14064 from tworec/read_gbq_full_long_support and squashes the following commits:

788ccee [Piotr Chromiec] BUG: fix read_gbq lost numeric precision
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Dtype Conversions Unexpected or buggy dtype conversions
Projects
None yet
Development

No branches or pull requests

3 participants