Quantile method on DataFrame fails when there are NaNs in some specific way. Some cases it handles properly. I guess it's somehow related to cases when some rows and/or columns have only NaNs.
First, some working examples:
>>> pd.DataFrame({"a": [1, 2, 3], "b": [np.nan, np.nan, np.nan]}).quantile(axis=0)
a 2.0
b NaN
Name: 0.5, dtype: float64
>>> pd.DataFrame({"a": [1, 2, 3], "b": [np.nan, np.nan, np.nan]}).quantile(axis=1)
0 1.0
1 2.0
2 3.0
Name: 0.5, dtype: float64
Code Sample, a copy-pastable example if possible
If I change the second element on the first column to NaN, it all breaks. Now there's one row and one column with only NaNs:
>>> pd.DataFrame({"a": [1, np.nan, 3], "b": [np.nan, np.nan, np.nan]}).quantile(axis=0)
a 1.0
b 3.0
Name: 0.5, dtype: float64
>>> pd.DataFrame({"a": [1, np.nan, 3], "b": [np.nan, np.nan, np.nan]}).quantile(axis=1)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-2-a62c307ef5d1> in <module>()
----> 1 pd.DataFrame({"a": [1, np.nan, 3], "b": [np.nan, np.nan, np.nan]}).quantile(axis=1)
/.../lib/python3.5/site-packages/pandas/core/frame.py in quantile(self, q, axis, numeric_only, interpolation)
5153 axis=1,
5154 interpolation=interpolation,
-> 5155 transposed=is_transposed)
5156
5157 if result.ndim == 2:
/.../lib/python3.5/site-packages/pandas/core/internals.py in quantile(self, **kwargs)
3142
3143 def quantile(self, **kwargs):
-> 3144 return self.reduction('quantile', **kwargs)
3145
3146 def setitem(self, **kwargs):
/.../lib/python3.5/site-packages/pandas/core/internals.py in reduction(self, f, axis, consolidate, transposed, **kwargs)
3071 for b in self.blocks:
3072 kwargs['mgr'] = self
-> 3073 axe, block = getattr(b, f)(axis=axis, **kwargs)
3074
3075 axes.append(axe)
/.../lib/python3.5/site-packages/pandas/core/internals.py in quantile(self, qs, interpolation, axis, mgr)
1325 values = _block_shape(values[~mask], ndim=self.ndim)
1326 if self.ndim > 1:
-> 1327 values = values.reshape(result_shape)
1328
1329 from pandas import Float64Index
ValueError: total size of new array must be unchanged
Problem description
In the first case, the quantiles are incorrect and in the second case, the quantile computation should not raise an error.
Expected Output
>>> pd.DataFrame({"a": [1, np.nan, 3], "b": [np.nan, np.nan, np.nan]}).quantile(axis=0)
a 2.0
b NaN
Name: 0.5, dtype: float64
>>> pd.DataFrame({"a": [1, np.nan, 3], "b": [np.nan, np.nan, np.nan]}).quantile(axis=1)
0 1.0
1 NaN
2 3.0
Output of pd.show_versions()
# Paste the output here pd.show_versions() here
INSTALLED VERSIONS
------------------
commit: None
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.9-gnu-1
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
pandas: 0.19.0
nose: None
pip: 8.1.2
setuptools: 27.2.0.post20161106
Cython: None
numpy: 1.11.1
scipy: 0.18.1
statsmodels: None
xarray: None
IPython: 5.1.0
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.7
blosc: None
bottleneck: None
tables: 3.2.3.1
numexpr: 2.6.1
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.13
pymysql: None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: 2.9.4
boto: None
pandas_datareader: None
Comment From: jreback
you are using an older version, these are fixed by 0.19.2, this is a dupe of #14357 , closed in #14536
In [1]: pd.DataFrame({"a": [1, np.nan, 3], "b": [np.nan, np.nan, np.nan]}).quantile(axis=0)
...:
Out[1]:
a 2.0
b NaN
Name: 0.5, dtype: float64
In [2]: pd.DataFrame({"a": [1, np.nan, 3], "b": [np.nan, np.nan, np.nan]}).quantile(axis=1)
Out[2]:
0 1.0
1 NaN
2 3.0
Name: 0.5, dtype: float64