Code Sample, a copy-pastable example if possible
pd.DataFrame({'b': [True, pd.np.nan, True, pd.np.nan] }).to_sparse().eval('b==True')
outputs:
Out[69]:
0 True
1 True
2
3
dtype: bool
BlockIndex
Block locations: array([0], dtype=int32)
Block lengths: array([3], dtype=int32)
Problem description
The eval above somehow sort out returned boolean mask, instead of getting actual indexes. I understand eval and sparse dataframes are quite special usecase and both are new, but it's something which would very efficiently solve usage of eval on big sparse matrices. At least a note what is (not) supported should be in the docs.
At least I haven't found a way how to get the indexes of True from the above.
The behavior of .query is similar.
Expected Output
behavior of dense output (in a sparse format I believe, but being able to get the meaningful value)
In [4]: pd.DataFrame({'b': [True, pd.np.nan, True, pd.np.nan] }).eval('b==True')
Out[4]:
0 True
1 False
2 True
3 False
dtype: bool
Output of pd.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.14.12-1-ARCH
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
pandas: 0.22.0
pytest: 3.3.1
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: None
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: 1.5.1
bottleneck: None
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.1.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: 0.1.2
fastparquet: None
pandas_gbq: None
pandas_datareader: None
Comment From: jreback
this is not supported by the underlying engine which is numexpr
Comment From: jreback
sparse indexing generally also has very little support PRa welcome - there are quite a few open issues about this
Comment From: hnykda
Got it, thanks
