Updating from 0.17.1 to 0.18.0 breaks group_keys=False.
>>> data = pd.DataFrame([1, 2, 3, 4], index=pd.date_range("20000101", periods=4))
>>> data.groupby(data.index, group_keys=False).resample('4H', how='last')
0.17.1:
0
2000-01-01 1
2000-01-02 2
2000-01-03 3
2000-01-04 4
0.18.0:
0
2000-01-01 2000-01-01 1
2000-01-02 2000-01-02 2
2000-01-03 2000-01-03 3
2000-01-04 2000-01-04 4
I also tried .resample('4H').last(), and it fails as well. With sum() in place of resample(), it works correctly.
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-79-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
pandas: 0.18.0
nose: None
pip: 8.0.2
setuptools: 19.6.2
Cython: None
numpy: 1.10.4
scipy: 0.17.0
statsmodels: 0.6.1
xarray: None
IPython: None
sphinx: None
patsy: 0.4.0
dateutil: 2.4.2
pytz: 2015.7
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
boto: None
Comment From: jreback
what exactly are you trying to do?
This is the same in 0.18.0.
In [5]: data.groupby(data.index, group_keys=True).resample('4H', how='last')
/Users/jreback/miniconda/bin/ipython:1: FutureWarning: how in .resample() is deprecated
the new syntax is .resample(...).last()
#!/bin/bash /Users/jreback/miniconda/bin/python.app
Out[5]:
0
2000-01-01 2000-01-01 1
2000-01-02 2000-01-02 2
2000-01-03 2000-01-03 3
2000-01-04 2000-01-04 4
actually I was passing group_keys=True.
Still, what are you trying to do?
Comment From: zstomp
I'm trying to resample without producing NaNs for missing periods. I could just dropna those but I don't like the idea of it eating up my memory.
Comment From: jreback
@zstomp what you are doing in prior versions (or 0.18.0) doesn't accomplish that. You can however, do http://pandas.pydata.org/pandas-docs/stable/timeseries.html#sparse-resampling.
The group_keys arg only affects .apply in any event. This argument is pretty useless actually.
In [63]: >>> data.groupby(data.index, group_keys=False).apply(lambda x: x.resample('4H').last())
Out[63]:
0
2000-01-01 1
2000-01-02 2
2000-01-03 3
2000-01-04 4
Comment From: zstomp
Thanks! The docs solution you suggested is very flexible but it looks like a really ugly hack. The one in your comment is much nicer. Why is it not in the docs, is it inferior performance-wise?
Comment From: jreback
the one in the docs is much more performant. its slightly ugly, hoping to have a better API at some point. Your example in this issue is a bit odd though, you don't have any data that is not on the final daily freq. So maybe you have an example closer to what you are actually doing?
Comment From: zstomp
I have 1-minute market data, 6 hours a day, 5 days a week. If I want to resample('5T') and don't groupby beforehand, I get the whole year filled including off hours, which are by far a majority.
Comment From: jreback
@zstomp these are about the same soln. neither has a good API atm. Thinking about this. If you have some suggestions on how to hint this all ears.
Comment From: zstomp
I'm not experienced enough with pandas to suggest. What I initially did (and the credits go to some StackOverflow user) made sense to me: you select the dates when there is data and resample only those days. My case may be too specific, tough.
Also after looking at the internals of pandas today I thought using TimeGrouper could help it. However, it's not in the docs and is only mentioned in a couple of cookbook recipes, so I'm not sure if it's "public".