Hi,
I am having an issue with the rolling apply method combinated with numpy's nanmean method. Here is my isssue:
import pandas as pd
import numpy as np
>>> df = pd.DataFrame({'A': [np.nan, np.nan, np.nan, 5, np.nan, np.nan]})
>>> df
A
0 NaN
1 NaN
2 NaN
3 5.0
4 NaN
5 NaN
>>> df.rolling(3).apply(np.nanmean)
A
0 NaN
1 NaN
2 NaN
3 NaN
4 NaN
5 NaN
Expected Output
After index 3 I was expecting to see 5 in every line, how come I got only Nan?
Thanks for your help!
Output of pd.show_versions()
# Paste the output here
## INSTALLED VERSIONS
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-100-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None
pandas: 0.19.0
nose: 1.3.7
pip: 8.1.2
setuptools: 24.0.3
Cython: None
numpy: 1.11.2
scipy: 0.18.0
statsmodels: 0.6.1
xarray: None
IPython: 5.0.0
sphinx: None
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.7
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.12
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.8
boto: None
pandas_datareader: None
Comment From: jreback
you need to supply min_periods, which defaults to the window size.
Note that using a numpy function directly with .apply is much slower (some are mapped directly to the pandas impl, e.g. np.mean); I suppose np.nan* should be though they only exist in later versions of pandas.
In [1]: df = pd.DataFrame({'A': [np.nan, np.nan, np.nan, 5, np.nan, np.nan]})
In [2]: df.rolling(3).apply(np.nanmean)
Out[2]:
A
0 NaN
1 NaN
2 NaN
3 NaN
4 NaN
5 NaN
In [3]: df.rolling(3).mean()
Out[3]:
A
0 NaN
1 NaN
2 NaN
3 NaN
4 NaN
5 NaN
In [4]: df.rolling(3, min_periods=1).apply(np.nanmean)
Out[4]:
A
0 NaN
1 NaN
2 NaN
3 5.0
4 5.0
5 5.0
In [5]: df.rolling(3, min_periods=1).mean()
Out[5]:
A
0 NaN
1 NaN
2 NaN
3 5.0
4 5.0
5 5.0
Comment From: ngenain
Thanks for your help!