The .expanding()
function is super cool. It's order of magnitude faster than
the obvious numpy
alternative:
import numpy
import pandas
numpy.random.seed(123)
df = pandas.DataFrame((numpy.random.normal(0, 1, [10000, 2])), columns=['Value', 'Funny'])
%timeit method_2 = df.Value.expanding(0).apply(lambda x: x.sum())
%timeit method_1 = numpy.array([df.Value.iloc[range(j + 1)].sum() for j in range(df.shape[0])])
But it only works for univariate functions. Would it be possible to extend it to multiple columns while keeping the time efficiency? The idea would be to be able to compute functions involving several columns on expanding window of the data. Say something like:
def loc_fun(x, y):
return numpy.linalg.det(numpy.cov([x, y]))
df.expanding(3).apply(lambda x: loc_fun(x.Value, x.Funny))
faster than with numpy
and list comprehension.
Expected Output
The result of say:
numpy.array([loc_fun(df.Value.iloc[range(j + 1)],df.Funny.iloc[range(j + 1)]) for j in range(3, df.shape[0])])
...but with the nicer speed;)
Output of pd.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-32-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
pandas: 0.21.0
pytest: None
pip: 9.0.1
setuptools: 38.2.5
Cython: None
numpy: 1.13.3
scipy: None
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.6.5
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 0.7.3
lxml: None
bs4: None
html5lib: 1.0b10
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
[paste the output of ``pd.show_versions()`` here below this line]
Comment From: TomAugspurger
This looks like a duplicate of https://github.com/pandas-dev/pandas/issues/15095
tldr: something we'd like to support, but we have backwards compatibility concerns so we have to come up with a new API. Post there if you have thoughts