Hi there,
I have a ginormous problem right now.
When doing simple operations, a python interpreter subprocess keeps running in the background (even when my application finished) and eats a lot of CPU (25%) continually, and forever.
It seems that whenever I do some simple operations, such as:
df / serie2
df.div(serie2)
df - serie2
df - 2
df.apply(minus)
df ** 2
where minus is:
def minus(serie):
serie - serie2
But the following operations do not produce the bug:
df * (1.0/serie2)
df * serie2**-1
df * df
And any operation using * is typically not affected by the bug.
The bug is that if the above operations are present in my code, it launches an instance of Python and then this instance never goes away, and keeps using 25 percent of my CPU!
I am using: Windows 7 x64 Python 2.7 Pandas 0.11 rc1 Numpy 1.7.1 IPython 0.13.2
What is weird is that I have a hard time reproducing it outside of my application (ie: directly inside IPython with simplified code), so I can't really post anything here.
Another thing to note is that when this bug happens compared to when it doesn't (using the workarounds I outlined above by replacing buggy operations by non buggy operations, and having the same result at the end), not only a python subprocess keeps running in the background, but also while my application is running, my app use a LOT more CPU and memory (CPU ~ 60% when buggy, ~10-20% when not buggy).
I'm still trying to debug the problem, but it seems it can't be profiled (because it's not a loop due to my application, my application ends correctly, and then the Python sub-process keeps processing something but I can't know what!).
If someone has an idea to help me debugging I would gladly hear your ideas. Meanwhile this is posted here as a reference if anyone else gets a similar problem.
Comment From: jreback
can you show a sample of your series/frame?
Comment From: lrq3000
As I said, the series/dataframe alone is not sufficient to reproduce the problem, nor is the whole function. You need to try directly my application, but it's not yet published publicly.
Meanwhile, I used the workarounds I've posted above and it seems to work, but I will surely experience this problem again with my next experiments.
My application will be published in about a month, I will then make a link here so that the problem can be reproduced easily.
Comment From: ghost
cool, we'll wait.
marking "someday"
Comment From: cpcloud
Does valgrind work with windows?
Comment From: ghost
never tried, you need to compile python with a special flag for it, might be a pain. see.... https://github.com/pydata/pandas/issues/2894#issuecomment-15651698
Comment From: cpcloud
Try running your app on a UNIX-based system. If you get the same issue you might be able to debug it more easily.
Comment From: lrq3000
Good idea for Valgrind. My app is meant to be run on Linux/Unix systems anyway, but I'm developping it on Windows.
I'll post news here when I'll have more infos.
Comment From: cpcloud
this might be of interest to the op.
Comment From: lrq3000
Yep thank's a lot Philip, I'll give it a try with Smiley, seems to be the right tool to track down my problem!
2013/5/31 Phillip Cloud notifications@github.com
this https://github.com/dhellmann/smiley might be of interest to the op.
— Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3407#issuecomment-18763958 .