When I create a pandas.Series or pandas.DataFrame having np.NaN values, I get the following warning.
Example:
>>> import pandas as pd, numpy as np
>>> pd.Series([np.nan,np.nan,np.nan],index=[1,2,3])
1 NaN
2 NaN
3 NaN
dtype: float64
C:\Users\domen\Anaconda\lib\site-packages\pandas\core\format.py:1995: RuntimeWarning: invalid value encountered in absolute
abs_vals = np.abs(self.values)
C:\Users\domen\Anaconda\lib\site-packages\pandas\core\format.py:1998: RuntimeWarning: invalid value encountered in greater
has_large_values = (abs_vals > 1e8).any()
C:\Users\domen\Anaconda\lib\site-packages\pandas\core\format.py:1999: RuntimeWarning: invalid value encountered in less
has_small_values = ((abs_vals < 10 ** (-self.digits)) &
C:\Users\domen\Anaconda\lib\site-packages\pandas\core\format.py:2000: RuntimeWarning: invalid value encountered in greater
(abs_vals > 0)).any()
I am using Anaconda-2.2.0-Windows-x86_64, with Python 2.7.9. and pandas 0.16.0.
I would ask you to advise.
Comment From: jreback
this was fixed by #9764 and will be in 0.16.1.
Comment From: j1729
I'm experiencing the exact same problem, but I have 0.16.1. Did the fix not make it in, or am I doing something wrong? Thanks in advance for your help.
>>> import pandas as pd, numpy as np
>>> pd.__version__
'0.16.1'
>>> pd.Series([np.nan,np.nan,np.nan],index=[1,2,3])
1 NaN
2 NaN
3 NaN
dtype: float64
C:\Anaconda\lib\site-packages\pandas\core\format.py:2009: RuntimeWarning: invalid value encountered in absolute
abs_vals = np.abs(self.values)
C:\Anaconda\lib\site-packages\pandas\core\format.py:2012: RuntimeWarning: invalid value encountered in greater
has_large_values = (abs_vals > 1e8).any()
C:\Anaconda\lib\site-packages\pandas\core\format.py:2013: RuntimeWarning: invalid value encountered in less
has_small_values = ((abs_vals < 10 ** (-self.digits+1)) &
C:\Anaconda\lib\site-packages\pandas\core\format.py:2014: RuntimeWarning: invalid value encountered in greater
(abs_vals > 0)).any()
>>>
Comment From: jreback
@domenb you should check whether you have a 'partially' updated installation. There was an older bug with conda that would if you were running python processes not update fully. Do this.
- make sure all python processes are killed
- conda remove pandas
- conda install pandas
Comment From: j1729
I did the following:
- check that there were no python processes in the task manager
- conda remove pandas
- conda remove numpy (just in case)
- confirmed that the pandas and numpy folders were gone (in C:\Anaconda\Lib\site-packages)
- confirmed that pandas and numpy would not import
- check for no python processes
- conda install pandas
C:\>conda install pandas
Fetching package metadata: Could not connect to https://repo.continuum.io/pkgs/f
ree/noarch/
.Could not connect to https://repo.continuum.io/pkgs/free/win-32/
Could not connect to https://repo.continuum.io/pkgs/pro/noarch/
.Could not connect to https://repo.continuum.io/pkgs/pro/win-32/
..
Solving package specifications: .
Package plan for installation in environment C:\Anaconda:
The following NEW packages will be INSTALLED:
numpy: 1.9.2-py27_0
pandas: 0.16.1-np19py27_0
Proceed ([y]/n)? y
Linking packages ...
[ COMPLETE ]|##################################################| 100%
I then repeated the above test, and the behavior was identical. I have conda 3.11.0. I just deleted and reinstalled Anaconda last week. It is Anaconda-2.2.0-Windows-x86. I have numpy 1.9.2. I don't know if @domenb got this working, but please let me know if you think this is unrelated, or if I should create a separate issue.
Thanks
Comment From: jorisvandenbossche
@jreback reopened, so we certainly look at it. The warnings also appeared in #10256 (also 0.16.1)
Comment From: jreback
This is a user installation problem. I would guess that they have maybe anaconda AND miniconda installed and picking up a different version. So as I said above. need to look at the environment.
In [7]: pd.__version__
Out[7]: '0.16.1'
In [8]: pd.Series([np.nan,np.nan,np.nan],index=[1,2,3])
Out[8]:
1 NaN
2 NaN
3 NaN
dtype: float64
Comment From: cousin333
I am the one who have created #10256, and - just as jorisvandenbossche wrote above - I've also experienced the problem, while writing to the console. That is, I wrote it into a "normal" console (included in the Spyder IDE), not an IPython one as you did above. In the IPython Console (or Notebook) it also works just fine. Besides I am using neither conda nor miniconda, but the latest WinPython, that includes 0.16.1 "by default".
>>> import numpy as np
>>> import pandas as pd
>>> pd.__version__
'0.16.1'
>>> pd.Series([np.nan,np.nan,np.nan],index=[1,2,3])
1 NaN
2 NaN
3 NaN
dtype: float64
R:\WinPython\python-3.4.3.amd64\lib\site-packages\pandas\core\format.py:2012: RuntimeWarning: invalid value encountered in greater
has_large_values = (abs_vals > 1e8).any()
R:\WinPython\python-3.4.3.amd64\lib\site-packages\pandas\core\format.py:2013: RuntimeWarning: invalid value encountered in less
has_small_values = ((abs_vals < 10 ** (-self.digits+1)) &
R:\WinPython\python-3.4.3.amd64\lib\site-packages\pandas\core\format.py:2014: RuntimeWarning: invalid value encountered in greater
(abs_vals > 0)).any()
>>>
I have already included my environment parameters in the above mentioned #10256.
Comment From: jreback
Well, this is 2.7.9, numpy 1.9.2, pandas 0.16.1 on windows. And its python, not ipython. Still no error. So you'll have to step thru the debugger and see if you can figure out what the issue.
Python 2.7.9 |Continuum Analytics, Inc.| (default, Dec 18 2014, 16:57:52) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>> import pandas as pd, numpy as np
>>> pd.Series([np.nan,np.nan,np.nan],index=[1,2,3])
1 NaN
2 NaN
3 NaN
dtype: float64
>>> pd.__version__
'0.16.1'
>>> np.__version__
'1.9.2'
>>> pd.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.9.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 62 Stepping 4, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
pandas: 0.16.1
nose: None
Cython: None
numpy: 1.9.2
scipy: 0.15.1
statsmodels: None
IPython: 2.4.1
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2015.2
bottleneck: None
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.2
openpyxl: None
xlrd: 0.9.3
xlwt: None
xlsxwriter: 0.7.2
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.4
pymysql: None
psycopg2: None
Comment From: quantstud
Hi I am encountering the same issue. I have the latest versions of pandas and numpy. Details are below.
h
/anaconda3/lib/python3.4/site-packages/pandas/core/format.py:2012: RuntimeWarning: invalid value encountered in greater
has_large_values = (abs_vals > 1e8).any()
/anaconda3/lib/python3.4/site-packages/pandas/core/format.py:2013: RuntimeWarning: invalid value encountered in less
has_small_values = ((abs_vals < 10 ** (-self.digits+1)) &
/anaconda3/lib/python3.4/site-packages/pandas/core/format.py:2014: RuntimeWarning: invalid value encountered in greater
(abs_vals > 0)).any()
Out[135]:
X0 X1 X2
DATE
2013-01-01 00:00:00+00:00 -0.220539 0.166106 NaN
2013-01-02 00:00:00+00:00 -0.229826 0.304425 -0.023634
pd.__version__
Out[136]: '0.16.1'
np.__version__
Out[137]: '1.9.2'
Comment From: iyer
I come across the same issue too. I did a fresh conda install (anaconda-3.2.3) on my linux machine and ran into issues when dataframes started containing nans
Below are the pandas and numpy versions
pd.version Out[16]: '0.16.2'
np.version Out[17]: '1.9.2'
This relates to formatting floats (which is what an np.nan is represented as). You can work around it by defining your own formatter function and setting it as an option.
pd.set_option('display.float_format', lambda x:'%f'%x)
Comment From: jdmcbr
conda install -f pandas solved this for me, whereas conda remove pandas followed by conda install pandas had not.
Comment From: nickeubank
@quantstud @jdmcbr @cousin333 Has this problem stayed fixed for you? I'm still having issues with this -- conda remove pandas / conda install pandas not fixing.
Comment From: dacoex
The issue as mentioned above under spyder (RuntimeWarning: with pandas.dataframes spyder-ide/spyder#2991) aslo occurs when using WinPython instead of conda.
Comment From: nickeubank
@dacoex have we concluded this is a spyder problem and not a pandas problem?
Comment From: jdmcbr
@nickeubank It was not actually fixed, as it turned out. I looked through the issue @dacoex mentions, and that matches my experiences perfectly. After trying the conda install -f pandas suggested here, I happened to have good (or bad, I suppose) luck to not run into the error in multiple tests, so incorrectly thought it was solved. I forgot to come back and say so next time I ran into the error. In any case, it looks like an issue with spyder, not pandas.
Comment From: nickeubank
K - @jreback , sounds like this is still an issue for many people. Could we re-open?
Comment From: jdmcbr
@nickeubank I just tested, and only saw this issue in a python console within Spyder. It did not occur in a python console outside of Spyder. So I think it is appropriately closed here.
Comment From: jreback
again I suspect its a non-complete update for some folks. esp on windows, you have to make sure that nothing is touching the env when you updated things. This has been fixed in newer versions of conda, but if you are doing it some other way then you may have issues.
Comment From: ccordoba12
(Spyder maintainer here). My findings are: 1. This problem only occurs inside Spyder consoles, and it doesn't have to do with wrong conda installations. I still don't understand why it occurs only in Spyder. 2. I can reproduce this problem with Pandas 0.17 but not with Pandas 0.18. So to those affected by this bug I suggest you to update :-) 3. For those who are unable to update, I'll provide a fix for it in Spyder 2.3.9 (to be released this week :-)
Conclusion: Please stop bothering @jreback and instead bother me in spyder-ide/spyder#2991 ;-)
Comment From: ccordoba12
@jreback, is it ok for us to simply filter this warning? I still don't understand where it's coming from (we import Pandas in several places, but those imports don't seem to have an effect on suppressing the warning).
Comment From: jreback
@ccordoba12 yeah all of this code was changed for 0.18.0 (unreleated to this, much more to make it DRY & to make sure FloatIndexes were printing correctly).
I relooked at this issue, its on 32-bit windows. See the warning (coming from numpy) below. I would say this is really a numpy bug. I'll create an issue. As far as filtering, sure I don't think it actually hurts anything, but I don't think you need to on 0.18.0.
[pandas_test] C:\Miniconda3-32\Scripts>python
Python 2.7.11 |Continuum Analytics, Inc.| (default, Jan 29 2016, 15:36:56) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import numpy as np
>>> np.__version__
'1.9.2'
>>> np.abs(np.array([0,np.nan],dtype='float64'))
__main__:1: RuntimeWarning: invalid value encountered in absolute
array([ 0., nan])
>>>
Comment From: jreback
xref https://github.com/numpy/numpy/issues/7440
Comment From: mhabets
I get the same warning message on linux 64bits in Spyder 2.3.8 as soon as I type the last parenthesis of df[df['Email Domain'].isin(
but not with the mentioned command np.abs(np.array([0,np.nan],dtype='float64'))
Python 3.5.1 |Anaconda 4.0.0 (64-bit)| (default, Dec 7 2015, 11:16:01) IPython 4.1.2 -- An enhanced Interactive Python. Numpy 1.10.4 Pandas 0.18.0
I can't reproduce the warning message with the standard ipython console.
Comment From: ccordoba12
@mhabets, as I said, this is going to be fixed in Spyder 2.3.9.
Comment From: linshilogin
@ccordoba12 the problem is seems not fixed in Spyder 2.3.9
Comment From: khairullahhamsafar
I am getting this type warnings again now in 2025. This problem is raised again. Right now, I am facing on Kaggle:
df1.head()