Hello,
I'm trying to install pandas on Mac OS X 10.11.6 in a Django project. Pandas library is used with one of the 3rd party libraries that I am using in the project.
Since I don't want to switch to different package manager (Anakonda) just for installing pandas (I'm using pip now), in order for me to get the C files compiled, I decided to install pandas by including --editable git+git://github.com/pydata/pandas.git@v0.18.1#egg=pandas in the requirements.txt file and the library is now installed in my src dir and there's a symlink in my virtualenv's site-packages which points to it.
This gets these headers compiled, but causes many strange importing errors in pandas source code itself when it is imported by a third party library that is also installed with --editable mode. F. ex. file at Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/src/knmi-py/knmi/parsers.py has this line import pandas as pd. And I had to edit this code block in virtualenv/src/pandas/pandas/__init__.py for things to work:
try:
# XXX: I had to comment this out for things to work:
# from pandas import hashtable, tslib, lib
import hashtable, tslib, lib
except ImportError as e: # pragma: no cover
module = str(e).lstrip('cannot import name ') # hack but overkill to use re
raise ImportError("C extension: {0} not built. If you want to import "
"pandas from the source directory, you may need to run "
"'python setup.py build_ext --inplace' to build the C "
"extensions first.".format(module))
But then another importing error appears here: virtualenv/src/pandas/pandas/core/config.py:
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/src/pandas/pandas/core/config.py", line 57, in <module>
import pandas.compat as compat
AttributeError: 'module' object has no attribute 'compat'
What should I do to get these things fixed ? I'd love to be able to use this library with pip and Django.
Comment From: Eimis
Edit: I think I solved this by wrapping editable pandas library in virtualenv/src with another pandas folder which contains the library itself and single __init__.py file. So now it looks like this:
Comment From: chris-b1
Is there any reason you need to compile? pip install pandas will install a pre-compiled wheel.
Comment From: jreback
installation instructions: http://pandas.pydata.org/pandas-docs/stable/install.html
using a distribution is best best.
Comment From: Eimis
@chris-b1 I need to compile, because my Django project uses a third party library KNMI-py and one of the dependencies for this 3rd party library is pandas. This KNMI-py library is importing pandas in its source code and I'm getting an error message that C files are not compiled. These C files are not compiled when you do pip install pandas, they're only compiled when you install the package as --editable. This is the code block from pandas source that does that:
try:
# XXX: these are actualy C files (hashtable.c and etc.):
from pandas import hashtable, tslib, lib
except ImportError as e: # pragma: no cover
module = str(e).lstrip('cannot import name ') # hack but overkill to use re
raise ImportError("C extension: {0} not built. If you want to import "
"pandas from the source directory, you may need to run "
"'python setup.py build_ext --inplace' to build the C "
"extensions first.".format(module))
^ I think my case counts as 'importing pandas from source directory', because a 3rd party library is doing that.
@jreback sorry, I'm not exactly sure I understand what you mean by 'using a distribution', but if you mean conda install pandas, this is not quite suitable for my case, because it would require to switch to a new package manager. And we have a lot of automation tools tied to the requirements.txt file used by pip, so I'd really like to continue using this pip approach..
So unless I'm missing something, I can come up with only one workaround which is forking pandas and including it as an --editable package in my requirements.txt file, which will compile these C files for me.
Comment From: jorisvandenbossche
I'm getting an error message that C files are not compiled. These C files are not compiled when you do pip install pandas, they're only compiled when you install the package as --editable
This is not necessarily true. There are binary wheels available on PyPI for many platforms/python versions, including MacOSX.
I think my case counts as 'importing pandas from source directory', because a 3rd party library is doing that.
KNMI-py is importing pandas in its source code (eg https://github.com/EnergyID/KNMI-py/blob/master/knmi/parsers.py#L1), but it is not importing pandas from the source, that is something different. KNMI-py will just import the installed pandas available in the python runtime, whether it is an editable install or not. So it is not needed to have pandas installed as editable.
So unless I'm missing something, I can come up with only one workaround which is forking pandas and including it as an --editable package in my requirements.txt file, which will compile these C files for me.
That should not be necessary. pip does normally install binary wheels. If that is not the case, you should first check why. And even then, when pip installs the source files, it should also try to compile those during installation.
Unless you want to develop on pandas, installing it as --editable is not needed.
Comment From: Eimis
@jorisvandenbossche But if I am gonna use wheels, how am I gonna automate the setup process of the Project for every developers platform, I can not specify which .whl file to download from pip in my requirements.txt file. I really love the smoothness of pip install -r requirements.txt when everything works for everyone just after executing one command.
Comment From: jorisvandenbossche
But if I am gonna use wheels, how am I gonna automate the setup process of the Project for every developers platform
The same applies for building pandas from source (different needs to build c extensions on each platform).
And again, if you list pandas as requirement, pip will just download the correct wheel, and if there is no wheel available for the specific platform (but there are for windows/linux/macosx), pip will also just install the source and try to build the C extensions, just the same as you would do with an editable install.
Comment From: Eimis
And even then, when pip installs the source files, it should also try to compile those during installation.
Unfortunately when pip installs .whl file, C files are not compiled:
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$ pip install --use-wheel pandas
Collecting pandas
Using cached pandas-0.18.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Requirement already satisfied (use --upgrade to upgrade): pytz>=2011k in ./virtualenv/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in ./virtualenv/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.0 in ./virtualenv/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): six in ./virtualenv/lib/python2.7/site-packages (from python-dateutil->pandas)
Installing collected packages: pandas
Successfully installed pandas-0.18.1
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$ ls virtualenv/lib/python2.7/site-packages/pandas/
__init__.py _sparse.so _version.pyc computation hashtable.so info.py json.so parser.so sparse tools types
__init__.pyc _testing.so algos.so core index.so info.pyc lib.so rpy stats tseries util
_period.so _version.py compat formats indexes io msgpack sandbox tests tslib.so
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
You can see that there are no C files compiled in this directory :(
So I'm still left with the only available solution which is specifying --editable git+git://github.com/pydata/pandas.git@v0.18.1#egg=pandas in my requirements.txt and then wrapping virtualenv/src/pandas/ in another pandas folder with __init__.py file to avoid those importing errors..
Comment From: jorisvandenbossche
You can see that there are no C files compiled in this directory :(
All the .so files are compiled C files
With what you show above, in you just open a python console and import pandas, what do you get? Does that give an error?
Comment From: Eimis
@jorisvandenbossche thanks and sorry for my mistake. It seems that you're right and now C files can be found after installing .whl file. But everything works only after wrapping virtualenv/lib/python2.7/site-packages/pandas in another pandas folder with __init__.py... I still have no idea why those import errors are present in pandas source code.
Comment From: jorisvandenbossche
But everything works only after wrapping virtualenv/lib/python2.7/site-packages/pandas in another pandas folder with init.py...
What does not work? What error do you get? This wrapping should certainly not be needed.
Comment From: Eimis
I'm just getting error from this code block at virtualenv/lib/python2.7/site-packages/pandas/__init__.py:
try:
from pandas import hashtable, tslib, lib
except ImportError as e: # pragma: no cover
module = str(e).lstrip('cannot import name ') # hack but overkill to use re
raise ImportError("C extension: {0} not built. If you want to import "
"pandas from the source directory, you may need to run "
"'python setup.py build_ext --inplace' to build the C "
"extensions first.".format(module))
If I change the import from from pandas import hashtable, tslib, lib to import hashtable, tslib, lib this error is not raised, but then other import errors are raised:
virtualenv/src/pandas/pandas/core/config.py:
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/src/pandas/pandas/core/config.py", line 57, in <module>
import pandas.compat as compat
AttributeError: 'module' object has no attribute 'compat'
Comment From: jorisvandenbossche
but then other import errors are raised
That sounds logical to me that you get errors, as you should totally not wrap the installed pandas in site-packages in another folder.
Please show exactly what you did (output from the console including input and output), and the error you got.
Typically, the error means that you are importing pandas in a session where the current working directory is set at virtualenv/lib/python2.7/site-packages/pandas (something you shouldn't do)
Comment From: Eimis
That sounds logical to me that you get errors, as you should totally not wrap the installed pandas in site-packages in another folder.
Those C files / path related import errors are raised without wrapping anything anywhere. They're raised after fresh install.
Here's my console output with fresh install from zero:
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$ pip install --use-wheel pandas
Collecting pandas
Using cached pandas-0.18.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Requirement already satisfied (use --upgrade to upgrade): pytz>=2011k in ./virtualenv/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in ./virtualenv/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.0 in ./virtualenv/lib/python2.7/site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): six in ./virtualenv/lib/python2.7/site-packages (from python-dateutil->pandas)
Installing collected packages: pandas
Successfully installed pandas-0.18.1
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
And then if I try to run a test in Django I'm getting errors that C files are not found:
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$ ./manage.py test mijnstroom.api.tests.v1.test_views.ApiViewDataTests.test_duplicate_data_points
Creating test database for alias 'default'...
/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1453: RuntimeWarning: DateTimeField EndexTariff.date received a naive datetime (2016-03-23 17:29:40.013550) while time zone support is active.
RuntimeWarning)
/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1453: RuntimeWarning: DateTimeField EndexTariff.added_on received a naive datetime (2016-03-23 17:32:55.578788) while time zone support is active.
RuntimeWarning)
created Group 'Installateurs'
created Group 'Investeerders'
created Group 'Energieleveranciers'
created Group 'Projecteigenaars'
/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/redactor/urls.py:19: RemovedInDjango110Warning: django.conf.urls.patterns() is deprecated and will be removed in Django 1.10. Update your urlpatterns to be a list of django.conf.urls.url() instances instead.
name='redactor_upload_file'),
/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/rest_framework_swagger/urls.py:10: RemovedInDjango110Warning: django.conf.urls.patterns() is deprecated and will be removed in Django 1.10. Update your urlpatterns to be a list of django.conf.urls.url() instances instead.
url(r'^api-docs/(?P<path>.*)/?$', SwaggerApiView.as_view(), name='django.swagger.api.view'),
/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/pandas/__init__.py:25: RuntimeWarning: datetime.datetime size changed, may indicate binary incompatibility. Expected 64, got 48
from pandas import hashtable, tslib, lib
/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/pandas/__init__.py:25: RuntimeWarning: datetime.date size changed, may indicate binary incompatibility. Expected 48, got 32
from pandas import hashtable, tslib, lib
/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/pandas/__init__.py:25: RuntimeWarning: datetime.date size changed, may indicate binary incompatibility. Expected 48, got 32
from pandas import hashtable, tslib, lib
/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/pandas/__init__.py:25: RuntimeWarning: datetime.datetime size changed, may indicate binary incompatibility. Expected 64, got 48
from pandas import hashtable, tslib, lib
E
======================================================================
ERROR: test_duplicate_data_points (mijnstroom.api.tests.v1.test_views.ApiViewDataTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/freezegun/api.py", line 428, in wrapper
result = func(*args, **kwargs)
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/mijnstroom/api/tests/v1/test_views.py", line 313, in test_duplicate_data_points
url = reverse('api:api_v1:v1_project_detail', kwargs={'pk': self.project.id})
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/django/core/urlresolvers.py", line 568, in reverse
app_list = resolver.app_dict[ns]
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/django/core/urlresolvers.py", line 360, in app_dict
self._populate()
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/django/core/urlresolvers.py", line 293, in _populate
for pattern in reversed(self.url_patterns):
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/django/utils/functional.py", line 33, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/django/core/urlresolvers.py", line 417, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/django/utils/functional.py", line 33, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/django/core/urlresolvers.py", line 410, in urlconf_module
return import_module(self.urlconf_name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/mijnstroom/urls.py", line 24, in <module>
url(r'^messaging/', include('mijnstroom.messaging.urls', namespace="messaging")),
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/django/conf/urls/__init__.py", line 52, in include
urlconf_module = import_module(urlconf_module)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/mijnstroom/messaging/urls.py", line 2, in <module>
from mijnstroom.messaging import views
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/mijnstroom/messaging/views.py", line 22, in <module>
from mijnstroom.utils.helpers import percentage_difference
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/mijnstroom/utils/helpers.py", line 15, in <module>
import knmi
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/src/knmi-py/knmi/__init__.py", line 1, in <module>
from knmi import get_day_data_raw, get_day_data_dataframe
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/src/knmi-py/knmi/knmi.py", line 3, in <module>
from .parsers import parse_day_data, parse_dataframe
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/src/knmi-py/knmi/parsers.py", line 1, in <module>
import pandas as pd
File "/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/pandas/__init__.py", line 31, in <module>
"extensions first.".format(module))
ImportError: C extension: hashtable not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.
----------------------------------------------------------------------
Ran 1 test in 1.039s
FAILED (errors=1)
Destroying test database for alias 'default'...
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
Only when I do the wrapping that I mentioned earlier, everything works fine and the test passes.
Comment From: jorisvandenbossche
Can you import pandas outside of the django application? Open a python console and import pandas? Do you get the same error?
Comment From: Eimis
Nope, all is good:
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$ which python
/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/bin/python
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$ python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import pandas
>>>
>>>
>>>
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
Comment From: jorisvandenbossche
So at least some version of pandas is correctly installed. Can you show the output of pd.__path__ to see that it is the correct pandas install that is imported?
Does it also work to import knmi from the console?
So it seems another version is imported in the django application.
Comment From: Eimis
Does it also work to import knmi from the console?
Yup, it works
Can you show the output of pd.path to see that it is the correct pandas install that is imported?
The path also seems to be correct..
(virtualenv)MacBook-Air-2:mijnstroom eimantas$
(virtualenv)MacBook-Air-2:mijnstroom eimantas$ python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>
>>> import pandas as pd
>>>
>>> print(pd.__path__)
['/Users/eimantas/Desktop/Projects/WeShareSolar/mijnstroom/virtualenv/lib/python2.7/site-packages/pandas']
>>>
>>>
Comment From: jorisvandenbossche
From the output above from testing the django app, I see you also get those warnings: RuntimeWarning: datetime.datetime size changed, may indicate binary incompatibility. Expected 64, got 48.
So apparantly, some other import has adapted datetime objects. To find the culprit, you can maybe in the console import packages that are used in the django app one by one, and see when importing pandas starts failing. Googling gave me this: https://github.com/spulec/freezegun/issues/13
Comment From: jorisvandenbossche
And apparantly you are using freezegun, as it appears in the test error output from above.
Comment From: Eimis
@jorisvandenbossche thank you! Apparently freezegun is causing the problems. I'll continue from there now.