Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[X] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Put this in `tmp.pyx`:
# In Cython code - any use of `_libs.khash` will trigger this
from pandas._libs.khash cimport kh_int64_t
Then run cython tmp.pyx. That will result in:
Error compiling Cython file:
------------------------------------------------------------
...
bint kh_exist_strbox(kh_strbox_t*, khiter_t) nogil
khuint_t kh_needed_n_buckets(khuint_t element_n) nogil
include "khash_for_primitive_helper.pxi"
^
------------------------------------------------------------
/home/rgommers/mambaforge/envs/pandas-dev/lib/python3.8/site-packages/pandas/_libs/khash.pxd:129:0: 'khash_for_primitive_helper.pxi' not found
Issue Description
I found this when following up on https://github.com/pandas-dev/pandas/pull/49115#issuecomment-1452662214:
Cython.Compiler.Errors.InternalError: Internal compiler error: 'khash_for_primitive_helper.pxi' not found
There are a couple of related issues that interact here:
- pandas is shipping lots of files in wheels that should not be there. In particular,
.pxdand.pyxfiles inpandas/_libs. - Use of absolute
cimport's which should probably be relative - Use of
include <name>.pxi"in.pxdfiles. This should be replaced by shared declarations in a common.pxdfile (see the warning in http://docs.cython.org/en/latest/src/userguide/language_basics.html#the-include-statement-and-include-files)
For (1), if you download any pandas 1.5.3 wheel, you'll see in pandas/_libs:
khash.pxd
khash_for_primitive_helper.pxi.in
And, notably, khash.pxd contains include "khash_for_primitive_helper.pxi" - and that file is not present (only the pxi.in template is). So basically a broken private .pxd here. Which is then picked up during the build in gh-49115 because of absolute from pandas._libs.khash cimport ... statements inside pandas itself.
That particular issue probably shows up in the Meson build but not during the setup.py-based build because in the latter the .pxi file is generated in-place rather than in the build dir. However, as my reproducer above shows, this is a bit of a house of cards, because the absolute from pandas._libs imports are actually broken.
Expected Behavior
Expected is that the .pxds aren't shipped, so anyone trying to access private .pxd files will get a clear exception. This will be automatically fixed when the Meson build is merged. However, that still leaves potential issues in any environments that already have pandas installed.
My suggestion is to:
- Use relative cimports for accessing anything within pandas (needs testing, because Cython's cimport mechanism is very fragile all around).
- Get rid of the .pxi.in and replace it with the recommended .pxd method.