This ImportError has been appearing in a few builds lately e.g.
https://github.com/pandas-dev/pandas/runs/6819149053?check_suite_focus=true https://github.com/pandas-dev/pandas/runs/6819297885?check_suite_focus=true
2022-06-10T05:56:44.5012047Z Traceback (most recent call last):
2022-06-10T05:56:44.5012729Z File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 1435, in <module>
2022-06-10T05:56:44.5013162Z main()
2022-06-10T05:56:44.5013705Z File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 1428, in main
2022-06-10T05:56:44.5014100Z commands[mode](args)
2022-06-10T05:56:44.5015119Z File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 1103, in main_discover
2022-06-10T05:56:44.5015666Z list_benchmarks(benchmark_dir, fp)
2022-06-10T05:56:44.5016169Z File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 1088, in list_benchmarks
2022-06-10T05:56:44.5016554Z for benchmark in disc_benchmarks(root):
2022-06-10T05:56:44.5017076Z File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 985, in disc_benchmarks
2022-06-10T05:56:44.5018181Z for module in disc_modules(root_name, ignore_import_errors=ignore_import_errors):
2022-06-10T05:56:44.5018904Z File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 967, in disc_modules
2022-06-10T05:56:44.5019725Z for item in disc_modules(name, ignore_import_errors=ignore_import_errors):
2022-06-10T05:56:44.5020409Z File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 950, in disc_modules
2022-06-10T05:56:44.5020821Z module = import_module(module_name)
2022-06-10T05:56:44.5021709Z File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/importlib/__init__.py", line 127, in import_module
2022-06-10T05:56:44.5022165Z return _bootstrap._gcd_import(name[level:], package, level)
2022-06-10T05:56:44.5022557Z File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-10T05:56:44.5022982Z File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-10T05:56:44.5023378Z File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
2022-06-10T05:56:44.5023914Z File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
2022-06-10T05:56:44.5024302Z File "<frozen importlib._bootstrap_external>", line 843, in exec_module
2022-06-10T05:56:44.5024698Z File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2022-06-10T05:56:44.5025141Z File "/home/runner/work/pandas/pandas/asv_bench/benchmarks/algorithms.py", line 5, in <module>
2022-06-10T05:56:44.5025505Z import pandas as pd
2022-06-10T05:56:44.5026003Z File "/home/runner/work/pandas/pandas/pandas/__init__.py", line 22, in <module>
2022-06-10T05:56:44.5026696Z from pandas.compat import is_numpy_dev as _is_numpy_dev
2022-06-10T05:56:44.5027294Z File "/home/runner/work/pandas/pandas/pandas/compat/__init__.py", line 15, in <module>
2022-06-10T05:56:44.5027968Z from pandas.compat.numpy import (
2022-06-10T05:56:44.5028331Z File "/home/runner/work/pandas/pandas/pandas/compat/numpy/__init__.py", line 4, in <module>
2022-06-10T05:56:44.5028851Z from pandas.util.version import Version
2022-06-10T05:56:44.5029223Z File "/home/runner/work/pandas/pandas/pandas/util/__init__.py", line 1, in <module>
2022-06-10T05:56:44.5030080Z from pandas.util._decorators import ( # noqa:F401
2022-06-10T05:56:44.5030653Z File "/home/runner/work/pandas/pandas/pandas/util/_decorators.py", line 14, in <module>
2022-06-10T05:56:44.5050795Z from pandas._libs.properties import cache_readonly # noqa:F401
2022-06-10T05:56:44.5051218Z File "/home/runner/work/pandas/pandas/pandas/_libs/__init__.py", line 13, in <module>
2022-06-10T05:56:44.5051586Z from pandas._libs.interval import Interval
2022-06-10T05:56:44.5051939Z File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
2022-06-10T05:56:44.5052312Z File "pandas/_libs/hashtable.pyx", line 1, in init pandas._libs.hashtable
2022-06-10T05:56:44.5052693Z File "pandas/_libs/missing.pyx", line 1, in init pandas._libs.missing
2022-06-10T05:56:44.5053108Z File "/home/runner/work/pandas/pandas/pandas/_libs/tslibs/__init__.py", line 35, in <module>
2022-06-10T05:56:44.5053514Z from pandas._libs.tslibs.conversion import localize_pydatetime
2022-06-10T05:56:44.5054108Z File "pandas/_libs/tslibs/conversion.pyx", line 69, in init pandas._libs.tslibs.conversion
2022-06-10T05:56:44.5054557Z File "pandas/_libs/tslibs/parsing.pyx", line 1, in init pandas._libs.tslibs.parsing
2022-06-10T05:56:44.5055164Z File "pandas/_libs/tslibs/offsets.pyx", line 1, in init pandas._libs.tslibs.offsets
2022-06-10T05:56:44.5056426Z ImportError: /home/runner/work/pandas/pandas/pandas/_libs/tslibs/timestamps.cpython-38-x86_64-linux-gnu.so: undefined symbol: pandas_datetime_to_datetimestruct
The flakiness might indicate that the parallel build process is not building everything correctly?
cc @jbrockmendel @lithomas1 @WillAyd
Comment From: rhshadrach
Tried some debugging here: https://github.com/pandas-dev/pandas/pull/47310#issuecomment-1152968616
When the build fails, symbols in timestamps.so are unresolved but exist in np_datetime.o (and np_datetime.so). Is it possible the build of timestamps.so is occurring before np_datetime.o exists (or is fully built)?
Comment From: MarcoGorelli
locally I run
while true; do
python setup.py build_ext -j 16 --force && python -c 'import pandas'
if [ $? -eq 0 ]; then
break
fi
done
, it usually takes a few attempts to build correctly but it's faster than -j 1
Comment From: rhshadrach
Minor tweak to avoid compiling when you don't need to:
python setup.py build_ext -j 16 && python -c 'import pandas';
while [ $? -ne 0 ]; do
python setup.py build_ext -j 16 --force && python -c 'import pandas';
done
Comment From: natmokval
Following the guide
https://pandas.pydata.org/docs/dev/development/contributing_environment.html
I run python setup.py build_ext -j 4 and the problem of parallel builds occurs.
I had to run python setup.py build_ext -j 1 --force. Should the documentation say something about this problem until it is fixed?
Comment From: MarcoGorelli
yeah, probably! In the meantime I've pinned this issue, should make it easier to find
Comment From: lithomas1
Note that we are changing build systems to meson pretty soon(in #49115), so hopefully we won't have to deal with this for much longer.
Comment From: lithomas1
Just keeping track of similar import errors for the C extensions.
ImportError: cannot import name 'abbrev_to_npy_unit' from 'pandas._libs.tslibs.dtypes' (/Users/Darbyshire/pandas/pandas/_libs/tslibs/dtypes.cpython-39-darwin.so)