The upcoming 1.5.0 pandas update proposes a minimum version of s3fs 2021.5.0 (see doc/source/whatsnew/v1.5.0.rst).
This will not be acceptable to any project that interacts with s3 professionally, due to s3fs extremely tight pinning to outdated botocore.
Specifically, s3fs>=0.5.0 requires aiobotocore which is very tightly binned to botocore, and that pinning is mutually incompatible with moto (some unresolved issues explaining the aiobotocore botocore problem). Therefore you cannot use s3fs and test s3 access in the same project without running into incompatible versions.
Attempting pip-compile on a project with
s3fs[boto3]>=2021.05.0 # what pandas is about to enforce
moto[s3]>=1.0 # incredibly lenient moto request
returns
botocore<1.24.22,>=1.24.21 (from aiobotocore==2.3.4->s3fs[boto3]==2022.5.0->my-project (setup.py))
botocore>=1.12.201 (from moto==3.1.16->my-project (setup.py))
botocore<1.28.0,>=1.27.30 (from boto3==1.24.30->moto==3.1.16->my-project (setup.py))
even with latest pip-tools update to handling dependencies, moto-s3fs is still unresolvable.
I would strongly prefer pandas to not (even in optional dependencies) spread this problem further.
PS. It isn't actively documented, but right now the pandas testing is implicitly pinned to two s3fs releases from March 2020 because pandas needs moto for local tests and moto cannot coexist with s3fs>0.5.0 and pandas requires s3fs>0.4.0
Describe the solution you'd like
Do not update s3fs min requirement until packaging problems from upstream aiobotocore requirements are resolved.
Comment From: MarcoGorelli
Thanks for the report - could you find the PR which set this minimum s3fs version please? Just to understand the context of the decision
Comment From: MarcoGorelli
latest CI shows both the latest moto and s3fs 2021.11.0 , so I think this has been solved now?
closing for now then, please please do ping / reopen if I've misunderstood