https://github.com/pandas-dev/pandas/pull/25063 made the 0.24.x sort=True behavior be enabled by sort=None. This allows us to use sort=True to mean "always sort".
So rather than raising
In [1]: import pandas as pd
In [2]: idx = pd.Index([1, 0])
In [3]: idx.union(idx, sort=True)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-3-92d6bafc02d6> in <module>
----> 1 idx.union(idx, sort=True)
~/sandbox/pandas/pandas/core/indexes/base.py in union(self, other, sort)
2319 Index(['a', 'b', 'c', 'd', 1, 2, 3, 4], dtype='object')
2320 """
-> 2321 self._validate_sort_keyword(sort)
2322 self._assert_can_do_setop(other)
2323
~/sandbox/pandas/pandas/core/indexes/base.py in _validate_sort_keyword(self, sort)
2263 if sort not in [None, False]:
2264 raise ValueError(
-> 2265 "The 'sort' keyword only takes the values of "
2266 f"None or False; {sort} was passed."
2267 )
ValueError: The 'sort' keyword only takes the values of None or False; True was passed.
We would instead return
Int64Index([0, 1], dtype='int64')
This is a (hopefully) exhaustive list of special cases not sorted when sort=None.
union:- equal
- length-0
- incomparable
- intersection
- equal
- incomparable
- difference
- incomparable
- symmetric_difference
- incomparable
Comment From: TomAugspurger
@jorisvandenbossche OK with moving off the 1.0 milestone?
Comment From: TomAugspurger
Pushing off 1.0.
Comment From: jbrockmendel
@TomAugspurger @jorisvandenbossche @jreback this issue was on the agenda for today's call but no one present knew what the reasoning was for having sort=None instead of sort=True in the first place. There are a bunch of comments in the tests to the effect of "# TODO: decide what sort=True means"
IIUC sort=True will differ from sort=None only in the handful of cases that currently go through a fastpath and avoid sorting like the [1, 0] example in the OP?
Comment From: mroeschke
This is a psudo-blocker for https://github.com/pandas-dev/pandas/issues/48553 as there are some test warnings involving join that hardcode sort, and passing through sort with how="outer"/"inner" hits Index.union/intersection which is not implemented when sort=True
pandas/tests/indexes/multi/test_missing.py::test_nan_stays_float
/home/runner/work/pandas/pandas/pandas/tests/indexes/multi/test_missing.py:92: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
idxm = idx0.join(idx1, how="outer")
pandas/tests/indexes/multi/test_missing.py::test_nan_stays_float
/home/runner/work/pandas/pandas/pandas/tests/indexes/multi/test_missing.py:99: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
dfm = df0 - df1
Comment From: jorisvandenbossche
That's maybe rather a bug in the MultiIndex non-sort case? Because the warning says to pass sort=False to suppress this warning, but doing that doesn't actually solve it:
In [5]: idx0 = MultiIndex(levels=[["A", "B"], []], codes=[[1, 0], [-1, -1]], names=[0, 1])
...: idx1 = MultiIndex(levels=[["C"], ["D"]], codes=[[0], [0]], names=[0, 1])
...: idxm = idx0.join(idx1, how="outer")
<ipython-input-5-1d74c2084c24>:3: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
idxm = idx0.join(idx1, how="outer")
In [6]: idxm = idx0.join(idx1, how="outer", sort=False)
<ipython-input-6-f0c3e74a3a0c>:3: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
idxm = idx0.join(idx1, how="outer", sort=False)
Comment From: jorisvandenbossche
no one present knew what the reasoning was for having sort=None instead of sort=True in the first place
The top post has a list of cases where the current default behaviour (sort=None) does not actually sort (and thus could potentially differ from a sort=True).
As a small example:
>>> idx1 = pd.Index(['c', 'b', 'a'])
>>> idx2 = pd.Index(['a', 'c', 'b'])
>>> idx1.union(idx2, sort=None)
Index(['a', 'b', 'c'], dtype='object') # <-- sorted
>>> idx1.union(idx2, sort=False)
Index(['c', 'b', 'a'], dtype='object')
>>> idx1.union(idx1.copy(), sort=None)
Index(['c', 'b', 'a'], dtype='object') # <-- not sorted
>>> idx1.union(idx1.copy(), sort=False)
Index(['c', 'b', 'a'], dtype='object')
So depending on the values, the default sort=None does sort or does not sort. That's already a good reason to not call this behaviour sort=True.
So we originally used sort=None as the default when adding this sort keyword to the Index setops methods, to allow adding sort=True in the future (but this part never happened in practice, I think).
In addition, at the time we also were thinking that we should actually move to sort=False as the default (Index.intersection already has this default).