Pandas version checks
-
[x] I have checked that this issue has not already been reported.
-
[x] I have confirmed this bug exists on the latest version of pandas.
-
[x] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
- Create a series with
Indexand a MultiIndex to use for reindexing later
>>> series = pd.Series(
... [26.7300, 24.2550],
... index=pd.Index([81, 82], name='a')
... )
>>> series
a
81 26.730
82 24.255
dtype: float64
>>> series.index
Index([81, 82], dtype='int64', name='a')
>>> other_index = pd.MultiIndex(
... levels=[
... pd.Index([81, 82], name='a'),
... pd.Index([np.nan], name='b'),
... pd.Index([
... '2018-06-01', '2018-07-01'
... ], name='c')
... ],
... codes=[
... [0, 0, 1, 1],
... [0, 0, 0, 0],
... [0, 1, 0, 1]
... ],
... names=['a', 'b', 'c']
... )
>>> other_index
MultiIndex([(81, nan, '2018-06-01'),
(81, nan, '2018-07-01'),
(82, nan, '2018-06-01'),
(82, nan, '2018-07-01')],
names=['a', 'b', 'c'])
reindextoMultiIndex(other_index) which expandsseries.indexby two more levels.- unfortunately the
reindexsets all values of the original series to NaN which can be fixed by turningseries.indexinto a 1-levelMultiIndexfirst
>>> series.reindex(other_index) # this removes all values of the series
a b c
81 NaN 2018-06-01 NaN
2018-07-01 NaN
82 NaN 2018-06-01 NaN
2018-07-01 NaN
dtype: float64
- apply
to_mi(...)to turn theseries.indexinto a 1-levelMultiIndex - rerun
reindexon the newserieswithMultiIndexand the values are maintained/filled as expected
>>> def to_mi(series):
... if isinstance(series.index, pd.MultiIndex):
... series_mi = series.index
... else:
... level_names = [series.index.name]
... level_values = [series.index]
... series_mi = pd.MultiIndex.from_arrays(level_values, names=level_names)
... series_with_mi = pd.Series(series.values, index=series_mi, name=series.name)
... return series_with_mi
...
>>> series_mi = to_mi(series)
>>> series_mi
a
81 26.730
82 24.255
dtype: float64
>>> series_mi.index
MultiIndex([(81,),
(82,)],
names=['a'])
>>> series_mi.reindex(other_index)
a b c
81 NaN 2018-06-01 26.730
2018-07-01 26.730
82 NaN 2018-06-01 24.255
2018-07-01 24.255
dtype: float64
Issue Description
In the above case, series.reindex(multi_index) will turn the series values to NaN when the series has a single Index. However when the series index is converted to a 1-level MultiIndex prior to the reindex, the values are maintained and filled as expected.
In my opinion it shouldn't matter if a 1-level MultiIndex or an Index is used for a reindex - the outcomes should be the same.
As a further discussion point (here or elsewhere), this issue (and others) also begs the question why a distinction between Index and MultiIndex is necessary (I suspect there are historic reasons). I would imagine that many issues (and code) would go away if MultiIndex was used exclusively (even for 1-dimensional indices).
Expected Behavior
The missing levels in series_mi (compared to other_index) are added and the values of the partial index from the original series are used to fill the places of the added indices.
>>> series_mi.reindex(other_index)
a b c
81 NaN 2018-06-01 26.730 # from index <81> of `series` (`series_mi`)
2018-07-01 26.730 # from index <81> of `series` (`series_mi`)
82 NaN 2018-06-01 24.255 # from index <82> of `series` (`series_mi`)
2018-07-01 24.255 # from index <82> of `series` (`series_mi`)
dtype: float64
Installed Versions
Comment From: rhshadrach
Thanks for the report! Is it possible to shorten your example? Making it as short as possible to demonstrate the behavior would certainly be appreciated.
Comment From: ssche
Thanks for the feedback. I managed to shorten the example and provided additional comments.