Pandas BUG: pandas halted in a busy loop when assigning new column with non-unique index

Pandas version checks

[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of pandas.
[ ] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

indices = (
    ["label 1" for _ in range(10000)]
    + ["label 2" for _ in range(10000)]
    + ["label 3" for _ in range(10000)]
)
data = {"values": np.random.randn(30000)}

print("create dataframe")
df = pd.DataFrame(data, index=indices)

"""""
The original DataFrame looks like this

         values
label 1  0.808733
label 1  1.112494
label 1  0.995864
label 1  0.884037
label 1  0.260556
            ...
label 3  -0.411496
label 3  -1.603830
label 3  0.058993
label 3  1.374946
label 3  0.172165
"""

print("create `new_value` of disordered index (which have duplicate) ...")
new_value = df["values"].sort_values()

"""""
The index of the Series `new_value` are disordered and look like this

label 1   -4.210096
label 2   -4.154975
label 2   -3.996140
label 3   -3.767073
label 1   -3.754259
             ...   
label 3    3.533630
label 1    3.593864
label 2    3.614379
label 2    3.728890
label 3    3.768827
Name: values, Length: 30000, dtype: float64
"""

# The program is halted in a busy loop while executing the following statement
print("assign `new_value` as column...")
df["new"] = new_value

Issue Description

The program is halted in a busy loop when assigning new column

Expected Behavior

I figured out the correct way to achieve my goal is to df.reset_index() to use the default ordinal index first then df.set_index('label') after the adding of new column. I suspect that this usage (with duplcate values in index) may not be a well-defined use case. But I could not found any documentation regarding this and pandas did not report error on the index being invald.

Installed Versions


INSTALLED VERSIONS
------------------
commit           : 2e218d10984e9919f0296931d92ea851c6a6faf5
python           : 3.10.9.final.0
python-bits      : 64
OS               : Linux
OS-release       : 6.1.6-arch1-1
Version          : #1 SMP PREEMPT_DYNAMIC Sat, 14 Jan 2023 13:09:35 +0000
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : en_US.UTF-8
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.5.3
numpy            : 1.24.2
pytz             : 2022.7.1
dateutil         : 2.8.2
setuptools       : 63.2.0
pip              : 22.2
Cython           : None
pytest           : 5.4.3
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : 4.11.1
bottleneck       : None
brotli           : None
fastparquet      : None
fsspec           : None
gcsfs            : None
matplotlib       : 3.6.3
numba            : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : 1.10.0
snappy           : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
zstandard        : None
tzdata           : None

Comment From: phofl

Hi, thanks for your report. Eventually, this would raise. Setting with a non-unique index is not allowed here. That said, we should do this check before calculating the indexer.