Pandas PERF: unnecessary casting in merge

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this issue exists on the latest version of pandas.

• [X] I have confirmed this issue exists on the master branch of pandas.

Reproducible Example

In reshape.merge we do some some casting motivated by what dtypes HashTable supports, see the comment '# upcast 'by' parameter because HashTable is limited'. That comment is no longer accurate, and we can avoid some of this casting. Something like core.algorithms._ensure_data may be reusable.

Installed Versions

Replace this line with the output of pd.show_versions()

Prior Performance

No response

Comment From: jbrockmendel

@phofl i think one of your recent PRs might have addressed this?

Comment From: phofl

I am in the process of finishing the final pr, will link to this issue then