Feature Type

  • [X] Adding new functionality to pandas

  • [ ] Changing existing functionality in pandas

  • [ ] Removing existing functionality in pandas

Problem Description

Pandas 2.0.0 seems to have seen a great increase in the use of lib.NoDefault type annotation, which allows using None as a regular value instead of a sentinel.

That is great but also means these APIs are now a nightmare to wrap, as the wrapper most likely will need to pass something even if the end user did not specify anything (a wrapper's default value for that parameter). The only way to do this atm is to craft a dict and use **kwargs, but that is quite painful to do.

For example this used to work on pandas 1.5.3 but now fails on 2.0.0 as index=None is not special anymore:

def mypivot(df, index=None):
    df.pivot(columns='cols', values='vals', index=index)

To restore the ability to wrap pandas APIs, we need that, but alas NoDefault is a private API:

def mypivot(df, index=lib.NoDefault):
    df.pivot(columns='cols', values='vals', index=index)

So the only way to achieve it "cleanly" in 2.0.0 is:

def mypivot(df, index=None):
    kwargs = dict() if index is None else dict(index=index)
    df.pivot(columns='cols', values='vals', **kwargs)

That will get old pretty fast.

Feature Description

Expose publicly pandas._libs.lib.NoDefault

Alternative Solutions

Craft dynamic keyword args to avoid passing arguments in the first place when they are not needed, but that is painful and does not allow treating None as a normal value, unless the wrapping API comes up with its own NoDefault.

Additional Context

Non-notebook users. This sort of problem does not arise when the end users uses the API directly/interactively, but building a library on top of pandas can become quickly challenging because of this sort of problem. In the same vein, I very much welcomed pandas 2.0.0 decision to make groupby() keys be 1-tuple when passed a column list with a single item.

Comment From: douglas-raillard-arm

Ah and while at it, if that feature ends up being implemented it might be a good idea to make bool(pandas._libs.lib.NoDefault) == False to match None behavior. This allows doing things like x = x or "bar"

Comment From: phofl

This is already public, 

from pandas.api.extensions import no_default