It'd be great to start linting the doctests (validate the docstring examples for pep8 issues).
To start with core/generic.py
, core/series.py
and core/frame.py
, I created the next issues to address the current errors: #22892, #22893, #22894, #22895, #22896, #22897 and #22898.
After those are fixed, the only error that will remain in these files is F821 undefined name 'pd'
. I checked for simple ways to avoid it, but I don't think we can prepend code for flake8 (like what we do in sphinx). So, the solutions I see are:
- Add
import pandas as pd
(or even betterimport pandas
) to every example - Do not lint the doctests
- Ignore
F821
from the doctest linting (this will also ignore errors likedf + 1
where df is not defined, as it's also anF821
error). - Implement a way to
grep -v
those exact errors, and change the exit code offlake8
based on that, so the linting passes when the output is clean after filtering these errors
I know this has been discussed before, and I'm not sure if anyone else besides me is in favor of option 1. But I think it's worth considering it again in the context of linting doctests. Option 1 has the advantage that the examples are self-contained (they can be copy-pasted, or in the future we can even add a button to open them in binder, or something similar).
Comment From: WillAyd
I think option 1 would just be a lot of churn and add a lot of repetition to the code samples, so I'm -1 with that. Option 4 seems like the most logical on initial glance - do you see a downside to that outside of the added grep
call?
Comment From: jreback
we always ‘import pandas as pd’
option 1 seems pretty verbose as well
Comment From: TomAugspurger
As long as we're running the examples, option 3 should be fine, right? The example df + 1
will fall the doctest.
So my vote is for option 3.
Comment From: jbrockmendel
@datapythonista how do i check if this is fixed?
Comment From: datapythonista
This is not yet fixed. What we'd like is to have this passing:
$ flake8 --doctests pandas/
This would prevent errors in the examples in our docstrings. But if you execute that, you'll see we've got plenty of:
./plotting/_core.py:595:18: F821 undefined name 'pd'
./plotting/_core.py:1286:24: F821 undefined name 'np'
I guess we could start by adding F821
to the list of flake8 errors we ignore (option 3 in the description), but there are some other errors we need to have a look at before we have flake8 validation in doctests passing:
pandas/core/generic.py:3343:33: F721 syntax error in doctest
pandas/errors/__init__.py:458:17: F721 syntax error in doctest
pandas/errors/__init__.py:461:17: F721 syntax error in doctest
pandas/errors/__init__.py:568:9: F401 'pandas.io.stata.StataReader' imported but unused
pandas/errors/__init__.py:571:30: F721 syntax error in doctest
pandas/io/formats/style.py:966:19: F721 syntax error in doctest
pandas/io/formats/style.py:987:19: F721 syntax error in doctest
pandas/io/formats/style.py:1874:23: F721 syntax error in doctest
pandas/io/formats/style.py:1882:30: F721 syntax error in doctest
pandas/io/formats/style.py:1884:23: F721 syntax error in doctest
pandas/io/formats/style.py:2787:23: F721 syntax error in doctest
pandas/io/formats/style.py:2793:23: F721 syntax error in doctest
pandas/io/formats/style.py:2799:23: F721 syntax error in doctest
pandas/io/formats/style.py:2805:23: F721 syntax error in doctest
pandas/io/formats/style.py:2811:23: F721 syntax error in doctest
pandas/io/formats/style.py:2820:23: F721 syntax error in doctest
pandas/io/formats/style.py:3508:21: F721 syntax error in doctest
pandas/io/formats/style_render.py:1081:17: F721 syntax error in doctest
Comment From: jbrockmendel
Looks like a lot of the syntax errors are from multi-line statements missing either parens or backslashes
Comment From: datapythonista
I created a "good first issue" issue for the errors other than the import. When those are fixed, we can see if we really want to skip F821, or maybe we can add a validation to the CI to make sure only pd
and np
can be missing. Something like the next command, which shows that we'd ignore other variable names if we simply ignore F821:
$ flake8 --doctest --ignore="" --select=F821 pandas/ | grep -v "F821 undefined name 'pd'" | grep -v "F821 undefined name 'np'"
pandas/core/apply.py:1339:9: F821 undefined name '_relabel_result'
pandas/core/apply.py:1339:33: F821 undefined name 'func'
pandas/core/arrays/datetimelike.py:262:13: F821 undefined name 'self'
pandas/core/dtypes/base.py:266:27: F821 undefined name 're'
pandas/core/generic.py:2247:17: F821 undefined name 'df'
pandas/core/generic.py:5890:13: F821 undefined name 'func'
pandas/core/generic.py:5890:18: F821 undefined name 'g'
pandas/core/generic.py:5890:20: F821 undefined name 'h'
pandas/core/generic.py:5890:22: F821 undefined name 'df'
pandas/core/generic.py:5890:32: F821 undefined name 'a'
pandas/core/generic.py:5890:41: F821 undefined name 'b'
pandas/core/generic.py:5890:49: F821 undefined name 'c'
pandas/core/generic.py:5894:14: F821 undefined name 'df'
pandas/core/generic.py:5894:22: F821 undefined name 'h'
pandas/core/generic.py:5895:22: F821 undefined name 'g'
pandas/core/generic.py:5895:30: F821 undefined name 'a'
pandas/core/generic.py:5896:22: F821 undefined name 'func'
pandas/core/generic.py:5896:33: F821 undefined name 'b'
pandas/core/generic.py:5896:41: F821 undefined name 'c'
pandas/core/generic.py:5903:14: F821 undefined name 'df'
pandas/core/generic.py:5903:22: F821 undefined name 'h'
pandas/core/generic.py:5904:22: F821 undefined name 'g'
pandas/core/generic.py:5904:30: F821 undefined name 'a'
pandas/core/generic.py:5905:23: F821 undefined name 'func'
pandas/core/generic.py:5905:43: F821 undefined name 'a'
pandas/core/generic.py:5905:51: F821 undefined name 'c'
pandas/io/formats/style.py:1873:25: F821 undefined name 'ret'
pandas/io/formats/style_render.py:1279:49: F821 undefined name 'upper'