API/COMPAT: support axis=None for logical reduction (reduce over all axes) by TomAugspurger · Pull Request #21486 · pandas-dev/pandas

TomAugspurger · 2018-06-14T19:56:00Z

closes CI: NumPy logical reductions (any, all) fail on DataFrame with NumPy master #19976
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

This is the minimal fix, just to get np.all / np.any working again.

Some followup items:

Expand to all aggregations, not just logical ones
Do logical reductions blockwiwse: DataFrame.any(axis={0, 1}) returns inconsistent values for non-zero timedelta #17667. Currently, we do DataFrame.values, which isn't necessary for logical reductions.

Accepts axis=None as reduce all dims

jorisvandenbossche · 2018-06-15T06:56:52Z

+
+
+This also provides compatibility with NumPy 1.15, which now dispatches to ``DataFrame.all``.
+With NumPy 1.15 and pandas 0.23.1 or earlier, :func:`numpy.all` will not reduce over every axis:


not -> no longer ?

jorisvandenbossche · 2018-06-15T06:57:34Z

+   B    False
+   dtype: bool
+
+With pandas 0.23.2, that will correctly return False.


maybe add ", as it did before with numpy < 1.15" ?

jorisvandenbossche · 2018-06-15T07:05:28Z

 ----------
-axis : int, default 0
-    Select the axis which can be 0 for indices and 1 for columns.
+axis : {None, 0 or 'index', 1 or 'columns'}, default None


I don't think default None is correct?
The default for our method is still 0? It's only when doing np.all(..) that we respect the axis=None ?

jorisvandenbossche · 2018-06-15T07:07:02Z

        """
        delegate = self._values
+        if axis is None:
+            axis = self._stat_axis_number


Why is this still needed?

Can you have a look at this one?

Sorry, missed this earlier. We still call self._get_axis_number(axis) to validate that the user passed axis is correct. I'll modify things to be clearer.

jorisvandenbossche · 2018-06-15T07:08:51Z

-        if skipna is None:
-            skipna = True
-        if axis is None:
-            axis = self._stat_axis_number


Shouldn't there be an override of this in case of Panel? (which is the only case where the stat_axis differs from 0)

jreback · 2018-06-15T16:39:36Z

                values = self.values
                result = f(values)
+
+                if (filter_type == 'bool' and values.dtype.kind == 'O' and


use is_object_dtype

jreback · 2018-06-15T16:41:20Z

        self._check_bool_op('any', np.any, has_skipna=True, has_bool_only=True)
        self._check_bool_op('all', np.all, has_skipna=True, has_bool_only=True)

-        df = DataFrame(randn(10, 4)) > 0


I would make a new test function here

jorisvandenbossche · 2018-06-21T09:57:53Z

Tests are still failing: np.any/np.all don't seem to work on timedelta data (at least for certain versions of numpy, locally for me it is failing on 1.13)

codecov · 2018-06-21T21:37:29Z

Codecov Report

Merging #21486 into master will decrease coverage by 0.02%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master   #21486      +/-   ##
==========================================
- Coverage   91.92%    91.9%   -0.03%     
==========================================
  Files         153      153              
  Lines       49563    49562       -1     
==========================================
- Hits        45559    45548      -11     
- Misses       4004     4014      +10

Flag	Coverage Δ
#multiple	`90.3% <85.71%> (-0.03%)`	⬇️
#single	`41.77% <53.57%> (-0.04%)`	⬇️

Impacted Files	Coverage Δ
pandas/util/_test_decorators.py	`92.68% <100%> (+0.18%)`	⬆️
pandas/core/generic.py	`96.21% <100%> (+0.08%)`	⬆️
pandas/core/series.py	`94.19% <100%> (ø)`	⬆️
pandas/core/panel.py	`97.43% <62.5%> (-0.44%)`	⬇️
pandas/core/frame.py	`97.19% <90.9%> (-0.05%)`	⬇️
pandas/util/testing.py	`85.27% <0%> (-0.7%)`	⬇️
pandas/core/indexing.py	`93.37% <0%> (-0.05%)`	⬇️
pandas/tseries/offsets.py	`97.16% <0%> (-0.04%)`	⬇️
pandas/core/dtypes/common.py	`94.63% <0%> (-0.02%)`	⬇️
pandas/core/indexes/multi.py	`94.97% <0%> (-0.01%)`	⬇️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b36b451...ae759bd. Read the comment docs.

jreback · 2018-06-22T23:28:44Z

I think you need to change

    @pytest.mark.xfail(
        not _np_version_under1p15,
        reason="failing under numpy-dev gh-19976")
    @pytest.mark.parametrize("axis", [0, 1, None])
    def test_clip_against_frame(self, axis):
        df = DataFrame(np.random.randn(1000, 2))

in pandas/tests/frame/test_analytics.py which was the original failing test, to use your new decorator?

otherwise lgtm.

TomAugspurger · 2018-06-23T03:05:15Z

Good catch. I've removed the xfail entirely, since I think it will pass on all versions now.

…axes) (pandas-dev#21486) * Compat with NumPy 1.15 logical func * Accepts axis=None as reduce all dims (cherry picked from commit f7ed7f8)

…axes) (#21486) * Compat with NumPy 1.15 logical func * Accepts axis=None as reduce all dims (cherry picked from commit f7ed7f8)

…axes) (pandas-dev#21486) * Compat with NumPy 1.15 logical func * Accepts axis=None as reduce all dims

TomAugspurger added 2 commits June 14, 2018 14:44

Compat with NumPy 1.15 logical func

a6114b4

Accepts axis=None as reduce all dims

whatsnew

4bb6e53

TomAugspurger added Numeric Operations Arithmetic, Comparison, and Logical operations Compat pandas objects compatability with Numpy or Python functions labels Jun 14, 2018

TomAugspurger added this to the 0.23.2 milestone Jun 14, 2018

TomAugspurger mentioned this pull request Jun 14, 2018

CI: NumPy logical reductions (any, all) fail on DataFrame with NumPy master #19976

Closed

remove old test

9636d54

jorisvandenbossche reviewed Jun 15, 2018

View reviewed changes

jreback requested changes Jun 15, 2018

View reviewed changes

TomAugspurger added 3 commits June 20, 2018 08:01

updated docs

18c1b11

Merge remote-tracking branch 'upstream/master' into logical-agg

3c7f4e5

Handle panel

1c32b2b

jorisvandenbossche changed the title ~~Logical agg~~ API/COMPAT: support axis=None for logical reduction (reduce over all axes) Jun 21, 2018

Skip for np 114

1f94469

jorisvandenbossche approved these changes Jun 22, 2018

View reviewed changes

TomAugspurger mentioned this pull request Jun 22, 2018

Support axis=None in all reductions #21597

Closed

TomAugspurger added 2 commits June 22, 2018 22:01

reuse numpy compat

50db719

remove xfail

9fd9740

Linting

ae759bd

jorisvandenbossche merged commit f7ed7f8 into pandas-dev:master Jun 26, 2018

jorisvandenbossche added Needs Backport and removed Needs Backport labels Jun 26, 2018

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

API/COMPAT: support axis=None for logical reduction (reduce over all …

78e1a67

…axes) (pandas-dev#21486) * Compat with NumPy 1.15 logical func * Accepts axis=None as reduce all dims



		This also provides compatibility with NumPy 1.15, which now dispatches to ``DataFrame.all``.
		With NumPy 1.15 and pandas 0.23.1 or earlier, :func:`numpy.all` will not reduce over every axis:

Uh oh!

Conversation

TomAugspurger commented Jun 14, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche commented Jun 21, 2018

Uh oh!

codecov Bot commented Jun 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jreback commented Jun 22, 2018

Uh oh!

TomAugspurger commented Jun 23, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Jun 21, 2018 •

edited

Loading