Add `{*Frame,Series,Expr}.drop_nans`

[dangotbanned]

Spotted that I never followed up on (#3031 (comment)), (#3031 (comment)).

I think we can make this easier by introducing drop_nans, aligning our support of {drop,fill}_{nans,nulls}

narwhals/docs/concepts/null_handling.md

Lines 12 to 26 in 8d93fb1

	## What's the difference?
	Most data tools except pandas make a clear distinction between:
	- Null values, representing missing data.
	- NaN values, resulting from "illegal" mathematical operations like `0/0`.
	In Narwhals, this is reflected in separate methods for Null/NaN values:
	| Operation | Null | NaN |
	| --------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
	| is | [`Expr.is_null`][narwhals.Expr.is_null] | [`Expr.is_nan`][narwhals.Expr.is_nan] |
	| fill | [`Expr.fill_null`][narwhals.Expr.fill_null] | [`Expr.fill_nan`][narwhals.Expr.fill_nan] |
	| drop | [`Expr.drop_nulls`][narwhals.Expr.drop_nulls] | *Not yet implemented (See [discussion](https://github.com/narwhals-dev/narwhals/issues/3031#issuecomment-3219910366))*<br>[`polars.Expr.drop_nans`](https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.drop_nans.html) |
	| count | [`Expr.null_count`][narwhals.Expr.null_count] | *No upstream equivalent* |

The *Frame-level variants seem the most useful for lazy backends, to avoid issues with checking for NaN against a non-float

Related

• https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.drop_nans.html
• https://docs.pola.rs/api/python/stable/reference/lazyframe/api/polars.LazyFrame.drop_nans.html
• https://docs.pola.rs/api/python/stable/reference/series/api/polars.Series.drop_nans.html
• https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.drop_nans.html

[MarcoGorelli]

i think i'm not keen on adding more nan-specific methods, drop_nans should already be doable with what we have, no?

[dangotbanned]

drop_nans should already be doable with what we have, no?

Sure, here's the examples from the polars docs.

import polars as pl

lf = pl.LazyFrame(
    {
        "foo": [-20.5, float("nan"), 80.0],
        "bar": [float("nan"), 110.0, 25.5],
        "ham": ["xxx", "yyy", None],
    }
)
df = pl.DataFrame({"a": [1.0, None, 3.0, float("nan")]})

print(lf.drop_nans().collect())
print(lf.drop_nans(subset=["bar"]).collect())
print(df.select(pl.col("a").drop_nans()))

Show output

shape: (1, 3)
┌──────┬──────┬──────┐
│ foo  ┆ bar  ┆ ham  │
│ ---  ┆ ---  ┆ ---  │
│ f64  ┆ f64  ┆ str  │
╞══════╪══════╪══════╡
│ 80.0 ┆ 25.5 ┆ null │
└──────┴──────┴──────┘
shape: (2, 3)
┌──────┬───────┬──────┐
│ foo  ┆ bar   ┆ ham  │
│ ---  ┆ ---   ┆ ---  │
│ f64  ┆ f64   ┆ str  │
╞══════╪═══════╪══════╡
│ NaN  ┆ 110.0 ┆ yyy  │
│ 80.0 ┆ 25.5  ┆ null │
└──────┴───────┴──────┘
shape: (3, 1)
┌──────┐
│ a    │
│ ---  │
│ f64  │
╞══════╡
│ 1.0  │
│ null │
│ 3.0  │
└──────┘

My questions are:

1. How would you propose writing these using narwhals APIs?
2. If possible, is the result an idiom that you expect users to remember?

[MarcoGorelli]

1. df.filter(nw.any_horizontal(nw.all().is_nan()))
2. yup

[dangotbanned]

1. df.filter(nw.any_horizontal(nw.all().is_nan()))

I'd appreciate it if you'd try that out first 😂

import narwhals as nw

data = {
    "foo": [-20.5, float("nan"), 80.0],
    "bar": [float("nan"), 110.0, 25.5],
    "ham": ["xxx", "yyy", None],
}

df = nw.DataFrame.from_dict(data, backend="pandas")
df.filter(nw.any_horizontal(nw.all().is_nan()))

TypeError: any_horizontal() missing 1 required keyword-only argument: 'ignore_nulls'

df.filter(nw.any_horizontal(nw.all().is_nan(), ignore_nulls=True))

InvalidOperationError: `.is_nan` only supported for numeric dtype and not String, did you mean `.is_null`?

df.filter(nw.any_horizontal(nw.all().is_nan(), ignore_nulls=False))

InvalidOperationError: `.is_nan` only supported for numeric dtype and not String, did you mean `.is_null`?

I'm pretty sure ignore_nulls=False would be an issue for pandas as well 🤔

2. If possible, is the result an idiom that you expect users to remember?

Surely one of these is preferable 😄

df.filter(nw.any_horizontal(nw.all().is_nan(), ignore_nulls=True))
df.drop_nans()

[MarcoGorelli]

if nan is just a floating point number, then i don't think this needs special-casing, same reason why we don't have drop_zeroes and similar methods for other floating point numbers