api: plan for narwhals.stable.v2

[MarcoGorelli]

I think we can aim for narwhals.stable.v2 some time in 2025

The main things I'd like to achieve are:

• remove "interchange" level of support, and instead only have:
  • "full": must support Series and have a defined row order (pandas, polars.DataFrame, pyarrow.Table, modin.DataFrame, cudf.DataFrame)
  • "lazy": must support collect, adding/removing/subsetting columns, filtering rows. Initially, at least, row-order-dependent operations (such as nw.cum_sum) will be left out. This will include Polars.LazyFrame, DuckDBPyRelation, pyspark.DataFrame, ibis.Table, dask.DataFrame
• make the breaking changes listed in breaking changes

Candidate changes:

• rework from_native. It will get simpler once we remove interchange-level things, but it's still fairly complicated. Not sure how much I like it to be completely honest, but I also haven't come up with anything better

Usage of narwhals.stable.v1 should remain unaffected

Tentative date: March 2025

---

Why remove support for the dataframe interchange protocol?

• the project has received no updates in over a year, issues and requests go unanswered, and it has no governance model in place so nobody is taking responsibility for it
• only a handful of dataframe libraries implement the protocol, and for most of them, we already provide (or are working to provide) native support in Narwhals. The only exception is Vaex, which itself seems to be deprecated (no releases in ~1.5 years), but we could support that too if there was demand
• from a Narwhals usage perspective, it adds a lot of complexity to have 3 layers of support. It's much simpler to just have: full, and lazy.

[MarcoGorelli]

I think I'd also like to remove row-order-dependent expressions:

- arg_max - arg_min - cum_count - cum_max - cum_min - cum_prod - cum_sum - diff - drop_nulls - ewm_mean - filter - gather_every - head - is_first_distinct - is_last_distinct - mode - rolling_mean - rolling_std - rolling_sum - rolling_var - sample - shift - sort - unique

These can all stay available for Series, so people would still be able to use them for eager execution

All of these would of course stay available in narwhals.stable.v1

---

In #1689 (comment), we find:

• Plotly is using Expr.cum_sum
• scikit-lego is using Expr.shift

So, it would take more planning to be able to do this safely

Example (maybe?) of how shift_by could be implemented for Polars:

def shift_by(expr, by):
    idx = pl.arg_sort_by(by)
    return pl.col(expr).gather(idx).shift().sort_by(idx)

[MarcoGorelli]

🤔 fill_null with strategies like forward would also become problematic

it's debatable whether to have by kwargs (which would be required for the lazy-only layer) or to have *_by variants

[MarcoGorelli]

I just tried the following:

import polars as pl
import numpy as np

rng = np.random.default_rng(1)
N = 10_000_000
a = rng.normal(size=N)
b = rng.normal(size=N)
c = rng.normal(size=N)
data = {'a': a, 'b': b, 'c': c}
df_pl = pl.DataFrame(data)
df_pd = pd.DataFrame(data)

def shift_by(expr, by, lag):
    idx = pl.arg_sort_by(by)
    return pl.col(expr).gather(idx).shift(lag).sort_by(idx)

def add_lags_polars_1(df, cols, lags):
    return df.with_columns(
        pl.col(col).shift(-lag).alias(f'{col}_{lag}')
        for col in cols
        for lag in lags
    )

def add_lags_polars_2(df, cols, lags):
    return df.with_row_index().with_columns(
        shift_by(col, 'index', -lag).alias(f'{col}_{lag}')
        for col in cols
        for lag in lags
    )

def add_lags_polars_3(df, cols, lags):
    return df.with_columns(
        df[col].shift(-lag).alias(f'{col}_{lag}')
        for col in cols
        for lag in lags
    )

Timings:

• add_lags_polars_1 (current): 1.63ms
• add_lags_polars_2 (using shift_by): ~1300ms
• add_lags_polars_3 (using Series.shift): 4.3ms

So, I'm not sure it's really warranted to remove Expr.shift

[MarcoGorelli]

Something else to discuss is nw.is_ordered_categorical, whose signature is:

 is_ordered_categorical(series) 

The signature means we need to have an (eager) Series to determine whether a column is ordered categorical, which is problematic for lazy implementations.

What we could have instead is:

is_ordered_categorical(frame, column_name) -> bool

Then, the logic would go:

• pandas, Polars, PyArrow: same as now
• duckDB / Ibis / PySpark (which have no Categorical data type, as far as I can tell?): always return False
• as a last fallback, for anything eager which implements the dataframe interchange protocol, check if this data type is described as ordered categorical

EDIT

The above change might not actually be necessary. In Altair, they need the unique categories:

https://github.com/vega/altair/blob/be5e9ecd61d099c847dca44fb3a4283940c1e3b8/altair/utils/core.py#L707-L710

So, we could do something like this there:

if (
    # Getting the unique categories would require materialising a LazyFrame,
    # so we only auto-infer ordinal in the eager case.
    isinstance(df, nw.DataFrame)
    and nw.is_ordered_categorical(df[col_name])
    and not (categories := df[col_name].cat.get_categories()).is_empty()
):
    return "ordinal", categories.to_list()