preprocessing.outlier.mark_outliers

preprocessing.outlier.mark_outliers(
    data,
    *,
    contamination=0.1,
    random_state=1234,
    verbose=False,
    columns=None,
)

Mark outliers as NaN in a new frame, using Isolation Forest.

Pure: data is never modified. cleaned is a NEW frame built with :meth:pandas.DataFrame.mask; the input frame survives the call untouched, so a caller no longer needs a defensive .copy().

Parameters

Name	Type	Description	Default
data	pd.DataFrame	The input dataset.	required
contamination	float	The (estimated) proportion of outliers per fitted column.	`0.1`
random_state	int	Random seed for reproducibility. Default is 1234.	`1234`
verbose	bool	Whether to print additional information. Printing happens only for the columns actually fitted (see `columns`).	`False`
columns	Sequence[str] \| None	Columns to fit. `None` (default) fits every column of `data`. When given, only the named columns are fitted; columns outside `columns` are passed through `cleaned` untouched and are all-`False` in `mask`.	`None`

Returns

Name	Type	Description
	tuple[pd.DataFrame, pd.DataFrame]	tuple[pd.DataFrame, pd.DataFrame]: `(cleaned, mask)`. `cleaned` is a new frame in which every cell `mask` marks `True` is NaN. `mask` is the full-width boolean frame from :func:`outlier_mask` (same index and columns as `data`, `True` == outlier); per-column counts are `mask[col].sum()`.

Raises

Name	Type	Description
	TypeError	If `data` is not a pandas DataFrame.
	ValueError	If `data` is empty or contains no columns.
	KeyError	If `columns` names a column not present in `data`.

Examples

import numpy as np
import pandas as pd

from spotforecast2_safe.preprocessing.outlier import mark_outliers

rng = np.random.default_rng(0)
# 50 normal values plus two clear outliers (1000, -1000)
values = np.concatenate([rng.normal(loc=10.0, scale=1.0, size=50), [1000.0, -1000.0]])
data = pd.DataFrame({"load": values})

cleaned, mask = mark_outliers(
    data, contamination=0.05, random_state=42, verbose=True
)
n_nan = cleaned["load"].isna().sum()
print(f"Outliers marked as NaN: {n_nan}")
assert n_nan >= 2, "Expected at least the two injected extreme outliers to be marked"
assert n_nan == int(mask["load"].sum())
assert data["load"].notna().all(), "The input frame is never modified"

# columns= restricts fitting to a subset; the rest passes through.
two_col = pd.DataFrame({"load": values, "other": np.zeros_like(values)})
cleaned_subset, mask_subset = mark_outliers(
    two_col, contamination=0.05, random_state=42, columns=["load"]
)
assert not mask_subset["other"].any()
assert cleaned_subset["other"].equals(two_col["other"])

Column 'load': Marked 5.7692% of data points as outliers.
Outliers marked as NaN: 3