Outlier Detection and Handling

Guide for identifying and handling outliers in time series data.

Overview

Outlier detection is crucial for time series forecasting as extreme values can distort model training and predictions. This module provides robust outlier detection and marking capabilities.

Key Functions

TipAPI Reference

Full API documentation for outlier_mask, mark_outliers and the outlier module is auto-generated in the Preprocessing Reference.

outlier_mask

outlier_mask is the module’s detector primitive: a boolean DataFrame, same index and columns as the input, True where Isolation Forest flags a cell. Every other outlier view derives from it — labels, counts, values.

mark_outliers

The mark_outliers function identifies outliers in a time series DataFrame and returns a new frame with the flagged cells replaced by NaN, alongside the outlier_mask it used to do so. It is pure: the input frame is never modified. See Preprocessing Reference for the full signature, parameters, and return values.

Examples

import pandas as pd
from spotforecast2_safe.preprocessing.outlier import mark_outliers

# Create sample time series data
data = pd.DataFrame({
    'value': [1, 2, 100, 4, 5, 6, 7, 8, 9, 10],  # 100 is an outlier
})

# Mark outliers. `data` is never modified, so no defensive copy is needed.
result_data, mask = mark_outliers(
    data,
    contamination=0.1,  # Expect 10% contamination
)

outlier_count = int(mask['value'].sum())
print(f"Outliers marked: {outlier_count} records")

pd.DataFrame({
    'value (before detection)':          data['value'],
    'value (after detection)':          result_data.squeeze(),
    'is_outlier':     mask['value'],
})
Outliers marked: 1 records
value (before detection) value (after detection) is_outlier
0 1 1.0 False
1 2 2.0 False
2 100 NaN True
3 4 4.0 False
4 5 5.0 False
5 6 6.0 False
6 7 7.0 False
7 8 8.0 False
8 9 9.0 False
9 10 10.0 False

Detection Methods

This module uses isolation forest and other statistical methods to detect:

  • Sudden spikes or drops
  • Seasonal anomalies
  • Drift in baseline values
  • Sudden shifts in variance