CatBoost Time Series

CatBoost for time series combines gradient boosting with lag features and native categorical feature handling. It's designed to work well with minimal hyperparameter tuning, making it robust and easy to use while excelling with categorical data.

When to Use CatBoost Time Series

CatBoost Time Series is best suited for:

• Time series with many categorical features (product IDs, store locations, user segments)
• When you want strong performance with minimal hyperparameter tuning
• High-cardinality categorical variables
• Datasets with missing values (handles automatically)
• Robust forecasting with less risk of overfitting
• Production systems requiring stable, reliable models
• When you want the power of gradient boosting without extensive tuning

Strengths

• Excellent handling of categorical features (best in class)
• Robust default parameters (works well out-of-box)
• Handles missing values automatically
• Built-in overfitting protection mechanisms
• Ordered boosting reduces overfitting
• Fast prediction time
• Feature importance for interpretability
• Good with small to medium datasets
• GPU acceleration available
• Minimal hyperparameter tuning needed

Weaknesses

• Slower training than LightGBM (more conservative algorithm)
• Requires feature engineering (lags, rolling stats)
• Needs historical data for lag features
• Cannot extrapolate beyond training distribution
• No native uncertainty quantification
• More memory-intensive than LightGBM
• Still requires exogenous features at forecast time
• Less mature ecosystem than XGBoost

Parameters

Common Time Series Parameters

All time series models share these parameters:

• Timestamp Column (required): Column containing dates/times
• Target Column (required): Numeric value to forecast
• Feature Columns (optional): Additional feature columns (especially categorical)
• Frequency (optional): Time spacing (D, H, W, M). Auto-inferred if not specified
• Forecast Steps (required, default=1): How many periods to predict

Feature Engineering Parameters

Lag Features

• Type: List of integers
• Default: [1, 2, 3, 7]
• Description: Past time steps to include as features
• Examples:
  • Daily data: [1, 7, 14, 30]
  • Hourly data: [1, 24, 168]
  • Monthly data: [1, 12]
• Guidance: Include lags at domain-relevant intervals

Rolling Mean Windows

• Type: List of integers
• Default: [7, 14]
• Description: Window sizes for rolling average features
• Examples:
  • Daily: [7, 14, 30]
  • Hourly: [24, 168]
• Purpose: Captures trends and smooths noise

CatBoost Model Parameters

Iterations

• Type: Integer
• Default: 1000
• Description: Number of boosting iterations (trees)
• Typical Range: 100-1000
• Guidance:
  • Default 1000 often works well
  • Use early stopping to find optimal number
  • CatBoost's conservative algorithm prevents overfitting even with many trees

Tree Depth

• Type: Integer
• Default: 6
• Description: Maximum depth of each tree
• Typical Range: 4-10
• Guidance:
  • 4-6: Balanced, good default
  • 7-10: Deeper trees for complex interactions
  • CatBoost handles depth well due to ordered boosting

Learning Rate

• Type: Float
• Default: 0.03
• Description: Step size for each iteration
• Typical Range: 0.01-0.1
• Guidance:
  • 0.03: Default, works well for most cases
  • 0.01-0.02: Slower, more robust
  • 0.05-0.1: Faster, fewer trees needed
• Note: CatBoost's default is lower than XGBoost/LightGBM (0.03 vs 0.1)

Configuration Tips

Quick Start Configuration

For most time series, simply use defaults:

add_lags=[1, 7]
add_rolling_mean=[7, 14]
iterations=1000
depth=6
learning_rate=0.03

CatBoost's robust defaults often work without tuning.

Feature Engineering by Frequency

Daily Data:

add_lags=[1, 7, 14, 30]
add_rolling_mean=[7, 14, 30]

Hourly Data:

add_lags=[1, 24, 168]
add_rolling_mean=[24, 168]

Monthly Data:

add_lags=[1, 12]
add_rolling_mean=[3, 6, 12]

When to Tune Hyperparameters

CatBoost often works well with defaults, but tune if:

• Initial results are unsatisfactory
• You have specific performance requirements
• You want to optimize speed vs accuracy trade-off

Conservative (Prevent Overfitting):

iterations=500
depth=4
learning_rate=0.03

Aggressive (Capture Complexity):

iterations=1500
depth=8
learning_rate=0.05

Leveraging Categorical Features

CatBoost excels with categorical data:

feature_columns=[
  'product_category',  # categorical
  'store_id',          # categorical
  'day_of_week',       # categorical
  'temperature'        # numeric
]

Advantages:

• No preprocessing needed (no one-hot encoding)
• Handles high-cardinality categories (1000s of unique values)
• Robust to category leakage
• Missing categories handled automatically

Preparation: Mark categorical columns appropriately (dtype='category' or pass cat_features parameter).

CatBoost's Unique Features

1. Ordered Boosting: Reduces overfitting by using different permutations
2. Ordered Target Statistics: Handles categorical features without target leakage
3. Built-in Overfitting Detection: Automatically monitors validation performance

Common Issues and Solutions

Issue: Training Is Slow

Solution:

• CatBoost is slower than LightGBM by design (more conservative)
• Reduce iterations to 500 or use early stopping
• Decrease depth to 4-5
• Use CPU multiprocessing (set thread_count)
• Enable GPU if available
• Subsample data for prototyping

Issue: Overfitting (Training Accuracy >> Validation)

Solution:

• CatBoost is designed to resist overfitting, but it can still happen
• Reduce depth to 4-5
• Decrease iterations or use early stopping
• Increase learning_rate to 0.05 (fewer iterations needed)
• Add L2 regularization (l2_leaf_reg parameter)
• Use fewer lag features

Issue: Underfitting (Poor Performance)

Solution:

• Increase iterations to 1500-2000
• Increase depth to 8-10
• Add more lag and rolling features
• Include relevant external features
• Ensure categorical features are properly marked
• Check data quality (missing values, outliers)

Issue: Poor Long-Term Forecasts

Solution:

• Gradient boosting can't extrapolate trends
• Use for short-term forecasts (1-30 steps)
• Combine with statistical models (ARIMA, Prophet)
• Ensure max lag >= forecast_steps
• Consider ensemble forecasts

Issue: Categorical Features Not Working

Solution:

• Verify categorical columns are marked correctly
• Pass cat_features parameter explicitly
• Check for missing values (CatBoost handles them, but verify)
• Ensure categories are strings or integers (not floats)
• Use categorical encoding if needed (ordinal relationships)

Issue: Need Prediction Intervals

Solution:

• CatBoost doesn't provide native intervals
• Use quantile regression (set loss_function='Quantile')
• Train separate models for different quantiles (0.1, 0.5, 0.9)
• Bootstrap methods
• Conformal prediction
• Ensemble with statistical models

Issue: Want Faster Training

Solution:

• CatBoost prioritizes accuracy over speed
• Use LightGBM if speed is critical
• Or reduce iterations and increase learning_rate
• Enable GPU acceleration
• Use sampling (subsample parameter)

Example Use Cases

Multi-Product Retail Forecasting

target: daily_sales
feature_columns: [product_id, store_id, category, subcategory, is_promotion]
add_lags: [1, 7, 14]
add_rolling_mean: [7, 30]
iterations: 1000
depth: 6

Excellent for high-cardinality categorical features (product_id, store_id).

User Behavior Prediction

target: daily_active_users
feature_columns: [user_segment, device_type, country, is_weekend]
add_lags: [1, 7]
add_rolling_mean: [7, 14]
iterations: 1000
depth: 6

Handles user segments and device types robustly.

Energy Consumption by Building

target: kwh_usage
feature_columns: [building_id, building_type, temperature, humidity, hour]
add_lags: [1, 24, 168]
add_rolling_mean: [24, 168]
iterations: 1500
depth: 7

Multiple buildings with categorical building types.

E-commerce Demand Forecasting

target: units_ordered
feature_columns: [product_category, brand, price_tier, day_of_week]
add_lags: [1, 7]
add_rolling_mean: [7, 14]
iterations: 1000
depth: 6

Robust handling of product categories and brands.

Comparison with Other Models

vs XGBoost Time Series:

• CatBoost: Better with categorical features, less tuning needed, slower training
• XGBoost: Faster training, more mature, similar accuracy

vs LightGBM Time Series:

• CatBoost: Better with categorical features, more robust defaults, slower
• LightGBM: Faster, lower memory, requires more tuning

vs ARIMA/Prophet:

• CatBoost: Non-linear, handles many features, flexible
• ARIMA/Prophet: Statistical, confidence intervals, better for pure time series

vs Neural Networks:

• CatBoost: Less data needed, faster, interpretable, easier to tune
• Neural Networks: Better for very complex patterns, sequential dependencies

Advanced Tips

Feature Engineering Enhancements

1. Time-Based Features:

   • day_of_week, month, quarter, is_weekend, is_holiday, week_of_year
   • Mark as categorical for CatBoost to learn non-linear day effects

2. Lag Transformations:

   • Differences: target[t] - target[t-1]
   • Ratios: target[t] / target[t-7]

3. Rolling Statistics:

   • rolling_std, rolling_min, rolling_max
   • Multiple windows: [7, 14, 30]

4. Categorical Interactions:

   • CatBoost learns these automatically
   • Example: category × day_of_week

5. Target Encoding (if not using CatBoost's native):

   • Encode categorical by mean target value
   • But CatBoost handles this internally with ordered target stats

Validation Strategy

Time Series Cross-Validation:

Split 1: Train [0:100], Validate [100:110]
Split 2: Train [0:110], Validate [110:120]
Split 3: Train [0:120], Validate [120:130]

Never shuffle! Respect temporal order.

Early Stopping

Enable early stopping to optimize iterations:

early_stopping_rounds=50
eval_metric='RMSE'

Stops if validation metric doesn't improve for 50 rounds.

Handling High-Cardinality Categorical Features

CatBoost is designed for this:

• Product IDs (1000s of unique values)
• User IDs (millions)
• Store locations (100s)

No need for dimensionality reduction or encoding tricks.

Model Interpretation

CatBoost provides:

• Feature Importance: Which features matter most
• SHAP Values: Individual prediction explanations
• Prediction Diff: Compare feature contributions

Use for understanding what drives forecasts.

Technical Details

Ordered Boosting

CatBoost uses ordered boosting to reduce overfitting:

• Uses random permutations of training data
• Ensures unbiased gradient estimates
• Prevents target leakage in categorical features

This makes CatBoost more robust but slower than LightGBM.

Ordered Target Statistics

For categorical features, CatBoost:

1. Computes target statistics (mean, count) per category
2. Uses ordered approach to prevent leakage
3. Handles new categories at inference time

This is why CatBoost excels with categorical data.

Symmetric Trees

CatBoost builds symmetric trees (balanced):

• All leaves at same level have same depth
• More interpretable than asymmetric trees (LightGBM)
• Faster prediction time

Multi-Step Forecasting

For forecast_steps > 1:

1. Train 1-step model
2. Predict t+1
3. Use prediction as lag for t+2
4. Repeat recursively

Recursive approach means errors accumulate.

Production Considerations

Model Deployment

CatBoost models:

• Fast inference (optimized C++ backend)
• Moderate model size
• Easy serialization (CatBoost native format, pickle)
• GPU inference support

Monitoring

Track:

• Forecast accuracy over time
• Feature distributions (drift detection)
• New categorical values
• Retrain schedule

Retraining

• Incremental: Add new data periodically (weekly/monthly)
• Full: Retrain from scratch to avoid concept drift
• Adaptive: Retrain when performance degrades

Handling New Categories

CatBoost handles unseen categories automatically:

• Uses a special "unknown" category
• Applies regularization
• Robust to new products, stores, users

Recommended Workflow

1. Start with Defaults: CatBoost's defaults are strong
2. Engineer Features: Add relevant lags and rolling stats
3. Mark Categoricals: Ensure categorical features are identified
4. Validate: Use time series cross-validation
5. Compare: Benchmark against other models
6. Tune (if needed): Adjust only if defaults underperform
7. Deploy: Monitor and retrain regularly

Why Choose CatBoost?

Choose CatBoost when:

• You have many categorical features
• You want robust performance with minimal tuning
• You need stable, production-ready models
• You're dealing with high-cardinality categories
• You want good performance without being a hyperparameter expert

CatBoost's philosophy: "It should just work" with sensible defaults.