Feature fraction gives different column selections when using a custom objective

[jrvmalik]

Description

I ran into this because I got different column subsampling behaviour when using a built-in objective like MSE and when using a custom objective. I thought either: the reported seed in the booster.txt was wrong, or that columns are subsampled twice each iteration when using a custom objective, but only once when using a built-in objective (via perhaps an extra call to 'ResetByTree').

Reproducible example

import lightgbm as lgb
import numpy as np
import pandas as pd

np.random.seed(1)
X = np.random.randn(300, 300)
y = np.random.randn(300)

data = lgb.Dataset(X, label=y, init_score=np.zeros_like(y))
params = {'max_depth': 1, 'learning_rate': 0.001, 'verbose': -1, 'objective': 'l2', 'feature_fraction': 1.0, 'seed': 0}

model = lgb.train(params, data, num_boost_round=1000)
model_dict = model.dump_model(num_iteration=3)['tree_info']

def fobj(predictions, dataset):
    return predictions - dataset.get_label(), np.ones_like(predictions)

model_custom = lgb.train(params, data, num_boost_round=1000, fobj=fobj)
model_custom_dict = model_custom.dump_model(num_iteration=3)['tree_info']

model_custom_dict == model_dict

This works when feature fraction is equal to 1.0 only. It breaks when feature_fraction = 0.5 for instance.

I can hack around the seeds to make it work:

import lightgbm as lgb
import numpy as np
import pandas as pd

np.random.seed(1)
X = np.random.randn(300, 300)
y = np.random.randn(300)

data = lgb.Dataset(X, label=y, init_score=np.zeros_like(y))
params = {'max_depth': 1, 'learning_rate': 0.001, 'verbose': -1, 'objective': 'l2', 'feature_fraction': 0.5, 'seed': 0, 'feature_fraction_seed': 974891790}

model = lgb.train(params, data, num_boost_round=1000)
model_dict = model.dump_model(num_iteration=3)['tree_info']

def fobj(predictions, dataset):
    return predictions - dataset.get_label(), np.ones_like(predictions)

params['feature_fraction_seed'] = 2
model_custom = lgb.train(params, data, num_boost_round=1000, fobj=fobj)
model_custom_dict = model_custom.dump_model(num_iteration=3)['tree_info']

model_custom_dict == model_dict

Environment info

LightGBM version or commit hash: 8d01d648942a427f6bb4962dc3f4330e005fa495

Command(s) you used to install LightGBM

pip install lightgbm

[jameslamb]

Thanks for using LightGBM and for your thorough report.

What is commit 8d01d648942a427f6bb4962dc3f4330e005fa495? I don't see that commit in LightGBM's history: 8d01d64.

I strongly suspect that that is from some version of LightGBM prior to v4.0.0. The parameter fobj was removed from lgb.train() in #5052, so your example code yields the following error on latest master (8203306).

Traceback (most recent call last):
File "", line 1, in
TypeError: train() got an unexpected keyword argument 'fobj'

---

I ran the code from your report today on that latest commit, with the following modifications:

• passing fobj through params (the new encouraged pattern, see [python-package] Where has the "fobj" parameter gone in lightgbm.cv() in v4.0.0? #6072 for example)
• passing "deterministic": True and "force_row_wise": True", to eliminate more possible sources of randomness
• setting "verbose": 1
  • we are debugging here, getting logs might be helpful
• reducing num_boost_round too 3
  • it's not necessary to train for 1000 rounds if the issue is detectable after 3

code (click me)

import lightgbm as lgb
import numpy as np
import pandas as pd

np.random.seed(1)
X = np.random.randn(300, 300)
y = np.random.randn(300)

data = lgb.Dataset(
    X,
    label=y,
    init_score=np.zeros_like(y),
)
params = {
    'deterministic': True,
    'force_row_wise': True,
    'max_depth': 1,
    'learning_rate': 0.001,
    'verbose': -1,
    'objective': 'l2',
    'feature_fraction': 0.5,
    'seed': 123
}

model = lgb.train(params, data, num_boost_round=1000)
model_dict = model.dump_model()


def fobj(predictions, dataset):
    return predictions - dataset.get_label(), np.ones_like(predictions)


model_custom = lgb.train(
    params={**params, "objective": fobj},
    train_set=data,
    num_boost_round=1000
)
model_custom_dict = model_custom.dump_model()

assert model_custom_dict == model_dict

I saw that the first splits were identical

assert model_custom_dict["tree_info"][0] == model_dict["tree_info"][0]

But then they started to diverge starting with the second split.

assert model_custom_dict["tree_info"][1] == model_dict["tree_info"][1]

I'm not sure exactly what's happening, but that narrows it down a bit.

---

I can hack around the seeds to make it work:

I did not understand this comment and example. Do you just mean that you experimented with different values of seeds until you found a combination of 2 different ones led to the models being identical?

[Shanxce]

If we set objective in python, in python code will call reset_parameter, which will call col_sampler.SetConfig, this function will call ResetByTree and randomly select feature columns. Thus, in custom objective, it will call one more random feature selection at first, result in all random selection is one time older than default pattern. @jameslamb @jrvmalik

[jameslamb]

Good investigation @Shanxce ! I haven't looked closely at the links you shared, but that sounds like a plausible explanation to me.

Are you interested in submitting a PR to try to fix this? I'm not quite sure what the best approach is, but I do think that this behavior from LightGBM is surprising and that how you provide the loss funciton should not change which columns are chosen when sampling columns.