Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: LGBM Results Not Reproducible #1368

Closed
dannycg1996 opened this issue Oct 24, 2024 · 0 comments · Fixed by #1369
Closed

[Bug]: LGBM Results Not Reproducible #1368

dannycg1996 opened this issue Oct 24, 2024 · 0 comments · Fixed by #1369
Assignees
Labels
bug Something isn't working

Comments

@dannycg1996
Copy link
Collaborator

Describe the bug

Currently there are a couple of issues around the reproducibility of the FLAML best loss, when the best estimator found was an LGBMEstimator.
These issues occur as follows:

  1. The best loss returned by FLAML is not reproducible, using the underlying LGBMClassifier or LGBMEstimator model (i.e. automl.model.model). This seems to be caused by n_estimators always being set to 1, regardless of what value it should be. Please note that n_estimators always seems to be set correctly on the FLAMLised LGBMEstimator - this issue exists exclusively on the underlying models.
  2. With certain configurations, the FLAML best loss can't be reproduced even when using the FLAMLised LGBMEstimator. This seems to only be the case when a time budget is set, so the issue is likely caused by the callbacks, similar to the issue found with CatBoostEstimators here

Steps to reproduce

In terms of showcasing these issues in a reproducible manner:

  1. Can be replicated by uncommenting and then running coverage run -m pytest -k test_reproducibility_of_underlying_regression_models[lgbm] (or the classification equivalent.
  2. A unit test will be added for this, which will initially fail, until the issue is fixed.

I'll open a PR for this shortly - hope that's okay

Model Used

LGBMEstimators

Expected Behavior

Taking the FLAMLised LGBMEstimator model, or the underlying model it wraps (LGBMClassifier/LGBMRegressor) and training and testing it on the same folds, should return the same best loss as FLAML.

Screenshots and logs

No response

Additional Information

No response

@dannycg1996 dannycg1996 added the bug Something isn't working label Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants