Choose the library based on the method and not the method based on the library
Find the Right Hyperparameter Tuning Method—Not Just the Library!
I often hear people say, "I use Hyperopt for hyperparameter tuning," or "Optuna produces the best results."
It almost sounds like these libraries work some kind of magic, but the reality is that they implement well-researched methods. Yes, they’re effective, but they’re not magic—they’re based on math.
Both Optuna and Hyperopt utilize common techniques, including:
Bayesian Optimization using Tree Parzen Estimators as surrogates
Randomized Search
However, they also have distinct differences:
Optuna makes it easier to set up searches
Optuna has better documentation
Optuna supports additional algorithms, including multifidelity optimization, Bayesian optimization with Gaussian processes, grid search, and genetic algorithms
That said, neither Optuna nor Hyperopt currently supports Bayesian optimization with random forests as a surrogate, which some studies suggest is the best option for hyperparameter tuning.
Additionally, scikit-learn provides randomized search and multifidelity models as well. So why would you use something else when training scikit-learn models?
In short, understanding how these libraries work empowers you to choose the one that best fits your needs rather than adapting your needs to fit a library.
Knowledge is power!
With this in mind, which method for hyperparameter tuning do you use? Let me know in the comments!
Ready to enhance your skills?
Our specializations, courses and books are here to assist you:
Advanced Machine Learning (specialization)
Forecasting with Machine Learning (course)