Generate

optuna

Dec 27, 2020. | By: @vsoch

Hyperparameter tuning has been around as long as any major cornerstone of machine learning. So why another package for tuning hyperparameters? This week on the Research Software Showcase we share optuna, which we assure you is worth trying out if you need to do this kind of optimization.


/rseng/assets/img/posts/showcase/optuna.png


Are you already familiar with this software? We encourage you to contribute to the research software encyclopedia and annotate the respository:

otherwise, keep reading!

What is optuna?

Tuning hyperparameters can often be pretty difficult. Essentially, it is an optimization problem – but in the most general setting there are no gradients to use, and problems can be fairly high-dimensional. For this reason many hyperparameter tuning libraries have focused on easily parallelizing the parameter search, while others have added pruning abilities (stop unpromising trials early to save time).

The main selling point of Optuna is allowing the user to dynamically construct their search space using a ‘define-by-run’ paradigm. This approach has been increasingly used for training neural networks (by allowing the user to dynamically define a computational graph) but until now has not been used for hyperparameter tuning. In their paper, the authors include a snippet of code that defines a search space dynamically versus statically. In order to achieve the same behavior with a statistically-defined search space, the code becomes extremely convoluted even in a simple example.

Optuna is also lightweight and easy to set up, efficient, parallelizable, and comes with a nice set of visualization tools. It is also highly modular because of the define-by-run design. Here is a quick example from their documentation to:

  1. Define an objective function to be optimized (in the example, we minimize (x - 2)^2
  2. Suggest a range of hyperparameter values (e.g., -10 to 10)
  3. Create a study object and optimize over 100 trials
import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

study = optuna.create_study()
study.optimize(objective, n_trials=100)

study.best_params  # E.g. {'x': 2.002108042}

This is really intuitive and simple, which makes it great! You should check out the code examples that show interaction with additional libraries like PyTorch, Tensorflow, Sci-kit learn, and others.

How do I cite it?

You can reference their pre-print paper on ArXiv or from proceedings (preferred):

@inproceedings{optuna_2019,
    title={Optuna: A Next-generation Hyperparameter Optimization Framework},
    author={Akiba, Takuya and Sano, Shotaro and Yanase, Toshihiko and Ohta, Takeru and Koyama, Masanori},
    booktitle={Proceedings of the 25rd {ACM} {SIGKDD} International Conference on Knowledge Discovery and Data Mining},
    year={2019}
}

@misc{akiba2019optuna,
      title={Optuna: A Next-generation Hyperparameter Optimization Framework}, 
      author={Takuya Akiba and Shotaro Sano and Toshihiko Yanase and Takeru Ohta and Masanori Koyama},
      year={2019},
      eprint={1907.10902},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

How do I get started?

The main site documentation is really great because it provides many different methods of learning, from reading, to examples, to video.

How do I contribute to the software survey?

or read more about annotation here. You can clone the software repository to do bulk annotation, or annotation any repository in the software database, We want annotation to be fun, straight-forward, and easy, so we will be showcasing one repository to annotate per week. If you’d like to request annotation of a particular repository (or addition to the software database) please don’t hesitate to open an issue or even a pull request.

Where can I learn more?

You might find these other resources useful:

For any resource, you are encouraged to give feedback and contribute!

Categories

News 2

Tutorials 2

Software 33

Recent Posts

Popular Tags

News (2) Tutorials (2) Software (33)

twitter

@vsoch