base on A hyperparameter optimization framework <div align="center"><img src="https://raw.githubusercontent.com/optuna/optuna/master/docs/image/optuna-logo.png" width="800"/></div> # Optuna: A hyperparameter optimization framework [](https://www.python.org) [](https://pypi.python.org/pypi/optuna) [](https://anaconda.org/conda-forge/optuna) [](https://github.com/optuna/optuna) [](https://optuna.readthedocs.io/en/stable/) [](https://codecov.io/gh/optuna/optuna) :link: [**Website**](https://optuna.org/) | :page_with_curl: [**Docs**](https://optuna.readthedocs.io/en/stable/) | :gear: [**Install Guide**](https://optuna.readthedocs.io/en/stable/installation.html) | :pencil: [**Tutorial**](https://optuna.readthedocs.io/en/stable/tutorial/index.html) | :bulb: [**Examples**](https://github.com/optuna/optuna-examples) | [**Twitter**](https://twitter.com/OptunaAutoML) | [**LinkedIn**](https://www.linkedin.com/showcase/optuna/) | [**Medium**](https://medium.com/optuna) *Optuna* is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, *define-by-run* style user API. Thanks to our *define-by-run* API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters. ## :loudspeaker: News <!-- TODO: when you add a new line, please delete the oldest line --> * **Nov 12, 2024**: We released Optuna 4.1 with new features, Python 3.13 support and much more! Check out [the release note](https://github.com/optuna/optuna/releases/tag/v4.1.0) for details. * **Nov 6, 2024**: A new article [AutoSampler: Automatic Selection of Optimization Algorithms in Optuna](https://medium.com/optuna/autosampler-automatic-selection-of-optimization-algorithms-in-optuna-1443875fd8f9) has been published. * **Oct 21, 2024**: We posted [an article](https://medium.com/optuna/an-introduction-to-moea-d-and-examples-of-multi-objective-optimization-comparisons-8630565a4e89) introducing [MOEA/D](https://hub.optuna.org/samplers/moead/) and an example comparison with other optimization methods. * **Oct 15, 2024**: We posted [an article](https://medium.com/optuna/introducing-a-new-terminator-early-termination-of-black-box-optimization-based-on-expected-9a660774fcdb) about `Terminator`, which is expanded in Optuna 4.0. * **Sep 18, 2024**: We posted [an article](https://medium.com/optuna/introducing-the-stabilized-journalstorage-in-optuna-4-0-from-mechanism-to-use-case-e320795ffb61) about `JournalStorage`, which is stabilized in Optuna 4.0. * **Sep 2, 2024**: Optuna 4.0 is available! You can install it by `pip install -U optuna`. Find the latest [here](https://github.com/optuna/optuna/releases) and check [our article](https://medium.com/optuna/optuna-4-0-whats-new-in-the-major-release-3325a8420d10). ## :fire: Key Features Optuna has modern functionalities as follows: - [Lightweight, versatile, and platform agnostic architecture](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/001_first.html) - Handle a wide variety of tasks with a simple installation that has few requirements. - [Pythonic search spaces](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html) - Define search spaces using familiar Python syntax including conditionals and loops. - [Efficient optimization algorithms](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html) - Adopt state-of-the-art algorithms for sampling hyperparameters and efficiently pruning unpromising trials. - [Easy parallelization](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/004_distributed.html) - Scale studies to tens or hundreds of workers with little or no changes to the code. - [Quick visualization](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/005_visualization.html) - Inspect optimization histories from a variety of plotting functions. ## Basic Concepts We use the terms *study* and *trial* as follows: - Study: optimization based on an objective function - Trial: a single execution of the objective function Please refer to the sample code below. The goal of a *study* is to find out the optimal set of hyperparameter values (e.g., `regressor` and `svr_c`) through multiple *trials* (e.g., `n_trials=100`). Optuna is a framework designed for automation and acceleration of optimization *studies*. <details open> <summary>Sample code with scikit-learn</summary> [](http://colab.research.google.com/github/optuna/optuna-examples/blob/main/quickstart.ipynb) ```python import ... # Define an objective function to be minimized. def objective(trial): # Invoke suggest methods of a Trial object to generate hyperparameters. regressor_name = trial.suggest_categorical('regressor', ['SVR', 'RandomForest']) if regressor_name == 'SVR': svr_c = trial.suggest_float('svr_c', 1e-10, 1e10, log=True) regressor_obj = sklearn.svm.SVR(C=svr_c) else: rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32) regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth) X, y = sklearn.datasets.fetch_california_housing(return_X_y=True) X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0) regressor_obj.fit(X_train, y_train) y_pred = regressor_obj.predict(X_val) error = sklearn.metrics.mean_squared_error(y_val, y_pred) return error # An objective value linked with the Trial object. study = optuna.create_study() # Create a new study. study.optimize(objective, n_trials=100) # Invoke optimization of the objective function. ``` </details> > [!NOTE] > More examples can be found in [optuna/optuna-examples](https://github.com/optuna/optuna-examples). > > The examples cover diverse problem setups such as multi-objective optimization, constrained optimization, pruning, and distributed optimization. ## Installation Optuna is available at [the Python Package Index](https://pypi.org/project/optuna/) and on [Anaconda Cloud](https://anaconda.org/conda-forge/optuna). ```bash # PyPI $ pip install optuna ``` ```bash # Anaconda Cloud $ conda install -c conda-forge optuna ``` > [!IMPORTANT] > Optuna supports Python 3.8 or newer. > > Also, we provide Optuna docker images on [DockerHub](https://hub.docker.com/r/optuna/optuna). ## Integrations Optuna has integration features with various third-party libraries. Integrations can be found in [optuna/optuna-integration](https://github.com/optuna/optuna-integration) and the document is available [here](https://optuna-integration.readthedocs.io/en/stable/index.html). <details> <summary>Supported integration libraries</summary> * [Catboost](https://github.com/optuna/optuna-examples/tree/main/catboost/catboost_pruning.py) * [Dask](https://github.com/optuna/optuna-examples/tree/main/dask/dask_simple.py) * [fastai](https://github.com/optuna/optuna-examples/tree/main/fastai/fastai_simple.py) * [Keras](https://github.com/optuna/optuna-examples/tree/main/keras/keras_integration.py) * [LightGBM](https://github.com/optuna/optuna-examples/tree/main/lightgbm/lightgbm_integration.py) * [MLflow](https://github.com/optuna/optuna-examples/tree/main/mlflow/keras_mlflow.py) * [PyTorch](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_simple.py) * [PyTorch Ignite](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_ignite_simple.py) * [PyTorch Lightning](https://github.com/optuna/optuna-examples/tree/main/pytorch/pytorch_lightning_simple.py) * [TensorBoard](https://github.com/optuna/optuna-examples/tree/main/tensorboard/tensorboard_simple.py) * [TensorFlow](https://github.com/optuna/optuna-examples/tree/main/tensorflow/tensorflow_estimator_integration.py) * [tf.keras](https://github.com/optuna/optuna-examples/tree/main/tfkeras/tfkeras_integration.py) * [Weights & Biases](https://github.com/optuna/optuna-examples/tree/main/wandb/wandb_integration.py) * [XGBoost](https://github.com/optuna/optuna-examples/tree/main/xgboost/xgboost_integration.py) </details> ## Web Dashboard [Optuna Dashboard](https://github.com/optuna/optuna-dashboard) is a real-time web dashboard for Optuna. You can check the optimization history, hyperparameter importance, etc. in graphs and tables. You don't need to create a Python script to call [Optuna's visualization](https://optuna.readthedocs.io/en/stable/reference/visualization/index.html) functions. Feature requests and bug reports are welcome!  `optuna-dashboard` can be installed via pip: ```shell $ pip install optuna-dashboard ``` > [!TIP] > Please check out the convenience of Optuna Dashboard using the sample code below. <details> <summary>Sample code to launch Optuna Dashboard</summary> Save the following code as `optimize_toy.py`. ```python import optuna def objective(trial): x1 = trial.suggest_float("x1", -100, 100) x2 = trial.suggest_float("x2", -100, 100) return x1 ** 2 + 0.01 * x2 ** 2 study = optuna.create_study(storage="sqlite:///db.sqlite3") # Create a new study with database. study.optimize(objective, n_trials=100) ``` Then try the commands below: ```shell # Run the study specified above $ python optimize_toy.py # Launch the dashboard based on the storage `sqlite:///db.sqlite3` $ optuna-dashboard sqlite:///db.sqlite3 ... Listening on http://localhost:8080/ Hit Ctrl-C to quit. ``` </details> ## OptunaHub [OptunaHub](https://hub.optuna.org/) is a feature-sharing platform for Optuna. You can use the registered features and publish your packages. ### Use registered features `optunahub` can be installed via pip: ```shell $ pip install optunahub # Install AutoSampler dependencies (CPU only is sufficient for PyTorch) $ pip install cmaes scipy torch --extra-index-url https://download.pytorch.org/whl/cpu ``` You can load registered module with `optunahub.load_module`. ```python import optuna import optunahub def objective(trial: optuna.Trial) -> float: x = trial.suggest_float("x", -5, 5) y = trial.suggest_float("y", -5, 5) return x**2 + y**2 module = optunahub.load_module(package="samplers/auto_sampler") study = optuna.create_study(sampler=module.AutoSampler()) study.optimize(objective, n_trials=10) print(study.best_trial.value, study.best_trial.params) ``` For more details, please refer to [the optunahub documentation](https://optuna.github.io/optunahub/). ### Publish your packages You can publish your package via [optunahub-registry](https://github.com/optuna/optunahub-registry). See the [OptunaHub tutorial](https://optuna.github.io/optunahub-registry/index.html). ## Communication - [GitHub Discussions] for questions. - [GitHub Issues] for bug reports and feature requests. [GitHub Discussions]: https://github.com/optuna/optuna/discussions [GitHub issues]: https://github.com/optuna/optuna/issues ## Contribution Any contributions to Optuna are more than welcome! If you are new to Optuna, please check the [good first issues](https://github.com/optuna/optuna/labels/good%20first%20issue). They are relatively simple, well-defined, and often good starting points for you to get familiar with the contribution workflow and other developers. If you already have contributed to Optuna, we recommend the other [contribution-welcome issues](https://github.com/optuna/optuna/labels/contribution-welcome). For general guidelines on how to contribute to the project, take a look at [CONTRIBUTING.md](./CONTRIBUTING.md). ## Reference If you use Optuna in one of your research projects, please cite [our KDD paper](https://doi.org/10.1145/3292500.3330701) "Optuna: A Next-generation Hyperparameter Optimization Framework": <details open> <summary>BibTeX</summary> ```bibtex @inproceedings{akiba2019optuna, title={{O}ptuna: A Next-Generation Hyperparameter Optimization Framework}, author={Akiba, Takuya and Sano, Shotaro and Yanase, Toshihiko and Ohta, Takeru and Koyama, Masanori}, booktitle={The 25th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining}, pages={2623--2631}, year={2019} } ``` </details> ## License MIT License (see [LICENSE](./LICENSE)). Optuna uses the codes from SciPy and fdlibm projects (see [LICENSE_THIRD_PARTY](./LICENSE_THIRD_PARTY)). ", Assign "at most 3 tags" to the expected json: {"id":"7721","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"