benchopt/benchopt

Collective Knowledge (CK), Collective Mind (CM/CMX) and MLPerf automations: community-driven projects to learn how to run AI, ML, and other emerging workloads more efficiently and cost-effectively across diverse models, datasets, software, and hardware using MLPerf methodology and benchmarks.

Python-based research interface for blackbox and hyperparameter optimization, based on the internal Google Vizier Service.

Large scale and asynchronous Hyperparameter and Architecture Optimization at your fingertips.

A hyperparameter optimization framework

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.

SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization

AIDE: an LLM agent for machine learning engineering - the research Weco grew out of. Referenced in OpenAI MLE-bench.

Track your Data Science. Skore's open-source Python library accelerates ML model development with automated evaluation reports, smart methodological guidance, and comprehensive cross-validation analysis.

MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineering