nlothian/llm-sql-benchmark

[NeurIPS 2025 Main] SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications

We assessed the ability of popular LLMs to generate accurate and efficient SQL from natural language prompts. Using a 200 million record dataset from the GH Archive uploaded to Tinybird, we asked the LLMs to generate SQL based on 50 prompts.

EvalBench is a flexible framework designed to measure the quality of generative AI (GenAI) workflows around database specific tasks.

[ICLR 2025 Oral] Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

Repository hosting code to reproduce our paper (with Stanford, TogetherAI and Univ. Wisconsin-Madison), "DBPLANBENCH: Test-Time Optimization of Physical Query Plans with LLMs"

Evaluate the accuracy of LLM generated outputs

GenDB, an LLM-Powered Generative Query Engine Built for the Future

Open-Source Agentic RL for SQL Tasks