base on Benchmarking long-form factuality in large language models. Original code for our paper "Long-form factuality in large language models". # Long-Form Factuality in Large Language Models This is the official code release accompanying our paper ["Long-form factuality in large language models"](https://arxiv.org/abs/2403.18802). This repository contains: 1. **LongFact**: A prompt set of 2,280 fact-seeking prompts requiring long-form responses. 2. **Search-Augmented Factuality Evaluator (SAFE)**: Automatic evaluation of model responses in long-form factuality settings. 3. **F1@K**: Extending F1 score to long-form settings using recall from human-preferred length. 4. **Experimentation pipeline** for benchmarking OpenAI and Anthropic models using LongFact + SAFE. ## Installation First, clone our GitHub repository. ```bash git clone https://github.com/google-deepmind/long-form-factuality.git ``` Then navigate to the newly-created folder. ```bash cd long-form-factuality ``` Next, create a new Python 3.10+ environment using `conda`. ```bash conda create --name longfact python=3.10 ``` Activate the newly-created environment. ```bash conda activate longfact ``` All external package requirements are listed in `requirements.txt`. To install all packages, and run the following command. ```bash pip install -r requirements.txt ``` ## Usage ### LongFact The full prompt set for LongFact is available in the `longfact/` folder. See the README in `longfact/` for more details about the dataset. To run the data-generation pipeline that we used to generate LongFact, use the following command. Refer to the README in `data_creation/` for additional details about the data-generation pipeline. ```bash python -m data_creation.pipeline ``` ### SAFE Our full implementation of SAFE is located in `eval/safe/`. See the README in `eval/safe/` for more information about how SAFE works. To run the pipeline for evaluating SAFE against FActScore human annotations, use the following command. Refer to the README in `eval/` for additional details about this experiment. ```bash python -m eval.correlation_vs_factscore ``` ### Benchmarking models To benchmark OpenAI and Anthropic models, first add your API keys to `common/shared_config.py` (see README in `common/` for more information; be sure not to publish these keys). To obtain model responses for a given prompt set, use the following command. Refer to the README in `main/` for additional details about our main experimentation pipeline. ```bash python -m main.pipeline ``` Next, to evaluate prompt-response pairs from our main experimentation pipeline using SAFE, use the following command, making sure to add the path to the `.json` file containing the prompt-response pairs to be evaluated to the `--result_path` argument. ```bash python -m eval.run_eval \ --result_path= ``` ## Unit Tests Each file in this directory has a corresponding unit test with the `_test` suffix (e.g., `file.py` would have `file_test.py` for unit tests). Run commands for individual tests are shown in the unit test files. To run all unit tests, use the following command. ```bash python -m unittest discover -s ./ -p "*_test.py" ``` ## Citing this work If you find our code useful, please cite our [paper](https://arxiv.org/abs/2403.18802): ```bibtex @misc{wei2024long, title={Long-form factuality in large language models}, author={Wei, Jerry and Yang, Chengrun and Song, Xinying and Lu, Yifeng and Hu, Nathan and Huang, Jie and Tran, Dustin and Peng, Daiyi and Liu, Ruibo and Huang, Da and Du, Cosmo and Le, Quoc V.}, year={2024}, url={https://arxiv.org/abs/2403.18802}, } ``` ## License and disclaimer Copyright 2024 DeepMind Technologies Limited All software is licensed under the Apache License, Version 2.0 (Apache 2.0); you may not use this file except in compliance with the Apache 2.0 license. You may obtain a copy of the Apache 2.0 license at: https://www.apache.org/licenses/LICENSE-2.0 All other materials are licensed under the Creative Commons Attribution 4.0 International License (CC-BY). You may obtain a copy of the CC-BY license at: https://creativecommons.org/licenses/by/4.0/legalcode Unless required by applicable law or agreed to in writing, all software and materials distributed here under the Apache 2.0 or CC-BY licenses are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the licenses for the specific language governing permissions and limitations under those licenses. This is not an official Google product. ", Assign "at most 3 tags" to the expected json: {"id":"8992","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"