base on Reverse Engineering: Decompiling Binary Code with Large Language Models <p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://github.com/albertan017/LLM4Decompile/blob/main/samples/logo-dark.png"> <img alt="LLM4Decompile" src="https://github.com/albertan017/LLM4Decompile/blob/main/samples/logo-light.png" width=55%> </picture> </p> <p align="left"> 📊&nbsp;<a href="#evaluation">Results</a> | 🤗&nbsp;<a href="#models">Models</a> | 🚀&nbsp;<a href="#quick-start">Quick Start</a> | 📚&nbsp;<a href="#humaneval-decompile">HumanEval-Decompile</a> | 📎&nbsp;<a href="#citation">Citation</a> | 📝&nbsp;<a href="https://arxiv.org/abs/2403.05286">Paper</a> | 🖥️&nbsp;<a href="https://colab.research.google.com/drive/1X5TuUKuNuksGJZz6Cc83KKI0ATBP9q7r?usp=sharing">Colab</a> | ▶️&nbsp;<a href="https://www.youtube.com/watch?v=x7knF3Z1yLk">YouTube</a> </p> Reverse Engineering: Decompiling Binary Code with Large Language Models [](https://trendshift.io/repositories/8664) ## Updates * [2024-10-17]: Release [decompile-ghidra-100k](https://huggingface.co/datasets/LLM4Binary/decompile-ghidra-100k), a subset of 100k training samples (25k per optimization level). We provide a [training script](https://github.com/albertan017/LLM4Decompile/blob/main/train/README.md) that runs in ~3.5 hours on a single A100 40G GPU. It achieves a 0.26 re-executability rate, with a total cost of under $20 for quick replication of LLM4Decompile. * [2024-09-26]: Update a [Colab notebook](https://colab.research.google.com/drive/1X5TuUKuNuksGJZz6Cc83KKI0ATBP9q7r?usp=sharing) to demonstrate the usage of the LLM4Decompile model, including examples for the LLM4Decompile-End and LLM4Decompile-Ref models. * [2024-09-23]: Release [LLM4Decompile-9B-v2](https://huggingface.co/LLM4Binary/llm4decompile-9b-v2), fine-tuned based on [Yi-Coder-9B](https://huggingface.co/01-ai/Yi-Coder-9B), achieved a re-executability rate of **0.6494** on the Decompile benchmark. * [2024-06-19]: Release [V2](https://huggingface.co/LLM4Binary/llm4decompile-6.7b-v2) series (LLM4Decompile-Ref). V2 (1.3B-22B), building upon **Ghidra**, are trained on 2 billion tokens to **refine** the decompiled pseudo-code from Ghidra. The 22B-V2 version outperforms the 6.7B-V1.5 by an additional 40.1%. Please check the [ghidra folder](https://github.com/albertan017/LLM4Decompile/tree/main/ghidra) for details. * [2024-05-13]: Release [V1.5](https://huggingface.co/LLM4Binary/llm4decompile-6.7b-v1.5) series (LLM4Decompile-End, directly decompile binary using LLM). V1.5 are trained with a larger dataset (15B tokens) and a maximum token **length of 4,096**, with remarkable performance (over **100% improvement**) compared to the previous model. * [2024-03-16]: Add [llm4decompile-6.7b-uo](https://huggingface.co/arise-sustech/llm4decompile-6.7b-uo) model which is trained without prior knowledge of the optimization levels (O0~O3), the average re-executability is around 0.219, performs the best in our models. ## About * **LLM4Decompile** is the pioneering open-source large language model dedicated to decompilation. Its current version supports decompiling Linux x86_64 binaries, ranging from GCC's O0 to O3 optimization levels, into human-readable C source code. Our team is committed to expanding this tool's capabilities, with ongoing efforts to incorporate a broader range of architectures and configurations. * **LLM4Decompile-End** focuses on decompiling the binary directly. **LLM4Decompile-Ref** refines the pseudo-code decompiled by Ghidra. ## Evaluation ### Framework <p align="center"> <img src="https://github.com/albertan017/LLM4Decompile/blob/main/samples/compile-decompile.png" alt="image" width="400" height="auto"> </p> During compilation, the Preprocessor processes the source code (SRC) to eliminate comments and expand macros or includes. The cleaned code is then forwarded to the Compiler, which converts it into assembly code (ASM). This ASM is transformed into binary code (0s and 1s) by the Assembler. The Linker finalizes the process by linking function calls to create an executable file. Decompilation, on the other hand, involves converting binary code back into a source file. LLMs, being trained on text, lack the ability to process binary data directly. Therefore, binaries must be disassembled by ```Objdump``` into assembly language (ASM) first. It should be noted that binary and disassembled ASM are equivalent, they can be interconverted, and thus we refer to them interchangeably. Finally, the loss is computed between the decompiled code and source code to guide the training. To assess the quality of the decompiled code (SRC'), it is tested for its functionality through test assertions (re-executability). ### Metrics * **Re-executability** evaluates whether the decompiled code can execute properly and pass all the predefined test cases. ### Benchmarks * **HumanEval-Decompile** A collection of 164 C functions that exclusively rely on **standard** C libraries. * **ExeBench** A collection of 2,621 functions drawn from **real** projects, each utilizing user-defined functions, structures, and macros. ### Results <p align="center"> <img src="https://github.com/albertan017/LLM4Decompile/blob/main/samples/results_end_final.png" alt="results" width="800" height="auto"> </p> <p align="center"> <img src="https://github.com/albertan017/LLM4Decompile/blob/main/samples/results_refine_final.png" alt="image" width="800" height="auto"> </p> ## Models Our LLM4Decompile includes models with sizes between 1.3 billion and 33 billion parameters, and we have made these models available on Hugging Face. | Model | Checkpoint | Size | Re-executability | Note | |-----------------------|-------------------------------------------------------------------|------|---------------------|----------------------| | **llm4decompile-1.3b-v1.5**| 🤗 [HF Link](https://huggingface.co/LLM4Binary/llm4decompile-1.3b-v1.5) | 1.3B | **27.3%** | Note 3 | | **llm4decompile-6.7b-v1.5**| 🤗 [HF Link](https://huggingface.co/LLM4Binary/llm4decompile-6.7b-v1.5) | 6.7B | **45.4%** | Note 3 | | **llm4decompile-1.3b-v2**| 🤗 [HF Link](https://huggingface.co/LLM4Binary/llm4decompile-1.3b-v2) | 1.3B | **46.0%** | Note 4 | | **llm4decompile-6.7b-v2**| 🤗 [HF Link](https://huggingface.co/LLM4Binary/llm4decompile-6.7b-v2) | 6.7B | **52.7%** | Note 4 | | **llm4decompile-9b-v2**| 🤗 [HF Link](https://huggingface.co/LLM4Binary/llm4decompile-9b-v2) | 9B | **64.9%** | Note 4 | | **llm4decompile-22b-v2**| 🤗 [HF Link](https://huggingface.co/LLM4Binary/llm4decompile-22b-v2) | 22B | **63.6%** | Note 4 | Note 3: V1.5 series are trained with a larger dataset (15B tokens) and a maximum token size of 4,096, with remarkable performance (over 100% improvement) compared to the previous model. Note 4: V2 series are built upon **Ghidra** and trained on 2 billion tokens to **refine** the decompiled pseudo-code from Ghidra. Check [ghidra folder](https://github.com/albertan017/LLM4Decompile/tree/main/ghidra) for details. ## Quick Start [](https://colab.research.google.com/drive/1X5TuUKuNuksGJZz6Cc83KKI0ATBP9q7r?usp=sharing) **Setup:** Please use the script below to install the necessary environment. ``` git clone https://github.com/albertan017/LLM4Decompile.git cd LLM4Decompile conda create -n 'llm4decompile' python=3.9 -y conda activate llm4decompile pip install -r requirements.txt ``` Here is an example of how to use our model (Revised for V1.5. For previous models, please check the corresponding model page at HF). Note: **Replace the "func0" with the function name you want to decompile**. **Preprocessing:** Compile the C code into binary, and disassemble the binary into assembly instructions. ```python import subprocess import os func_name = 'func0' OPT = ["O0", "O1", "O2", "O3"] fileName = 'samples/sample' #'path/to/file' for opt_state in OPT: output_file = fileName +'_' + opt_state input_file = fileName+'.c' compile_command = f'gcc -o {output_file}.o {input_file} -{opt_state} -lm'#compile the code with GCC on Linux subprocess.run(compile_command, shell=True, check=True) compile_command = f'objdump -d {output_file}.o > {output_file}.s'#disassemble the binary file into assembly instructions subprocess.run(compile_command, shell=True, check=True) input_asm = '' with open(output_file+'.s') as f:#asm file asm= f.read() if '<'+func_name+'>:' not in asm: #IMPORTANT replace func0 with the function name raise ValueError("compile fails") asm = '<'+func_name+'>:' + asm.split('<'+func_name+'>:')[-1].split('\n\n')[0] #IMPORTANT replace func0 with the function name asm_clean = "" asm_sp = asm.split("\n") for tmp in asm_sp: if len(tmp.split("\t"))<3 and '00' in tmp: continue idx = min( len(tmp.split("\t")) - 1, 2 ) tmp_asm = "\t".join(tmp.split("\t")[idx:]) # remove the binary code tmp_asm = tmp_asm.split("#")[0].strip() # remove the comments asm_clean += tmp_asm + "\n" input_asm = asm_clean.strip() before = f"# This is the assembly code:\n"#prompt after = "\n# What is the source code?\n"#prompt input_asm_prompt = before+input_asm.strip()+after with open(fileName +'_' + opt_state +'.asm','w',encoding='utf-8') as f: f.write(input_asm_prompt) ``` Assembly instructions should be in the format: <FUNCTION_NAME>:\nOPERATIONS\nOPERATIONS\n Typical assembly instructions may look like this: ``` <func0>: endbr64 lea (%rdi,%rsi,1),%eax retq ``` **Decompilation:** Use LLM4Decompile to translate the assembly instructions into C: ```python from transformers import AutoTokenizer, AutoModelForCausalLM import torch model_path = 'LLM4Binary/llm4decompile-6.7b-v1.5' # V1.5 Model tokenizer = AutoTokenizer.from_pretrained(model_path) model = AutoModelForCausalLM.from_pretrained(model_path,torch_dtype=torch.bfloat16).cuda() with open(fileName +'_' + OPT[0] +'.asm','r') as f:#optimization level O0 asm_func = f.read() inputs = tokenizer(asm_func, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=2048)### max length to 4096, max new tokens should be below the range c_func_decompile = tokenizer.decode(outputs[0][len(inputs[0]):-1]) with open(fileName +'.c','r') as f:#original file func = f.read() print(f'original function:\n{func}')# Note we only decompile one function, where the original file may contain multiple functions print(f'decompiled function:\n{c_func_decompile}') ``` ## HumanEval-Decompile Data are stored in ``llm4decompile/decompile-eval/decompile-eval-executable-gcc-obj.json``, using JSON list format. There are 164*4 (O0, O1, O2, O3) samples, each with five keys: * ``task_id``: indicates the ID of the problem. * ``type``: the optimization stage, is one of [O0, O1, O2, O3]. * ``c_func``: C solution for HumanEval problem. * ``c_test``: C test assertions. * ``input_asm_prompt``: assembly instructions with prompts, can be derived as in our [preprocessing example](https://github.com/albertan017/LLM4Decompile?tab=readme-ov-file#quick-start). Please check the [evaluation scripts](https://github.com/albertan017/LLM4Decompile/tree/main/evaluation). ## On Going * Larger training dataset with the cleaning process. (done:2024.05.13) * Support for popular languages/platforms and settings. * Support for executable binaries. (done:2024.05.13) * Integration with decompilation tools (e.g., Ghidra, Rizin) ## License This code repository is licensed under the MIT and DeepSeek License. ## Citation ``` @misc{tan2024llm4decompile, title={LLM4Decompile: Decompiling Binary Code with Large Language Models}, author={Hanzhuo Tan and Qi Luo and Jing Li and Yuqun Zhang}, year={2024}, eprint={2403.05286}, archivePrefix={arXiv}, primaryClass={cs.PL} } ``` ## Star History [](https://star-history.com/#albertan017/LLM4Decompile&Timeline) ", Assign "at most 3 tags" to the expected json: {"id":"8664","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"