base on Fast RISC-V-based scripting backend for game engines # RVScript RVScript is a game engine oriented scripting system backed by a [low latency RISC-V sandbox](https://github.com/fwsGonzo/libriscv). By using a fast virtual machine with low call overhead and memory usage, combined with modern programming techniques we can have a type-safe and memory-safe script that is able to call billions of functions within a limited frame budget. [](https://github.com/fwsGonzo/rvscript/actions/workflows/engine.yml) [](https://github.com/fwsGonzo/rvscript/actions/workflows/unittests.yml) For discussions & help, [visit Discord](https://discord.gg/n4GcXr66X5). ## Introduction This project aims to change how scripting is done in game engines. Lua, Luau and even LuaJIT have fairly substantial overheads when making function calls into the script, especially when many arguments are involved. The same is true for WebAssembly emulators that I have measured, eg. wasmtime. I have yet to find another low latency emulator, actually. As a result, script functions are considered expensive to call, regardless of how little or how much they do, especially when used from a game engine where there is a tight deadline every frame. That changes thinking and design in projects accordingly. This repository is an attempt at making game scripting low latency, so that even automation games where interactions between complex machinery requires billions of script function calls, can still be achieved in a timely manner. Further, the emulator has a binary translation mode where the latency is lowered even further. And this translation can be cross-compiled to end-user platforms (eg. Windows), where it can be used to the same effect, transparently to users. ## Demonstration This repository is built as a demonstration of how you could use advanced techniques to speed up and blur the lines between native and emulated modern C++. The main function is in [engine/src](engine/src/main.cpp). See also the [unit tests](/tests/). All the host-side code is in the engine folder, and is written as if it was running inside a tiny fictional game framework. The script programs are using modern C++20 using a GNU RISC-V compiler with RTTI and exceptions enabled, but optional. Several CRT functions have been implemented as system calls, and will have native performance. The example programs have some basic timers and threads, as well as some examples making calls between machines. ## Getting started Install cmake, git, and a new g++ or Clang for your distro. Run [setup.sh](/setup.sh) to make sure that libriscv and some OpenGL dependencies is initialized properly. If you don't have the Debian distro or equivalent packages, you can just use `distrobox create --image ubuntu:22.04` and build this project from that. Even if you have an old Ubuntu version you can still use distrobox. All the dependencies are listed in [setup.sh](/setup.sh). Then go into the engine folder and run: ```sh bash build.sh ``` There are also some benchmarks that can be performed with `BENCHMARK=1 ./build.sh`. The scripting system itself is its own CMake project, and it has no external dependencies outside of libriscv and strf. libriscv has no dependencies, and strf can be replaced with libfmt. Running the engine is only half the equation as you will also want to be able to modify the scripts themselves. To do that you need a RISC-V compiler. ## Using a RISC-V toolchain from system package This is the simplest option. ```sh sudo apt install g++-12-riscv64-linux-gnu cd engine ./build.sh ``` The build script will detect which version of the GNU toolchain you have installed. Any RISC-V GCC compiler version 10 and later should be compatible. The C-library that is used by this toolchain, glibc, will use its own POSIX multi-threading, and it will be required that it works in order for C++ exceptions to work. So, be careful with mixing microthread and C++ exceptions. ## Getting a newlib RISC-V compiler If you want to produce performant and nimble executables, use riscvXX-unknown-elf from the RISC-V GNU toolchain. You can install it like this: ```sh git clone https://github.com/riscv/riscv-gnu-toolchain.git cd riscv-gnu-toolchain git submodule update --depth 1 --init <install dependencies for GCC on your particular system here> ./configure --prefix=$HOME/riscv --with-arch=rv64g_zba_zbb_zbc_zbs --with-abi=lp64d make -j8 ``` Add `$HOME/riscv` to your PATH by adding `export PATH=$PATH:$HOME/riscv/bin` to your `~/.bashrc` file. After you have done this, close and reopen your terminals. You should be able to tab-complete `riscv64-unknown-elf-` now. This compiler will be preferred by the build script in the programs folder because it is more performant. Check out the [compiler detection script](/programs/detect_compiler.sh) for the selection process. ```sh $ riscv64-unknown-elf-g++ --version riscv64-unknown-elf-g++ (gc891d8dc23e) 13.2.0 ``` ## Building and running If you have installed any RISC-V compiler the rest should be simple: Run [build.sh](/engine/build.sh) in the engine folder. It will also automatically start building script programs from the [programs folder](/programs). The actual script programs are located in [the scripts folder](/engine/scripts). ```sh cd engine ./build.sh ``` If you want to select a specific compiler, you can edit [detect_compiler.sh](/programs/detect_compiler.sh) to make it prioritize another compiler. ## Live-debugging Running `DEBUG=1 ./build.sh` in the [programs](programs) folder will produce programs that are easy to debug with GDB. Run `DEBUG=1 ./build.sh` in the engine folder to enable remote debugging with GDB. The engine will listen for a remote debugger on each breakpoint in the code. It will also try to start GDB automatically and connect for you. Remote GDB is a little bit wonky and doesn't like microthreads much but for the most part it works well. Install gdb-multiarch from your distro packaging system: ```sh sudo apt install gdb-multiarch ``` Connecting manually: ``` gdb-multiarch myprogram.elf target remote :2159 ``` ## WSL2 support Follow this to install WSL2 on Windows 10: https://docs.microsoft.com/en-us/windows/wsl/install-win10 There is nothing different that you have to do on WSL2. Install dependencies for GCC, then clone and install the RISC-V toolchain like above. It will just work. ## Creating an API The general API to adding new functionality inside the VM is to add more system calls, or use dynamic calls. System calls can only capture a single pointer, require hard-coding a number (the system call number), and is invoked from inline assembly inside the guest. Performant, but clearly not very conducive to auto-generated APIs or preventing binding bugs. Dynamic calls are string names that when invoked from inside the script will call a std::function on the host side, outside of the script. An example: ```C++ Script::set_dynamic_call("void sys_lazy ()", [] (auto&) { strf::to(stdout)("I'm not doing much, tbh.\n"); }); ``` Will assign the function to the dynamic call "void sys_lazy ()", which is done to help minimize ABI mistakes like passing wrong argument types. To call this function from inside the script programs we have to amend [dynamic_calls.json](/programs/dynamic_calls.json), like so: ```json "lazy": "void sys_lazy ()" ``` The build system will see the JSON changed and rebuild some API files (see [generate.py](/programs/dyncalls/generate.py)), and it will expose the callable function `sys_lazy`: ```C++ sys_lazy(); ``` A slightly more complex example, where we take an integer as argument, and return an integer as the result: ```C++ Script::set_dynamic_call("object_id", [] (Script& script) { const auto [id] = script.args <int> (); strf::to(stdout)("Object ID: ", id, "\n"); script.machine().set_result(1234); }); ``` Or, let's take a struct by reference or pointer: ```C++ Script::set_dynamic_call("struct_by_ref", [] (Script& script) { struct Something { int a, b, c, d; }; const auto [s] = script.args <Something> (); strf::to(stdout)("Struct A: ", s.a, "\n"); }); ``` Also, let's take a `char* buffer, size_t length` pair as argument: ```C++ Script::set_dynamic_call("void sys_big_data (const char*, size_t)", [] (Script& script) { // A Buffer is a general-purpose container for fragmented virtual memory. // The <riscv::Buffer> consumes two registers (A0: pointer, A1: length). const auto [buffer] = script.args <riscv::Buffer> (); handle_buffer(buffer.to_string()); // Or, alternatively (also consumes two registers): const auto [view] = script.args <std::string_view> (); handle_buffer(view); }); ``` In this case we would add this to [dynamic_calls.json](/programs/dynamic_calls.json): ```json "big_data": "void sys_big_data (const char*, size_t)" ``` See also [memory.hpp](/ext/libriscv/lib/libriscv/memory.hpp) for a list of helper functions, each with a specific purpose. The helper functions exist to simplify string and memory operations. ## Other examples [Dynamic calls](/tests/basic.cpp) [Event callbacks](/tests/events.cpp) [Event loop unit test](/tests/event_loop.cpp) ## Development notice This repository is still under development and will change over time. There are no stable APIs currently. ", Assign "at most 3 tags" to the expected json: {"id":"6982","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"