base on youtube video renderer # SwapTube This is the repository I use to render [my YouTube videos](https://www.youtube.com/@twoswap). SwapTube is built on FFMPEG, but most of the functionalities above the layer of video and audio encoding are custom-written. The project does not use any fancy graphics libraries, with a few exceptions for particular functionalities. ## Setup ### External Dependencies The following external dependencies are required for specific functionalities within the project. These dependencies must be installed if you want to use the related features. | Item | What functionality is it needed for? | Used Where? | Used How? | Sample Ubuntu Installation | |------------|---------|---------|----------------|--------------| | CMake | Everything | go.sh script | Compiles the project | `sudo apt install cmake` | | FFMPEG 5.0 or higher, and associated development libraries | Everything | audio_video folder | Encoding and processing video and audio streams | `sudo apt install ffmpeg libswscale-dev libavcodec-dev libavformat-dev libavdevice-dev libavutil-dev libavfilter-dev` Note: compiling ffmpeg from source, it will likely be compiled with support for extra features detected on your system, which are not baked into my CMake config. I suggest installing a precompiled binary. | | CUDA | Accelerating simulations and video rendering | Anything that references the CUDA src dir (most computationally expensive scenes) | Various | Hardware-dependent | | gnuplot | Debug plot generation | DebugPlot.h | Data dumped in out/ is rendered to a PNG | `sudo apt install gnuplot` | | GLM | Graphs and 3D Graphics | 3d_scene.cpp, Graph.cpp | Vectors and quaternions to represent and rotate objects in space | `sudo apt install libglm-dev` | | MicroTeX | In-Video LaTeX, LatexScene | visual_media.cpp | Converts LaTeX equations into SVG files for rendering | Instructions are here: https://github.com/NanoMichael/MicroTeX/ You should install MicroTeX in MicroTeX-master alongside the swaptube checkout. Instructions will be printed if not found. | | RSVG and GLib | In-Video LaTeX | visual_media.cpp | Loads and renders SVG files into pixel data | `sudo apt install librsvg2-dev libglib2.0-dev` | | Cairo | In-Video LaTeX | visual_media.cpp | Renders SVG files onto Cairo surfaces and converts them to pixel data | `sudo apt install libcairo2-dev` | | LibPNG | PNG scenes | visual_media.cpp | Reads PNG files and converts them to pixel data | `sudo apt install libpng-dev` | | nlohmann/json | Reading and writing json files in I/O | Connect 4 data structures, GraphScene | GraphScene can write graphs to disk in json, Connect 4 steady states and compute caches are read from json | `sudo apt install nlohmann-json3-dev` | ## Docker Setup For easy deployment with all dependencies included (except nvidia/cuda), see the [docker/README.md](docker/README.md) for containerized setup instructions. This is optional and community-made for Docker users. I (2swap) personally don't use or maintain it. # How to Run It When you have created a project file in `projects/yourprojectname.cpp`, you can compile and run the whole project by executing: ```bash ./go.sh yourprojectname 640 360 ``` Some example code and demos can be found in `src/Projects/Demos/`. How to run a demo (code run from project root): ```bash ./go.sh LoopingLambdaDemo 640 360 ``` Swaptube defaults to a framerate of 30 FPS and a sample rate of 48000 Hz. If you need to change these for whatever reason, they are specified in `go.sh` and `record_audios.py`. # Repository Structure ### Top-Level Files and Folders - **./src/**: Source folder structure is documented in the readme inside of it. - **./out/**: Contains the output files (videos, corresponding subtitle files, data tables, and gnuplots) generated by swaptube. - Each subfolder corresponds to a project, and under that project, each render is stored in a separate folder named by timestamp. - **./media/**: Stores input media files used by the project. This includes script recordings, generated LaTeX, source MP4s, and source PNGs. - You should not ever need to manually modify anything here, with the exception of placing source PNGs and MP4s. Audio should be recorded using `record_audios.py` after rendering your project. - `Some_Project/`: Put media for your project here. - `record_list.tsv`: This will be generated by the program after rendering your project, and is read by the `record_audios.py` script so that you can record your script easily in bulk. - **./build/**: Contains various files and directories created during the build process, such as CMake cache, object files, and build scripts, but most importantly, the compiled binary. Caches and miscellaneous data products may also be dumped here, for example discovered connect 4 steady states and graphs. - **record_audios.py**: Reads the record_list.tsv file and permits you to quickly record all of the audio files for your video script. - **go.sh**: The program entry point! # Design Philosophy ### Time Control Swaptube uses a 2-layer time organization system. At the highest level, the video is divided into Macroblocks, which can be thought of as atomic units of audio. Macroblocks are divided into Microblocks, which are represent atomic time units controlling visual transformations. Such division permits the user to define a video with an in-line script, such that SwapTube will do all time management and the user does not need to manually time each segment of video. Furthermore, this permits native transitions: since a transition occurs over either a Macroblock or Microblock, Swaptube knows the duration of time over which the transition occurs, and can manage that transition automatically through State. ##### Macroblocks There are a few types of macroblocks: FileBlocks, SilenceBlocks, GeneratedBlocks, etc. FileBlocks are defined by a filepath to an audio file inside the media folder. SilenceBlocks are defined by a duration in seconds, and GeneratedBlocks are defined by a buffered array of audio samples generated in the project file. A macroblock can be created using `yourscene.stage_macroblock(FileBlock("youraudio_no_file_extension"), 2);` which stages the macroblock to contain 2 microblocks. ##### Microblocks After a Macroblock has been staged with `n` microblocks, the project file will render each microblock by calling `yourscene.render_microblock();`. Be sure to call this function `n` times, or else SwapTube will failout. ### Smoketesting In order to ensure that BOTH your time control is defined correctly (the appropriate number of microblocks are rendered) and that the project file does not crash due to a runtime error in the project file definition WITHOUT potentially kicking off a multi-hour render, Swaptube has a `smoketest` feature. By default, smoketest is always run on any Swaptube run. Things that happen during smoketesting: - One frame per microblock is staged to be rendered, but not actually rendered - DataObjects are modified as normal - State transitions are performed as normal to test validity of state equation definitions - The record_list.tsv file is re-populated, so you can record your audio script after smoketesting without performing a full render. - Subtitles will be generated with incorrect timestamps reflecting one-frame-per-microblock timing. Things that do NOT happen during smoketesting: - No video or audio is encoded or rendered - Since nothing is rendered, occasional frames are not drawn to stdout - Video width, height, and framerate are ignored entirely except insofar as they affect State equations and DataObject modifications. You can run `./go.sh MyProjectName 640 360 -s`, using the -s flag to indicate "smoketest only". Using this flag merely skips the full render after the smoketest. In addition to smoketesting, there is an additional exposed boolean variable `FOR_REAL` which can be toggled to true or false in the project file, effectively enabling smoketest mode for sections of a true render. This allows you to, say, work on the last section of a video without having to re-render the beginning each time. ### Scenes, State, and Data The data structure that a single frame is rendered as a function of has three parts, roughly split up to differences in their nature: - **Scene**: The Scene is the object which is constructed by the user in the project file. It fundamentally defines **what** is rendered. For example, a MandelbrotScene is responsible for rendering Mandelbrot Sets. - **State**: State can be thought of as any numerical information used by the Scene to render a particular frame. This controls things such as the opacity of certain objects, or, following the Mandelbrot example, the zoom level of the Mandelbrot set. All scenes have a StateManager, and when the user whishes to modify the scene's state, they can do so by calling functions on the StateManager. Usually these will be `set` and `transition` function calls. Since State uniquely contains numerical information, swaptube will handle all the clean transitions of state. - **Data**: Data is the non-numerical stateful information which is remembered by the Scene. A good example is the LambdaScene, which draws a Tromp Lambda Diagram, and stores as data that particular lambda expression. Similarly, a GraphScene needs to statefully track a Graph (of nodes and edges). This type of information is non-numerical, and cannot be naively interpolated as a transition, so it must be kept in a DataObject with an interface defined between the Scene and DataObject. ### Guts of a Scene If there is one thing you should read first to get how Swaptube works, it is src/Scenes/Scene.cpp. This is the abstract base class for all scenes, and TODO I will explain more about how Scenes work here. ", Assign "at most 3 tags" to the expected json: {"id":"14674","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"