base on Emulate guitar amps and pedals with deep learning, in Go. # Waveny Waveny is a Go library and command-line utility designed for emulating guitar amplifiers and pedals through deep learning. The project takes inspiration from [Neural Amp Modeler] (NAM) and has adapted significant components from related repositories into Go: * [neural-amp-modeler]: a Python project for model training and WAVE file processing (reamping), leveraging PyTorch and Lightning. * [NeuralAmpModelerCore]: the core DSP library, written in C++, suited to real-time plugin development. ## Development Status and Constraints Waveny is in the early stages of development, and as such, the public APIs and functionalities are subject to change. The current codebase includes placeholders, indicating ongoing development, and minimal documentation. The project ambitiously applies digital signal processing in Go. The `live` command aims to offer decent real-time processing on modern CPUs, but optimization is ongoing. Key technical constraints include: * Sole support for the WaveNet model. * Support limited to a 48kHz sample rate. * Requirement for WAVE files to be PCM 48kHz 24-bit mono, for training or reamping. * Training on CPU only. Future updates will address these limitations. ## Utilization Guide ### Command Line Interface The `waveny` command-line interface offers several commands to interact with the deep learning models for guitar amp emulation. Build it with `go build ./cmd/waveny`, or compile-and-run it with `go run ./cmd/waveny`. The single executable allows to run different sub-commands, in this form: ```shell waveny COMMAND [arguments...] ``` Run `waveny help` to see a list of available commands. Here is a recap: * `train`: train a new WaveNet model using SpaGO, producing both SpaGO and `.nam` models. * `process-spago`: process a WAVE file using a pre-trained WaveNet SpaGO model, loaded from a file in "native" format. * `process-rt`: process a WAVE file using the custom Waveny real-time-capable WaveNet model, loaded from a `.nam` model-data file. * `process-torch`: process a WAVE file using a WaveNet SpaGO model, loaded and converted from a pre-trained NAM PyTorch/Lightning checkpoint file. * `live`: process audio input in real-time using the custom Waveny WaveNet model, loaded from a `.nam` model-data file. It uses PortAudio for I/O. For detailed usage and arguments of each command, execute: ```shell waveny COMMAND -h ``` The following sections illustrate several use cases. #### Training To train a new WaveNet model from scratch, we first need to prepare some files. * A clean/unprocessed audio file (WAVE PCM 48kHz 24-bit mono). You are free to create or record your own. However, [neural-amp-modeler] Python project comes with a set of standardized files: see [this section](https://github.com/sdatkinson/neural-amp-modeler/blob/v0.7.3/README.md#standardized-reamping-files) from the README. We recommend to use v3.0.0. * A "reamped" target audio file (same format as above), that is, the audio signal from the previous file processed through the real amp or pedal that you want to emulate. * A JSON configuration that describes structure and shapes of the WaveNet model. A good starting point is this [wavenet.json](https://github.com/sdatkinson/neural-amp-modeler/blob/v0.7.3/bin/train/inputs/models/wavenet.json) file from NAM Python project. Clean and reamped files must have the same size and be perfectly aligned. Their coupled audio content will be split into a training set and a validation set: the splitting points can be specified with dedicated command line arguments. If you are using NAM standardized reamping file [v3_0_0.wav](https://drive.google.com/file/d/1Pgf8PdE0rKB1TD4TRPKbpNo1ByR3IOm9/view?usp=drive_link), then the default values are already a good fit. Please have a look at the output of `waveny train -h` to learn about additional arguments. Here is a minimal example: ```shell waveny train \ -config path/to/wavenet.json \ -input path/to/v3_0_0.wav \ -target path/to/v3_0_0_reamped.wav \ -out path/to/output-folder/ ``` A new sub-folder, named with the current date and time, is created under the specified output directory. Here, at the end of each training epoch, two checkpoint files are created: a SpaGo binary model file, with extension `.spago`, and an equivalent `.nam` data-model file. The latter is a JSON file in the so-called "exported" NAM format, compatible with Neural Amp Modeler projects and plugins. Loss values are shown for each epoch, and they also become part of the checkpoint file names, for convenience. You can stop the training manually at any moment with Ctrl+C. When you are satisfied with the accuracy, pick the best pair of model files from the output folder, and see further steps described below. #### Process an audio file with pre-trained models ##### Loading a SpaGO model If you trained your own model as described above, a `.spago` checkpoint file is created at the end of each training epoch. You can now use a pre-trained `.spago` model to process an audio file. Get or record a "clean" audio file (WAVE PCM 48kHz 24-bit mono), choose your preferred model, and run: ```shell waveny process-spago \ -input path/to/input.wav \ -output path/to/output.wav \ -model path/to/model.spago ``` This command creates the output WAVE file, processing your input with the given amp/pedal emulation model. ##### Loading a PyTorch model, running with SpaGO If you trained a WaveNet model with [neural-amp-modeler] Python project, you can use another command to load a PyTorch/Lightning model/checkpoint file, convert it into a corresponding SpaGO model (on-the-fly, in memory), and process an input file just like we did above. In this case, you also need the accompanying WaveNet JSON model-configuration file. For example: ```shell waveny process-torch \ -input path/to/input.wav \ -output path/to/output.wav \ -config path/to/config_model.json \ -model path/torch_model.ckpt ``` ##### Loading a `.nam` model, running with Waveny custom implementation There is yet another way to process an audio file. This time, we are going to provide a `.nam` data-model file. This is NAM "exported" format, most commonly used to share NAM models, and intended to work with existing NAM audio plugins. If you trained your own model as described some sections above, a `.nam` checkpoint file is created at the end of each training epoch. Furthermore, an amazing source for pre-trained models in this special format is [ToneHunt] website. Download your desired amp/pedal emulation model, and make sure it uses WaveNet architecture. Once you have a `.nam` model file, you can run: ```shell waveny process-rt \ -input path/to/input.wav \ -output path/to/output.wav \ -model path/to/model.nam ``` The "rt" suffix in the command name indicates that we are using a custom WaveNet DSP processor: this implementation is most suitable for real-time processing, a topic discussed in the next section. #### Process audio in real-time Pre-trained `.nam` models can also be used for real-time processing. Plug in your musical instrument, and run: ```shell waveny live -model path/to/model.nam ``` This command uses Waveny custom WaveNet implementation to process audio input in real-time. I/O is possible thanks to [PortAudio]. ### Library Integration Integrate Waveny as a Go module in your projects with: ```shell go get github.com/nlpodyssey/waveny ``` From now on, we will refer to the root package `github.com/nlpodyssey/waveny` as simply `waveny`. Currently, the library only implements the WaveNet deep learning network. Under `waveny/models` you will find two different model implementations. Package `waveny/models/spago/wavenet` implements the model with [SpaGO] machine learning library. A WaveNet SpaGO model can be trained from scratch - see `waveny/models/spago/wavenet/training` subpackage. The training process produces SpaGO model files (`.spago`), and equivalent models in NAM "exported" format (`.nam`). It's also possible to load a PyTorch model file, pre-trained with [neural-amp-modeler] Python project, and convert it into a SpaGO model - see `waveny/models/spago/wavenet/torchconv` subpackage. It uses [GoPickle] library to read torch models without the need to run Python. A pre-trained SpaGO model can be effectively used to process WAVE files (non-real-time reamping). It is less suitable for real-time processing, mostly due to memory allocations and usage of goroutines. For real-time use, we provide another custom implementation of WaveNet, in package `waveny/models/realtime/wavenet`. The real-time-capable model can load `.nam` files (the ones trained with Waveny, or NAM WaveNet models from sources like [ToneHunt]). This implementation takes advantage of a self-contained package for handling matrices and vectors, implemented in `waveny/models/realtime/mat`. Inspired by the original [NeuralAmpModelerCore] implementation, and the underlying [Eigen] library, it allows to minimize the amount of memory allocations, permitting a predictable execution time, suitable for real-time processing. Package `waveny/liveplay` implements real-time processing procedures, using [PortAudio] go bindings for I/O. Package `waveny/wave` provides utilities for reading and writing WAVE files. [SpaGO]: https://github.com/nlpodyssey/spago [GoPickle]: https://github.com/nlpodyssey/gopickle [ToneHunt]: https://tonehunt.org [Neural Amp Modeler]: https://www.neuralampmodeler.com [neural-amp-modeler]: https://github.com/sdatkinson/neural-amp-modeler [NeuralAmpModelerCore]: https://github.com/sdatkinson/NeuralAmpModelerCore [Eigen]: https://eigen.tuxfamily.org [PortAudio]: https://github.com/gordonklaus/portaudio ", Assign "at most 3 tags" to the expected json: {"id":"4849","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"