Trendshift - Ask AI

base on Home of test harnesses used in LTT Labs MarkBench <img src="images/labs_icon.png" alt="Logo" width="100" height="56" /> <h1>MarkBench Test Harnesses</h1>  ## About The Project Welcome to the official MarkBench testing platform developed by the LTT Labs team. MarkBench serves as the orchestration and data collection framework, while the tests themselves form the core of this process. The tests featured in this repository are actively employed to generate the data showcased in LTT (Linus Tech Tips) videos. We've made the code available here, allowing anyone to execute the very same tests that we use. It's worth noting that you do not require MarkBench to execute the tests provided within this project. ### Project versions The versions of tests that are available here are taken from snapshots of our private working repository where we maintain and update existing tests as well as develop and add new tests. We are making the effort to provide new versions of our code to the public at least once a quarter (i.e. every three months). However, we may occasionally release versions more often than this, should we have changes that we feel are worth sharing sooner rather than later. Depending on the changes we have made during the time between release versions, the differences in versions may vary in the amount and significance of changes made.  ## Table of Contents - [Getting Started](#getting-started) - [A test and its harness](#a-test-and-its-harness) - [Harness Manifest](#harness-manifest) - [JSON Report](#json-report) - [Creating a test harness](#creating-a-test-harness) - [Tools in the toolbox](#tools-in-the-toolbox) - [Keras OCR](#keras-ocr) - [Keyboard and Mouse Input](#keyboard-and-mouse-input) - [License](#license) ## Getting Started Configuring your system to execute these tests is straightforward; you'll only need Python, Poetry, and git. However, it's important to note that some of the tests in this repository may necessitate additional services or specific applications to be installed. For instance, if you intend to run the game tests, you will need to possess a valid copy of the respective game title.  ### Prerequisites  #### Python 3.10+ Most of the test harnesses are written in Python, which you will need on your system. We use Python 3.11 on our test benches, but should work on versions since 3.10.  ##### Installation We recommend you install python from the [official downloads page](https://www.python.org/downloads/) and not the Windows Store.  #### Poetry This project uses [Poetry](https://python-poetry.org/docs/) for dependency management.  ##### Installation Open a powershell terminal and execute the following command to download and execute the install script. ```powershell (Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py - ``` After installation you will want to add poetry to the path. On Windows this path to add is `%APPDATA%\Python\Scripts`. Test that poetry is working by executing `poetry --version`, a version number should be returned, not an error.  ##### Downloading dependencies 1. Open a terminal in the root directory. 2. Execute `poetry install` Poetry installs dependencies into virtual environments. You can read more about [managing poetry environments here.](https://python-poetry.org/docs/managing-environments/). (<a href="#readme-top">back to top</a>)  ### Running your first test Once you've successfully installed Python and Poetry, it's time to kick off our initial test. We'll begin by launching MSI Kombustor, which serves as our primary choice for testing and exploring new MarkBench functionalities. MSI Kombustor provides an excellent starting point for acquainting yourself with our test harnesses, as it doesn't necessitate any additional automation tools from our toolkit. Let's take a look at the folder structure - msikombustor (dir) - run (dir) - manifest.yaml - msikombustor.py - README.md When we mention "test harness," we are specifically referring to the entire directory and its contents. Within this directory, we anticipate, at the very least, the presence of an executable file that follows the guideline of returning either 1 or 0 to signify the completion status. Additionally, the manifest.yaml serves as metadata that allows MarkBench to identify it; however, for now, we can disregard it. The run directory, on the other hand, is the designated location for all outputs, including log files or screenshots. While it's not an absolute requirement, it has been the prevailing convention thus far. 1. First [install MSI Kombustor](https://geeks3d.com/furmark/kombustor/) using the default install location and options. 2. Second open a Powershell terminal and navigate to the root of the msikombustor directory. <img src="images/run_your_first_test_image1.png" alt="Logo" width="500" height="180" /> 3. From this directory run the command ```powershell python .\msikombustor.py --test vkfurrytorus --resolution "1080,1920" -b true ``` Executing this command initiates MSI Kombustor in benchmark mode, specifically launching the (VK) FurMark-Donut test at a resolution of 1920 x 1080. After the benchmark run concludes, you'll find the log and any captured assets stored within the **msikombustor/run** directory. It's important to note that the arguments required for each harness may vary. To ensure smooth test execution, consult the README of each harness, which provides detailed instructions on any unique requirements for running that specific test. (<a href="#readme-top">back to top</a>) ## A test and its harness MarkBench has the capability to accommodate any test that can be carried out on a Windows system and concludes with a success code of 0 or a failure code of 1. For MarkBench to recognize a test harness as automatable, it must include a manifest.yaml file containing essential metadata about the harness. The test harness is responsible for: 1. Setup 2. Execution 3. Gathering of assets 4. Cleanup ### Harness Manifest In MarkBench, the manifest.yaml file serves as a configuration file containing metadata about a test, providing essential information for MarkBench to execute the test correctly. Example ```yaml friendly_name: "Blender Benchmark" executable: "blender.py" process_name: "blender.exe" disable_presentmon: true hidden: 0 output_dir: "run" options: - name: scene type: select values: [all, classroom, junkshop, monster] - name: version type: select values: ["3.6.0", "3.5.0", "3.4.0", "3.3.0"] - name: device type: select values: ["CPU", "GPU"] ``` (<a href="#readme-top">back to top</a>) ### JSON Report Every harness (optionally) will write out some results in JSON format to a file called report.json. The JSON contents are read and stored in the database by MarkBench.  #### Non Game Report Any test that isn't a game *should* include a report for MarkBench to upload to the database. This report if present requires a value for the following properties: - **test** string - indicates any subparameters. For example BlenderBenchmark has the tests (scenes) classroom, junkshop, and monster. - **version** string - applicable version of the test or program under test. This property is optional if no version is available. - **score** string - a scalar output from the test. This could be a benchmark score, or duration. - **unit** string - the unit of measurement of the scalar. If duration, this could be seconds or minutes. If the score is simply a score, than this property can be omitted or left empty. - **label** string - optional friendly name for the unit of measurement. For example "fps" vs "Frames Per Second". This property is mainly used to override an axis on a graph downstream. The JSON report can be a single object, or array of reports, indicating to MarkBench there is more than one result to record. ```json { "test": "FireStrike", "version": "4.5.1", "score": "16099", "unit": "", "label": "", "start_time": 1702514174861, "end_time": 1702514209166 } ```  #### Game Report Game reports don't require a report as the score is the FPS which is calculated downstream of MarkBench. It is helpful if the report does include things such as resolution, start time, and end time. ```json { "resolution": "1920x1080", "start_time": 1702514174861, "end_time": 1702514209166 } ``` > Note, start_time and end_time given in the report.json of a game test will be used as the markers of when to start measuring FPS and when to stop. (<a href="#readme-top">back to top</a>) ## Creating a test harness Let's create a harness for the test FurMark. ```python import os.path import sys DEFAULT_FURMARK_DIR = "C:\\Program Files (x86)\\Geeks3D\\Benchmarks\\FurMark" EXECUTABLE = "FurMark.exe" ABS_EXECUTABLE_PATH = os.path.join(DEFAULT_FURMARK_DIR, EXECUTABLE) if os.path.isfile(ABS_EXECUTABLE_PATH) is False: raise ValueError('No FurMark installation detected! Default installation expected to be present on the system.') # omit the first arg which is the script name args = sys.argv[1:] command = f'"{ABS_EXECUTABLE_PATH}" ' for arg in args: command += arg + ' ' command = command.rstrip() os.system(command) ``` This is a very simple harness which takes in the arguments passed from the commandline and then executes the `FurMark.exe` test. A test harness can vary wildly in complexity depending on the test the harness is implementing. A canned game benchmark might require use of libraries like PyAutoGui to navigate around a game menu, or edit registry to setup configuration. Harness entry points and any supporting files should live in a named directory in the root harness directory. (<a href="#readme-top">back to top</a>) ## Tools in the toolbox ### Keras OCR We employ a deployment of [Keras OCR](https://github.com/faustomorales/keras-ocr) integrated into an HTTP API to assist in navigating game menus. This service accepts an image and a designated target word, and in return, it provides the coordinates of the word's location within the image. If the word cannot be located, it returns a "false" response. For detailed instructions on setting up this Keras Service locally, please refer to our [Keras Service repository linked here](https://github.com/LTTLabsOSS/keras-ocr-service). > Please note that although a CUDA-capable GPU is not mandatory, it's worth mentioning that certain games may not function correctly due to slower response times when this hardware is absent. If Keras is taking images on the wrong monitor, the primary display can be modified by changing `monitor_1 = sct.monitors[2] # Identify the display to capture` in `keras_service.py` ### Keyboard and Mouse Input For keyboard and mouse input, we employ two distinct methods. The first method involves using Virtual Key Codes (VKs) with the deprecated Win32 functions mouse_event() and keybd_event(). The second method utilizes Send Input. Specifically, [PyAutoGui](https://pyautogui.readthedocs.io/en/latest/) implements the first approach, while [PyDirectInput](https://pypi.org/project/PyDirectInput/) implements the second. (<a href="#readme-top">back to top</a>) ## License Distributed under the GNU GENERAL PUBLIC LICENSE Version 3. See `LICENSE` for more information. (<a href="#readme-top">back to top</a>)", Assign "at most 3 tags" to the expected json: {"id":"2787","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"

AI prompts

AI prompts