This tutorial describes how to launch automated tests and profile via Gauntlet by triggering in-game events, executing game commands and generating detailed performance charts / benchmarking your game.
You will need a source build of the engine, preferably 4.23 or above. It will also work on 4.22 with a few minor tweaks, but not sure about earlier versions since I only have 4.22.
Some of the instructions are taken from the UE4 documentation site, but there are several gotchas not mentioned there.
Performance Report Tool
First, let's build PerfReportTool, the program responsible for generating those nice performance charts.
The tool lives under the CSVTools package which is situated at Engine\Source\Programs\CSVTools
You're not going to see the CSVTools folder if you're on 4.22, since it only got added in 4.23.
Don't panic though, it's super easy to pull in: all you have to do is grab it from the engine's master branch and copy the directory under Engine\Source\Programs
Inside you should see a CSVTools.sln, go ahead and open it.
Select Release from the configuration box and click on Build -> Rebuild Solution
This should put the binaries under Engine\Binaries\DotNET\CsvTools
With this out of the way, you can now run your game and generate some test charts!
Start your game and once it's running, bring up the console and execute this command:
Leave it running for a few seconds, then do:
This should genereate a CSV file under your game's Saved\Profiling directory, with the file being named Profile(<date>).csv
Now let's run PerfReportTool on it!
Go to the Engine\Binaries\DotNET\CsvTools folder and run this in a command prompt or powershell window:
PerfreportTool.exe -csv <path_to_your_csv>
For me, the command is:
This should create a Profile(<date>).html file with a bunch of nice graphs inside!
With the profiler out of the way, let's make everything automated! The idea is to launch your game via Gauntlet and run a simple test that will trigger some in-game events, profile for a couple of seconds, call PerfReportTool to generate a report and copy the report out into a custom reports directory.
The first thing to do is create a new C# project in Visual Studio. The project's type should be Class Library (.NET Framework) Visual C#. It's important that you pick .NET Framework and not .NET Standard or .NET Core! If you're not seeing .NET Framework then you'll need to install the .NET desktop development module first.
- Project name: YourGameTests.Automation
I've named mine HoruTests.Automation
- Location: <YourUE4SourceBuild>\Engine\Source\Programs
Mine is at d:\Work\UE4Source_422\Engine\Source\Programs
- Solution: Create new solution
- Place solution and project in the same directory: checked
- Framework: .NET Framework 4.6.2, though more recent frameworks might also work
Close Visual Studio and edit the project's .csproj file.
The first line of the file is typically this:
Change it to these two lines instead:
Save the file and open your project's solution file in VS as you normally would.
Find Class1.cs in the solution explorer and Rename it to YourGameTestScript.cs for me it's HoruTestScript.cs. Paste this in, more on how it works later:
Look for the line
and modify the PerfReportToolPath and GauntletController variables. GauntletController is initially set to YourGameGauntletController which is the name I recommend you use to get everything going a couple of paragraphs below.
Now click on Build -> Configuration Manager…
Rename the Release configuration to Development by clicking on the two dropdowns and selecting Edit… then Rename.
Next, right click your project and select Properties.
Go to the Build tab and enter this for Output path:
Close Visual Studio, go to your engine's root directory and run GenerateProjectFiles.bat
If using Visual Studio 2019, run GenerateProjectFiles.bat -2019
Now open UE4.sln in VS: your project should be under UE4 -> Programs -> Automation
Right click on your project to add some references via Add -> Reference
Click on the Projects tab on the left and check these three projects:
Now you'll need to pull in Gauntlet as a dependency in your game project.
Edit your game's .uproject file and add this under Plugins:
Next, open YourGame.build.cs and put this in:
Now create two new source files in your game, let's call them YourGameGauntletController.h and YourGameGauntletController.cpp. You can put them anywhere you want under the game's Source folder.
4.22 and Below
If you're on 4.22 or below, you have to edit the Gauntlet.uplugin file since there's a bug in it.
The file is located at Engine\Experimental\Gauntlet\Gauntlet.uplugin
Find this line:
"Type" : "Developer",
Change it to:
"Type" : "Runtime",
Gauntlet won't be able to run without this modification.
Cook & Run
Now you'll need to cook your game in order for Gauntlet to be able to use it. While there are several ways to doing so, I've found executing the following command works best:
RunUAT.bat BuildCookRun -project=<path_to_your_uproject_file> -platform=Win64 -configuration=Development -build -cook -pak -stage
In my case I'm using:
RunUAT.bat BuildCookRun -project=c:\Work\Horu\Horu\Horu.uproject -platform=Win64 -configuration=Development -build -cook -pak -stage
It's important to include the entire path to your .uproject file in the -project argument.
This cooks your game and puts the cooked build under YourGame\Saved\StagedBuilds
Now let's run the tests!
RunUAT.bat RunUnreal -project=<name_of_your_game> -platform=Win64 -configuration=Development -test=MyFirstTest -build=<path_to_your_game>\Saved\StagedBuilds -uploaddir=<path_to_your_game>\PerfTests
That's a long command! There are a couple of things to watch out for:
- -project=<name_of_your_game> just the name of your game, no paths or anything!
- test=MyFirstTest this is the name of your test class in YourGameTestScript.cs
- build=<path_to_your_game>\Saved\StagedBuilds this is where your cooked build resides, without the WindowsNoEditor sub-directory!
- -uploaddir=<path_to_your_game>\PerfTests you can point this anywhere, doesn't have to be inside your game folder
For me, the command is:
RunUAT.bat RunUnreal -project=Horu -platform=Win64 -configuration=Development -test=LargeFleetTest -build=c:\Work\Horu\Horu\Saved\StagedBuilds -uploaddir=c:\Work\Horu\Horu\PerfTests
This will start a single instance of your game, wait for a set amount of time, optionally execute any in-game functions you told it to and starts measuring performance for a couple of seconds. After that, it will copy the CSV file with the profiling data to the given upload directory and run PerfReportTool on it to generate charts from the performance metrics.
To customize report generation, look into the command line options for PerfReportTool.
Just execute PerfReportTool.exe without any arguments.
For customizing what build to test, you can change the arguments of the RunUAT.bat's RunUnreal command, e.g. -platform, -build, -configuration, etc.
RunUnreal does not come with a help menu but you can look at its source file for clues:
Search for “Globals.Params.ParseValue” via Ctrl+F in Visual Studio to get an idea of what the arguments are and what they do.
If you run the PerfReportTool on a computer that's set to a locale other than English you might find that the graphs inside your reports are mangled. To fix this, you'll have to modify CsvToSVG.cs
Add the following to the top of the file:
Next, find the class's Run() method, which is around line 260 for me and tell C# to use the “invariant” culture:
This is necessary because lots of languages use comma as the decimal separator for floats which is not a valid separator in SVGs. I've created a pull request (PR) for this fix: https://github.com/EpicGames/UnrealEngine/pull/6960
Big thanks to Jack Knobel @jack_knobel from Beethoven and Dinosaur for pointing me to the metrics tools and Gauntlet and answering my annoying questions along the way, his help was indispensable. Check out their game! @theartfulescape
A great article on Gauntlet in Japanese that works well with Google Translate.
The official Gauntlet docs.
Covers how to add an automation project.
Doc pages for CSVProfiler and CSVToSVG.
If you are interested in upcoming tutorials and articles, subscribe to the newsletter below: