Introduction

This tutorial describes how to launch automated tests and profile via Gauntlet by triggering in-game events, executing game commands and generating detailed performance charts / benchmarking your game.

You will need a source build of the engine, preferably 4.23 or above. It will also work on 4.22 with a few minor tweaks, but not sure about earlier versions since I only have 4.22.

Some of the instructions are taken from the UE4 documentation site, but there are several gotchas not mentioned there.

Performance Report Tool

First, let’s build PerfReportTool, the program responsible for generating those nice performance charts.
The tool lives under the CSVTools package which is situated at Engine\Source\Programs\CSVTools

You’re not going to see the CSVTools folder if you’re on 4.22, since it only got added in 4.23.
Don’t panic though, it’s super easy to pull in: all you have to do is grab it from the engine’s master branch and copy the directory under Engine\Source\Programs

Inside you should see a CSVTools.sln, go ahead and open it.
Select Release from the configuration box and click on Build -> Rebuild Solution
This should put the binaries under Engine\Binaries\DotNET\CsvTools

With this out of the way, you can now run your game and generate some test charts!

Start your game and once it’s running, bring up the console and execute this command:

CsvProfile Start

Leave it running for a few seconds, then do:

CsvProfile Stop

This should genereate a CSV file under your game’s Saved\Profiling directory, with the file being named Profile(<date>).csv

Now let’s run PerfReportTool on it!

Go to the Engine\Binaries\DotNET\CsvTools folder and run this in a command prompt or powershell window:

PerfreportTool.exe -csv <path_to_your_csv>

For me, the command is:

PerfreportTool.exe c:\Work\Horu\Horu\Saved\Cooked\WindowsNoEditor\Horu\Saved\Profiling\CSV\Profile(20200515_002552).csv

This should create a Profile(<date>).html file with a bunch of nice graphs inside!

With the profiler out of the way, let’s make everything automated! The idea is to launch your game via Gauntlet and run a simple test that will trigger some in-game events, profile for a couple of seconds, call PerfReportTool to generate a report and copy the report out into a custom reports directory.

Automation project

The first thing to do is create a new C# project in Visual Studio. The project’s type should be Class Library (.NET Framework) Visual C#. It’s important that you pick .NET Framework and not .NET Standard or .NET Core! If you’re not seeing .NET Framework then you’ll need to install the .NET desktop development module first.

  • Project name: YourGameTests.Automation
    I’ve named mine HoruTests.Automation
  • Location: <YourUE4SourceBuild>\Engine\Source\Programs
    Mine is at d:\Work\UE4Source_422\Engine\Source\Programs
  • Solution: Create new solution
  • Place solution and project in the same directory: checked
  • Framework: .NET Framework 4.6.2, though more recent frameworks might also work

Close Visual Studio and edit the project’s .csproj file.
The first line of the file is typically this:

<Project Sdk="Microsoft.NET.Sdk">

Change it to these two lines instead:

<?xml version="1.0" encoding="utf-8"?>
<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="Current" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

Save the file and open your project’s solution file in VS as you normally would.

Find Class1.cs in the solution explorer and Rename it to YourGameTestScript.cs for me it’s HoruTestScript.cs. Paste this in, more on how it works later:

using System;

using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
using System.IO.Compression;
using Gauntlet;
using EpicGame;
using System.Diagnostics;

namespace UE4Game
{
	/// <summary>
	/// Runs Gauntlet automated tests.
	/// </summary>
	public class MyFirstTest : DefaultTest
	{
		//TODO: replace these!
		private string PerfReportToolPath = "d:/Work/UE4Source_422/Engine/Binaries/DotNET/CsvTools/PerfreportTool.exe";
		private string GauntletController = "YourGameGauntletController";

		public MyFirstTest(Gauntlet.UnrealTestContext InContext)
			: base(InContext)
		{
		}

		public override UE4TestConfig GetConfiguration()
		{
			UE4TestConfig Config = base.GetConfiguration();
			UnrealTestRole ClientRole = Config.RequireRole(UnrealTargetRole.Client);
			ClientRole.Controllers.Add(GauntletController);
			Config.MaxDuration = 10 * 60; // 10 minutes: this is a time limit, not the time the tests will take
			return Config;
		}

		public override void TickTest()
		{
			base.TickTest();
		}

		public override void CreateReport(TestResult Result, UnrealTestContext Contex, UnrealBuildSource Build, IEnumerable<UnrealRoleArtifacts> Artifacts, string ArtifactPath)
		{
			UnrealRoleArtifacts ClientArtifacts = Artifacts.Where(A => A.SessionRole.RoleType == UnrealTargetRole.Client).FirstOrDefault();

			var SnapshotSummary = new UnrealSnapshotSummary<UnrealHealthSnapshot>(ClientArtifacts.AppInstance.StdOut);

			Log.Info("My First Performance Report");
			Log.Info(SnapshotSummary.ToString());

			base.CreateReport(Result, Contex, Build, Artifacts, ArtifactPath);
		}

		public override void SaveArtifacts_DEPRECATED(string OutputPath)
		{
			string uploadDir = Globals.Params.ParseValue("uploaddir", "");

			Log.Info("====== Upload directory: " + uploadDir);

			if (uploadDir.Count() > 0 && Directory.CreateDirectory(uploadDir).Exists)
			{
				string artifactDir = TestInstance.ClientApps[0].ArtifactPath;
				string profilingDir = Path.Combine(artifactDir, "Profiling");
				string csvDir = Path.Combine(profilingDir, "CSV");
				string targetDir = Path.Combine(uploadDir, "CSV");

				if (!Directory.Exists(targetDir))
					Directory.CreateDirectory(targetDir);

				string[] csvFiles = Directory.GetFiles(csvDir);
				foreach (string csvFile in csvFiles)
				{
					string targetCSVFile = Path.Combine(targetDir, Path.GetFileName(csvFile));
					File.Copy(csvFile, targetCSVFile);

					if (!File.Exists(PerfReportToolPath))
					{
						Log.Error("Can't find PerfReportTool.exe at " + PerfReportToolPath, ", aborting!");
						break;
					}

					ProcessStartInfo startInfo = new ProcessStartInfo();
					startInfo.FileName = PerfReportToolPath;
					startInfo.Arguments = "-csv ";
					startInfo.Arguments += targetCSVFile;
					startInfo.Arguments += " -o ";
					startInfo.Arguments += uploadDir;

					try
					{
						using (Process exeProcess = Process.Start(startInfo))
						{
							exeProcess.WaitForExit();
						}
					}
					catch
					{
						Log.Error("Error running PerfReportTool.exe, aborting!");
					}
				}
			}
			else
				Log.Error("No UploadDir specified, not copying performance report! Set one with -uploaddir=c:/path/to/dir");
		}
	}
}

Look for the line

//TODO: replace these!

and modify the PerfReportToolPath and GauntletController variables. GauntletController is initially set to YourGameGauntletController which is the name I recommend you use to get everything going a couple of paragraphs below.

Now click on Build -> Configuration Manager…

Rename the Release configuration to Development by clicking on the two dropdowns and selecting Edit… then Rename.

Next, right click your project and select Properties.

Go to the Build tab and enter this for Output path:

..\..\..\Binaries\DotNET\AutomationScripts\

Close Visual Studio, go to your engine’s root directory and run GenerateProjectFiles.bat

If using Visual Studio 2019, run GenerateProjectFiles.bat -2019

Now open UE4.sln in VS: your project should be under UE4 -> Programs -> Automation

Right click on your project to add some references via Add -> Reference

Click on the Projects tab on the left and check these three projects:

  • AutomationUtils.Automation
  • Gauntlet.Automation
  • UnrealBuildTool

Source: https://docs.unrealengine.com/en-US/Programming/BuildTools/AutomationTool/HowTo/AddingAutomationProjects/index.html

Now you’ll need to pull in Gauntlet as a dependency in your game project.

Edit your game’s .uproject file and add this under Plugins:

		{
			"Name": "Gauntlet",
			"Enabled": true
		}

Next, open YourGame.build.cs and put this in:

PrivateDependencyModuleNames.AddRange(new string[] { "Gauntlet" });

Now create two new source files in your game, let’s call them YourGameGauntletController.h and YourGameGauntletController.cpp. You can put them anywhere you want under the game’s Source folder.

 

YourGameGauntletController.h:

#pragma once

#include "GauntletTestController.h"
#include "YourGameGauntletController.generated.h"

UCLASS()
class YOURGAME_API UYourGameGauntletController : public UGauntletTestController
{
	GENERATED_BODY()

private:
	// Time to wait after game start before doing anything.
	const float SpinUpTime = 3.f;

	// Time to run the profiler for.
	const float ProfilingTime = 7.f;

	UFUNCTION()
	void StartTesting();

	void StartProfiling();

	UFUNCTION()
	void StopProfiling();

	void StopTesting();

protected:
	virtual void OnInit() override;
	virtual void OnTick(float DeltaTime) override;
};

 

YourGameGauntletController.cpp:

#include "YourGameGauntletController.h"
#include "ProfilingDebugging/CsvProfiler.h"
#include "TimerManager.h"
#include "Engine/World.h"
#include "Async/Async.h"

void UYourGameGauntletController::OnInit()
{
	UE_LOG(LogGauntlet, Display, TEXT("YourGameGauntletController started"));

	FTimerHandle dummy;
	GetWorld()->GetTimerManager().SetTimer(dummy, this, &UYourGameGauntletController::StartTesting, SpinUpTime, false);
}

void UYourGameGauntletController::StartTesting()
{
	//TODO: this is where you put your custom game code that should be run before profiling starts

	StartProfiling();
}

void UYourGameGauntletController::StartProfiling()
{
	FCsvProfiler::Get()->BeginCapture();

	// set a timer for when profiling should end
	FTimerHandle dummy;
	GetWorld()->GetTimerManager().SetTimer(dummy, this, &UYourGameGauntletController::StopProfiling, ProfilingTime, false);
}

void UYourGameGauntletController::StopProfiling()
{
	UE_LOG(LogGauntlet, Display, TEXT("Stopping the profiler"));

	TSharedFuture<FString> future = FCsvProfiler::Get()->EndCapture();

	// launch an async task that polls the Future for completion
	// will in turn launch a task on the game thread once the CSV file is saved to disk
	AsyncTask(ENamedThreads::AnyThread, [this, future]()
		{
			while (!future.IsReady())
				FPlatformProcess::SleepNoStats(0);

			AsyncTask(ENamedThreads::GameThread, [this]()
				{
					StopTesting();
				}
			);
		}
	);
}

void UYourGameGauntletController::OnTick(float DeltaTime)
{
	//TODO: this is where you can put stuff that should happen on tick
}

void UYourGameGauntletController::StopTesting()
{
	UE_LOG(LogGauntlet, Display, TEXT("YourGameGauntletController stopped"));
	EndTest(0);
}

4.22 and Below

If you’re on 4.22 or below, you have to edit the Gauntlet.uplugin file since there’s a bug in it.

The file is located at Engine\Experimental\Gauntlet\Gauntlet.uplugin

Find this line:

"Type" : "Developer",

Change it to:

"Type" : "Runtime",

Gauntlet won’t be able to run without this modification.

Cook & Run

Now you’ll need to cook your game in order for Gauntlet to be able to use it. While there are several ways to doing so, I’ve found executing the following command works best:

RunUAT.bat BuildCookRun -project=<path_to_your_uproject_file> -platform=Win64 -configuration=Development -build -cook -pak -stage

In my case I’m using:

RunUAT.bat BuildCookRun -project=c:\Work\Horu\Horu\Horu.uproject -platform=Win64 -configuration=Development -build -cook -pak -stage

It’s important to include the entire path to your .uproject file in the -project argument.

This cooks your game and puts the cooked build under YourGame\Saved\StagedBuilds
Now let’s run the tests!

RunUAT.bat RunUnreal -project=<name_of_your_game> -platform=Win64 -configuration=Development -test=MyFirstTest -build=<path_to_your_game>\Saved\StagedBuilds -uploaddir=<path_to_your_game>\PerfTests

That’s a long command! There are a couple of things to watch out for:

  • -project=<name_of_your_game> just the name of your game, no paths or anything!
  • test=MyFirstTest this is the name of your test class in YourGameTestScript.cs
  • build=<path_to_your_game>\Saved\StagedBuilds this is where your cooked build resides, without the WindowsNoEditor sub-directory!
  • -uploaddir=<path_to_your_game>\PerfTests you can point this anywhere, doesn’t have to be inside your game folder

For me, the command is:

RunUAT.bat RunUnreal -project=Horu -platform=Win64 -configuration=Development -test=LargeFleetTest -build=c:\Work\Horu\Horu\Saved\StagedBuilds -uploaddir=c:\Work\Horu\Horu\PerfTests

This will start a single instance of your game, wait for a set amount of time, optionally execute any in-game functions you told it to and starts measuring performance for a couple of seconds. After that, it will copy the CSV file with the profiling data to the given upload directory and run PerfReportTool on it to generate charts from the performance metrics.

Source: https://docs.unrealengine.com/en-US/Programming/Automation/Gauntlet/RunningTests/index.html

Customization

To customize report generation, look into the command line options for PerfReportTool.
Just execute PerfReportTool.exe without any arguments.

For customizing what build to test, you can change the arguments of the RunUAT.bat’s RunUnreal command, e.g. -platform, -build, -configuration, etc.

RunUnreal does not come with a help menu but you can look at its source file for clues:
Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs

Search for “Globals.Params.ParseValue” via Ctrl+F in Visual Studio to get an idea of what the arguments are and what they do.

Non-English Locales

If you run the PerfReportTool on a computer that’s set to a locale other than English you might find that the graphs inside your reports are mangled. To fix this, you’ll have to modify CsvToSVG.cs
Add the following to the top of the file:

using System.Threading;

Next, find the class’s Run() method, which is around line 260 for me and tell C# to use the “invariant” culture:

void Run(string[] args)
{
	Thread.CurrentThread.CurrentCulture = CultureInfo.InvariantCulture;

This is necessary because lots of languages use comma as the decimal separator for floats which is not a valid separator in SVGs. I’ve created a pull request (PR) for this fix: https://github.com/EpicGames/UnrealEngine/pull/6960

Questions?

Ask in the comments below or join Horu’s discord and ask in the #offtopic channel!

https://discord.gg/HXv2xUF

Acknowledgements

Big thanks to Jack Knobel @jack_knobel from Beethoven and Dinosaur for pointing me to the metrics tools and Gauntlet and answering my annoying questions along the way, his help was indispensable. Check out their game! @theartfulescape

References

https://qiita.com/donbutsu17/items/cd17d500a9fed143e061
A great article on Gauntlet in Japanese that works well with Google Translate.

https://docs.unrealengine.com/en-US/Programming/Automation/Gauntlet/Overview/index.html
https://docs.unrealengine.com/en-US/Programming/Automation/Gauntlet/RunningTests/index.html
The official Gauntlet docs.

https://docs.unrealengine.com/en-US/Programming/BuildTools/AutomationTool/HowTo/AddingAutomationProjects/index.html
Covers how to add an automation project.

https://docs.unrealengine.com/en-US/Engine/Performance/CSVProfiler/index.html
https://docs.unrealengine.com/en-US/Engine/Performance/CSVToSVG/index.html
Doc pages for CSVProfiler and CSVToSVG.

If you are interested in our upcoming tutorials and articles, subscribe to the newsletter below:

Published
Views 1242

Comments (3)

  • donbutsu17
    05/16/2020 at 17:04 Reply
    Great examples and introductions! It helps me and anyone a lot. Actually it's important to change the .plugin you focus on to "runtime". This was fixed in 4.23. Thanks!
    • GlassBeaver
      05/16/2020 at 19:42 Reply
      Glad you liked it, couldn't have made it without your article!

Leave a Reply