About Automated tests using SK


#1

Hey mad scientists!

I didn’t have enough time to use SK in it’s full potential as i’m currently making a project in C++ because of certain -academic- requirements. But i read somewhere in the page that i could use SK for making automated tests (functional tests).

I was wondering if you guys could be so kind to provide some examples about the subject. Or any hint of how could i do this. Is there an assert equal function? Or can i use SK and integrate it with the UE4 automated testing tool?

Thanks in advance!


#2

What we have done on past projects is to make a SmokeTest overlay that contains a SmokeTest Mind class that contains and runs a bunch of associated test scenarios.

We had this connected to our build system so that whenever anyone checked in any changes including code or assets, the build server would build the game, and then run it with the SmokeTest scenarios enabled. Only if it completed all the scenarios successfully would the changes be considered accepted and good.

You could start off with something that is activated by hand rather than completely automated.

Make an overlay and it should be the last overlay applied - just after your project overlay.

The SmokeTest class could have class constructor that starts off a series of scenarios to test things out.

To comment out the overlay, you would just add a minus to its name as is described in the docs at Overlay files.

Alternatively you could have it so that the smoke tests are run if run a particular Sk command via the SkookumIDE workbench or when you walk through a special trigger region or press a special hotkey. If it is set up as a SkookumIDE command you can even associate it to a hotkey in Visual Studio.

You can also call from C++ into SkookumScript based on a particular build setting or other events that you detect on the C++ side:

Ideally you might want to have it so that the SmokeTest stuff is triggered by a special command line argument that is sent to your game when it is run in smoke test mode - though you might not need that level of sophistication.

Also take a look at the commands in the Sk Debug class. All Debug commands are removed from the code for a shipping build of your project.

The Break.assert() or some of the other calls in there might be what you are looking for.

If there are additional commands that you think could help, just let us know.

I’m sure Sk could be integrated ingo the UE4 Automated testing tool though we haven’t looked into that yet. You could certainly give it a shot yourself if you like.

Hope this helps as a start. This is a big topic that deserves a lot of though and discussion.


I don’t know what your academic requirements are, though SkookumScript is designed to be integrated with C++ code and projects. Here is some info on how to do that if you are interested:


#3

I’ve taken a shot at this and it’s no different than adding :sk: to any other blueprint, you simply inherit from FunctionalTest. The only downside for pure blueprint projects is that FinishTest and EFunctionalTestResult aren’t visible to :sk: so you’ll probably want to make a base BP class that provides a wrapper for :sk: as well as a mock equivalent of EFunctionalTestResult to avoid writing the boilerplate for every test.

enum class EFunctionalTestResult : uint8
{
	/**
	 * When finishing a test if you use Default, you're not explicitly stating if the test passed or failed.
	 * Instead you're instead allowing any tested assertions to have decided that for you.  Even if you do
	 * explicitly log success, it can be overturned by errors that occurred during the test.
	 */
	Default,
	Invalid,
	Error,
	Running,
	Failed,
	Succeeded
};

Other than that, just throw a map in your Tests folder and drag out some of your functional test bp’s into it and it’ll show up in the UI. The :ue4: built-in stuff for automation is still bad in my opinion unless you’re working on the engine etc, I posted a pretty long plea to :ue4: 2 years ago but nothing has changed :frowning:

So I heartily recommend :sk: for testing (not just because the :ue4: automation stuff will make you want to throw in the towel and watch Grey’s Anatomy all day with a bowl of ice cream) because writing functional tests in :sk: is simple - get all actors of this class, call these functions, wait until this happens, check the result.


#4

Thanks both of you for the answers. I’m going to give it a try and post my results. For now, i’m starting to see that :sk: might be a better solution (because i can easily run tests in games that are already packaged).


#5

For those interested.
I tried using Debug.break and it’s way better than i expected. I’m not sure if “soft” breakpoints are a common feature in other languages but i love it!
What i did is that i created an Overlay “AutomatedTests”, created a “SmokeTests” class member (subclass of Mind), then i created some class methods for the tests and i’m just calling those methods using the Workbench. SmokeTests.mytest or SmokeTests.runalltests

My tests methods looks like this

//Do test
//...
//
if testfailed? [Debug.break("ErrorMessage")]

I’m new making functional tests btw so i’m not sure if i’m missing something. I only made some unit tests years ago and i barely remember about it lol.
PD: I’m only testing in the editor for now.


#6

Sounds great!

Here are some alternate ways that you can write your tests:

if test_failed? [Debug.break("ErrorMessage")]

// If you don't want to use brackets:
Debug.break("ErrorMessage") when test_failed?

// If trigger if something isn't true:
Debug.break("ErrorMessage") unless test_succeeded?

#7

Is there a reason you cannot just use Debug.assert(not testfailed?, "ErrorMessage")?


#8

Hi GreatGuru! Actually not. I’ve used Debug.assert() without an issue but i like how Debug.break() opens the test that failed inside the :sk: editor. I can then see the callstack to get more information and manually place some breakpoints to see what went wrong (in case there’s more code after Debug.break()).