Test results – interpretation and configuration

Result status overview

Based on the combination of functions used in the Test logic, the tests can result in several different result statuses. The most common statuses are self-explanatory - Passed and Failed - but there may be other. Warning may indicate a less serious issue (e.g. missing data are not critical, or missing data may be explained by a delay). Tests that do not include any evaluation, only display tables or text outputs, are marked by status Info and tests that include no output at all (for example because they are not completed yet) are marked by status Unresolved.

An additional result status, not included in the overview table, is Error. A test results in an Error if the evaluation of the test fails on some event because the Query logic is not valid or there is some error in the code of the Test logic.

Name	Reason
Passed	All included assert() and warn() functions passed
Failed	Condition of at least one assert() function failed
Warning	All included (if any) assert() functions passed but at least one warn() function failed
Info	Only message() or table() function used (no assert() or warn() function)
Unresolved	No output of test

Interpretation example

Waaila has a built-in single-test template to allow easier creation of new tests. You can create it by following the process below from inside of any existing set of tests (if you need to create a set of tests, follow these instructions to create GA4 set )

1. Select button "+ ADD TEST".
2. Fill in the Test Name.
3. Select all the checkboxes next to the parts of test you want to have included in the template of the Test logic.
4. Confirm by clicking "Create".

When you run this test on your data, you will see the following output:

• a message stating 'The total amount of sessions is ' + the total amount of sessions for the last three days in the selected data
• a table showing the daily sessions in the last three days (if you are using Google Analytics data) or previous week (if you are using Piano Analytics)
• if there are at least 3000 sessions for last three days in total, the result status will be Passed (marked with a green tick), otherwise you will see a message that there are less than 3000 sessions
• if there are less than 3000 sessions in the last three days but there are some, the result status will be Warning (marked with an orange exclamation mark)
• if there are no sessions in the last three days, the status will be Failed (marked with a red x or cross)

You can adjust the threshold value for sessions from 3000 to a different positive number by changing the value on the line const sessionsWarningThreshold = 3000; and running the test again. Try setting it above the total sessions observed in last three days to see the warning status and messages.

The resulting score depends on the result of the test. If the test output is Passed, the score is 100, for a Warning it is 80 and if the test output is Failed because there were no sessions, the resulting score is 0.

Score and maximum score

Score interpretation

Each test has a Max Score defined in the configuration of the test which is the maximum score the test can reach. The Max Score serves for the evaluation of the performance of the tests for a whole dataset or depot. When comparing results of multiple tests, you want to have an overview if the checks that have failed are critical or only some minor bugs. This can be achieved by setting the Max Score of a critical check to a very high number, whereas for minor checks you can set the Max Score to a lower number (and potentially use the warn() function instead of the assert() function).

When a condition is satisfied, the realized Score for given function is equal to the Max Score and is added to the total Score of the results. When the condition fails, the realized Score for given function is zero (and nothing is added to the total Score). When you have results for a group of tests, you receive the sum of all realized scores and sum of all Max Scores. From these two values the performance value is calculated as the division of the total realized score and the total Max Score.

In the example below the set has 48 % performance because the sum of realized scores is 280 and sum of Max scores is 580 (280 / 580 ~ 0.48). Only two tests passed and so they have realized score equal to Max score (the first one has score 150 and the second one 80). There are two warnings with realized score 0. One test failed but it has realized score 50 which means some of the checks passed and some not. All other tests have realized score 0.

How to set score

In general, tests with at least one assert() or warn() function should have a positive Max Score, while tests which have only informative outputs (using the message() or table() functions) should have zero Max score. The scores of each test should correspond to their importance. For example in the test set in section above the test W4270 - Hostname configuration check has lower significance than the test 4250 - Session to user ratio but still higher than the tests resulting in a Warning (W4280 - Demographic data and W4290 - Content grouping overview).

When you have a single assert() or warn() function, the Max Score should be equal to the score you input into that function as the third parameter. If you have multiple functions, the Max Score should correspond to the maximum possible achievable sum of all scores in all assert() and warn() functions. This means that unless you use some loops or conditions, the Max Score should be the sum of all scores. For this example, the Max Score for the script below should be equal to 150.

waaila.assert(total > 0, 100).break
waaila.assert(total > check, 50).break

if (typeof visits[0] === undefined) {
  waaila.assert(total > 0, 100).break
} else {
  waaila.assert(furtherCheck, 50).break
}