Test results – interpretation and configuration

Result status overview for single tests

Based on the combination of functions used in the Test logic, the tests can result in several different result statuses.

Name	Reason	Realized Score
Passed	All included assert() and warn() functions passed	Sum of all assert() and warn() functions' scores
Failed	Condition of at least one assert() function failed	0
Warning	All included assert() functions passed but at least one warn() function failed	Sum of all passing assert() and warn() functions' scores
Info	Only message() or table() function used (no assert() or warn() function)	0
Unresolved	No main Waaila function used in the evaluation	0

An additional result status, unspecified in the potential outputs for the test types, is Error. A tests results in an Error if the evaluation of the test fails on some event because the Query logic is not valid or there is some error in the code of the Test logic.

Score and maximum score

Interpretation

Each test has a Max Score defined in the configuration of the test which is the maximum score the test can reach. The Max Score serves for the evaluation of the performance of the tests for a given dataset or depot. While for a single test the most informative value of the result lies in the result status and potential displayed messages or tables, when comparing results of multiple tests, you want to have an overview if the checks that have failed are critical or only some minor bugs. This can be achieved by setting the Max Score of a critical check to a very high number, whereas for minor checks you can set the Max Score to a lower number and potentially use the warn() function instead of the assert() function. When a condition is satisfied, the realized Score for given function is equal to the Max Score and is added to the total Score of the results. When the condition fails, the realized Score for given function is zero (and so zero is added to the total Score). To summarize, the total realized Score is 0 if the test has status Failed, Info or Unresolved. For test with status Passed or Warning the total realized score is the sum of Max Scores for all assert() and warn() functions with satisfied conditions.

When you have results for a group of tests, you receive the sum of all realized scores and sum of all Max Scores. From these two values the performance value is calculated as the division of the total realized score and the total Max Score. For example, if you have three tests where each for simplification has Max Score equal to 100, the total Max Score is 300. When two of these tests result in Passed status (score 100) and one in Failed status (score 0), the total realized score is 200. Thus, the performance is 200/300 = 66 %.

How to set score

In general, tests with at least one assert() or warn() function should have a positive Max Score, while tests which have only informative outputs (using the message() or table() functions) should have zero Max score. When you have a single assert() or warn() function, the Max Score should be equal to the score you input into that function as the third parameter. If you have multiple functions, the Max Score should correspond to the maximum possible achievable sum of all scores in all assert() and warn() functions. This means that if you have them on the same level, one below each other, the Max Score should be the sum of all scores. For this example, the Max Score should be equal to 150:

waaila.assert(total > 0, 100).break
waaila.assert(furtherCheck, 50).break

If you have if-else logic within your test, the Max Score should be the maximum of sums of scores within each possible combination of condition results. For a single if-else condition with one assert() function for each option, this means the Max Score should be equal to the maximum of the two scores (for this example 100):

if (typeof visits[0] === undefined) {
  waaila.assert(total > 0, 100).break
} else {
  waaila.assert(furtherCheck, 50).break
}

In case of more assert() functions, such as below, you need to sum all scores within each if condition. In the following example, the Max Score should be equal to 120, which is the maximum from 100 and sum of 50 and 70:

if (typeof visits[0] === undefined) {
  waaila.assert(total > 0, 100).break
} else {
  waaila.assert(furtherCheck, 50).break
  waaila.assert(furtherCheck2, 70).break
}