Test logic of the test

Structure of the Test logic

In the Test logic field of the Test editor window you specify how you want to evaluate the loaded data. The structure of the Test logic is the following, first you need to import the output of the Query logic under the name results and import the waaila instance which contains the evaluation and transformation functions defined for the purpose of Waaila application (the functions are described more in detail in their individual subsections below). Then you need to access the data within the query results. To access the first query result for a Google Analytics request with ID "sessions" you can write results ['sessions'][0].

Below is an example evaluation of Google Analytics data comparing the highest share of sessions of a hostname to a configurable threshold. This test and its construction along with the procedure to write the Query logic is specified in the Use case 1 (GA), so you can refer there for more detail. Shortly, the evaluation takes the output data, access the total and maximum amount of sessions and extracts the first hostname (hostnames are ordered from the highest number of sessions). Then the assert() function is called to ascertain that the share is higher than the specified threshold. If this condition is not true, the assert message is displayed and the result status is set to Failed.

(results, waaila) => {    
    // Test Configuration:
    const sessionsRatio = 0.98
    const sessions = waaila.functions.normalizeGaResult(results['sessions'][0]);
    const sessionsSummary = waaila.functions.summarizeGaResult(results['sessions'][0]);
    const totalSessions = + sessionsSummary['sessions']['total'];
    const maxSessions = + sessionsSummary['sessions']['maximum']
    const hostName = sessions[0]['hostname'];
    waaila.assert(maxSessions / totalSessions > sessionsRatio, 100)
        .fail.message('The share of the most visited hostname (' + hostName + ') on all sessions is not higher than ' + sessionsRatio*100 + ' %');

Test types and results statuses

Test type is an indicator of test’s potential outputs and of its importance. It is directly derived from the combination of the main Waaila functions used in the tests. There are five different types of tests. The following table summarizes the 5 types along with their definition and a list of potential outputs the test can result in.

Test typeDefinitionPotential outputs
Assertion + Warningwarn() and assert() functionsPassed, Failed, Warning
Assertionassert() function (but no warn() function)Passed, Failed
Warningwarn() function (but no assert() function)Passed, Warning
Infoonly message() or table() functionInfo
Emptyno functionUnresolved

The default test type is Empty. This means that the test does not contain any of the main Waaila functions. It can serve as a preparation of a query for some other test or for checking raw data in the console. The output is marked as Unresolved.

The simplest test type with some Waaila functions is an Info. A test is of type Info if it contains only the informative Waaila functions, one or more message() or table() function. The purpose of tests of this type is to provide the user with an overview of aggregate data or a list of specific examples that require human interpretation of the results. The user can determine based on the results whether any action is required. For example, you can output a list of GET parameters to know the parameters used in your URLs. This allows you to exclude some of them according to your objectives in order to reduce granularity of pages in Google Analytics reports. Additionally, an Info can be used as the first phase of a customizable test, such as a test to verify that no unwanted hostname is covered in the measurement. In a customizable test, you may first receive a list of hostnames that are measured and based on the list you add an assert() function which should fail if a hostname occurs that is not in the specified list. The output of a test with type info is called Info.

The most common type of a test is Assertion which is based on evaluation of at least one assert() function. As it does not include any warn() function, it can result in Passed or Failed result status. Such a test is typically applied for verification of a condition which if not satisfied, indicates a problem and needs to be noticed and acted upon. An example can be verifying uniqueness of transaction ID value. If there are duplicate transactions in your data, the same data will be evaluated more than once and will bias your measurement. The severity of the issue tested by an Assertion is indicated by the height of the Max Score in comparison to other tests.

Tests of type Warning work similarly as the Assertion but they are considered less critical. If you check a condition only to be informed and it does not (have to) indicate a problem, you want the test to issue only a Warning, not a Failed result. As an example of a Warning, you can check whether demographic data are measured. The output of a Warning can be either Passed (if all conditions are satisfied) or Warning (if at least one condition is not satisfied).

The difference in the use between an Assertion and a Warning is that the assert() function typically ends with .break so the test exits directly when a condition of the assert() function is not satisfied. Further difference to an Assertion is the fact that a Warning is marked differently in the result report which you can you to filter the results in report based on the output status.

The most complex type of a test is a Assertion with Warning. This type contains both assert() and warn() functions, that means that there is both more important and less important conditions to evaluate in the test. Typically, a combination of assert() and warn() functions is useful for first verifying that data is present using a warn() function, then using an assert() function evaluating whether there is some wrongly measured user or purchase or whether the aggregate values correspond to other measures.

A more specific example of an Assertion with Warning can be a test which checks the total number of hits against two thresholds. Free version of Google Analytics has a limit on number of hits it can collect. Ingesting more than this limit has impact on how your data are processed, stored and sampled. To test it, you can set the assert() function on a higher threshold. If you reach this threshold, the test ends as Failed and you should do some action. Additionally, you can also set additional warn() function for a lower threshold. If this threshold is reached, the test results in a Warning and you can prepare in advance. If not even the smaller threshold is reached, the test Passed.

Adding an informative Waaila function, such as message() or table() function, to an Assertion, a Warning or an Assertion with Warning does not change the type of the test or the output result. It only produces an informative message in additional to potential assert or warning message.