Test logic of the test

Structure of the Test logic

In the Test logic field of the Test editor window you specify how you want to evaluate the loaded data. The structure of the Test logic is the following, first you need to import the output of the Query logic under the name results and import the waaila instance which contains the evaluation and transformation functions defined for the purpose of Waaila application (the functions are described more in detail in their individual subsections below). Then you need to access the data within the query results. To access the first query result for a Google Analytics request with ID "sessions" you can write results ['sessions'][0].

Below is an example evaluation of Google Analytics data comparing the highest share of sessions of a hostname to a configurable threshold. This test and its construction along with the procedure to write the Query logic is specified in the Use case 1 (GA), so you can refer there for more detail. Shortly, the evaluation takes the output data, access the total and maximum amount of sessions and extracts the first hostname (hostnames are ordered from the highest number of sessions). Then the assert() function is called to ascertain that the share is higher than the specified threshold. If this condition is not true, the assert message is displayed and the result status is set to Failed.

(results, waaila) => {    
    // Test Configuration:
    const sessionsRatio = 0.98
    
    const sessions = waaila.functions.normalizeGaResult(results['sessions'][0]);
    const sessionsSummary = waaila.functions.summarizeGaResult(results['sessions'][0]);
    const totalSessions = + sessionsSummary['sessions']['total'];
    const maxSessions = + sessionsSummary['sessions']['maximum']
    const hostName = sessions[0]['hostname'];
    waaila.assert(maxSessions / totalSessions > sessionsRatio, 100)
        .fail.message('The share of the most visited hostname (' + hostName + ') on all sessions is not higher than ' + sessionsRatio*100 + ' %');
}

Waaila functions

assert()

The assert() function is constructed for important checks where a failure signifies a serious problem, such as no sessions measured for yesterday or presence of completed orders with no products. The basic structure of the function without any messages or output tables is:

waaila.assert([passing condition], [score value]).break

When the condition in the function is not satisfied, assert() function fails (due to the inclusion of .break method) and the evaluation of the test ends. The result status is set to Failed. When the assert() function does not fail, that is when the condition is satisfied, the evaluation of the test continues. If there is no other assert() or warn() functions or if all other assert() and warn() functions do not fail as well, the result status is set to Passed.

If the assert() function does not fail, the score value is the score added to the total test score. The score is used for aggregate comparison of tests within one dataset or one depot. The realized score of tests is compared to the maximum score to calculate the performance of the data evaluation. The score needs to be a positive integer (zero score is not allowed).

Most often you also want to display a message showing what happened or display some data illustrating the situation. For example, if you test visits on largest hostname by calculating the hostname in the query and the test fails, you can display the name of the hostname in the assert message to verify it corresponds to your expectations. Moreover, if you compare to a threshold in the condition, you can output the value along with the threshold to see whether the threshold is missed only marginally or by a large degree.

For this purpose, you can specify a message or a table (or both) to be displayed in case of pass or failure (or both) by including further methods in a chain behind the main assert() function. These methods are optional. However, when you decide to include message or table in the chain, you need to specify if they should be outputted in case of pass or failure of the function's condition. You can do it by writing the .message() or .table() in the chain behind the .pass or behind .fail as you can see in the full example below. Just a note, you do not have to separate the code into new lines but it helps to improve legibility of the code.

waaila.assert([passing condition], [score value])
    .pass.message([string message in case condition passes]).table([normalized data displayed as table in case condition passes])
    .fail.message([string message in case condition fails]).table([normalized data displayed as table in case condition fails])
    .break
Note:
The table() method requires an appropriately structured JSON input which can be achieved by passing query results to normalizeGaResult() or normalizeAtiResult() function (based on the data source).
Waaila tip:
You can omit the .break method at the end of the chain if you do not want the evaluation to end right after the assert() function fails. This can be useful if you include in a test multiple conditions but you cannot combine them into a single condition for assert function because the output messages or tables differ between them and you need to display only output messages or tables for conditions that are not satisfied. An exemplar use of this would be checking for presence of several wrong values in the page path and wanting to know in case of failure which in particular are present. However, you could simplify such a test by combining all conditions and testing if any of the values is present and then outputing a table showing the observations containing the wrong values.

warn()

The warn() function has similar functionality to the assert() function but is intended for low risk tests or when something needs to be checked before the main evaluation. The basic structure of the function is:

waaila.warn([passing condition], [score value])

When the condition in the function is not satisfied, the result status is set to Warning. Unless the .break method is included in the end, the evaluation of the test continues. If there are other assert() or message() functions and all their conditions are satisfied, the status stays set to Warning. It can change only in case of assert() function failure (to Failed).

When the condition in the function is satisfied, the evaluation of the test continues as well and the result status is set to Passed. If there is some further assert() or warn() functions, the result status can change to Warning or Failed based on other functions' results.

The message and table outputs and the score value work in the same manner as for the assert() function which can be seen in the included example of the full set of options for the warn() function. The score needs to be a non-negative integer - contrary to the assert() function, zero score is allowed because the warn() function can be used for additional verifications that do not necessarily need to impact the quality of the data.

Note:
The table() method requires an appropriately structured JSON input which can be achieved by passing query results to normalizeGaResult() or normalizeAtiResult() function (based on the data source).
waaila.warn([passing condition], [score value])
    .pass.message([string message in case condition passes]).table([normalized data displayed as table in case condition passes])
    .fail.message([string message in case condition fails]).table([normalized data displayed as table in case condition fails])
    .break

Informative functions - message() and table()

The message() and table() can be included both separately and as a method for the assert() or warn() function. For inclusion as a method for assert() or warn() function, refer to the specific sections. Below is described how they can be included separately. When no assert() or warn() function included, the inclusion of table() or message() functions sets the type of the test and the result status to Info.

The message() function provides the opportunity to output some values from the data loaded by the query. The structure of the function is very simple:

waaila.message([string message to be displayed])

The table() function is an extended version of the message function. It allows you to display the data in the form of a table. This is very practical for an overview of the rows with the highest value, like top 10 hostnames. Alternatively, the table() function can be used to get an example of rows which do not satisfy a certain condition, so that you can decide whether it signifies a serious problem or only some minor misspecification in the data.

The structure of the function is the following:

waaila.table([normalized data input])

The table() function requires an appropriately structured JSON input which can be achieved by passing query results to normalizeGaResult() or normalizeAtiResult() function (based on the data source).

Waaila tip:
The message() or table() functions can be used in the process of test customization when you want to display for example the total number of hits or the share of the most visited hostname.

The table() function can be adjusted to display some of the columns color-coded to allow easier overview of the results. To display a column of a table color-coded, include a second parameter to the table() function as an array of objects for each column. Each object is defined for a unique column and includes a condition which if satisfied, colors given cell in green as an okay value, while otherwise the cells are colored in red as critical. The objects have the following structure:

  • 'column' = the name of the column which should be color-coded
  • 'condition' = condition to evaluate which consists of the type of the condition and the value for comparison
    • 'EQUAL': [value] (condition satisfied only if column is equal to given value)
    • 'GREATER_THAN': [value] (condition satisfied only if column is strictly greater than given value)
    • 'LESS_THAN': [value] (condition satisfied only if column is strictly less than given value)
Waaila tip:
When the simple conditions are not sufficient for the comparison you need to accomplish, you can easily construct indicator columns, such is the "isOK" column in the example code below and use it in combination with the 'EQUAL' condition. The example compares visits to a threshold of 100.000 where values below (or equal) are colored in red while values above are colored in green. Additionally, it compares device types to the array of allowed devices - rows with devices included in the array have green-colored cells, while other rows have red-colored cells.
const allowedDevicesArray = ['Desktop', 'Mobile Phone', 'Tablet'];
visits.forEach(function (row) {
    if (allowedDevicesArray.includes(row['Device - Type'])){
        row['isOK'] = 'OK'
    } else {
        row['isOK'] = 'NOK'
    }   
})
waaila.table(visits, 
             [{'column': 'visits', 'condition': {'GREATER_THAN' : 100000}},
              {'column': 'isOK', 'condition': {'EQUAL' : 'OK'}}]
            ); 

Google Analytics data transformation

The normalizeGaResult() function is constructed to normalize data from Google Analytics in order to simplify manipulation with the data and to prepare the data for transformation into a table (using the table() function). It makes the structure of the data more compact and thus clearer for extracting values from the data. The structure of results for both functions, normalizedResult and summarizedResult, is described more in detail in the section Structure of the results.

The input of the normalizeGaResult is the data from a single query of a request to Google Analytics. In general, the function has the following format: normalizeGaResult(results['<id-of-query>'][<position-of-query>], <position-of-dateRange>). The parameter specifying the position of dateRange takes default value 0, i.e. the first dateRange available. Therefore if you only have one dateRange specified in the query, this parameter can be omitted. For illustration, for the first query of a request with id equal to 'sessions' and only one dateRange, the function can be specified as normalizeGaResult(results['sessions'][0]). This creates an Array of data rows where each contains key-value pairs for all dimensions and metrics.

The function summarizeGaResult() allows you to extract summary of the data from the query output. The data are inputted in the same way as using the normalizeGaResult() function. The output of the function is an Object with summary. In the summary there is the sum of all values (called 'total'), maximum and minimum value (called 'maximum' and 'minimum') and the type (marked as 'type') for every metric included in the data.

AT Internet data transformation

The normalizeAtiResult() function is constructed to normalize data from AT Internet in order to simplify manipulation with the data and to prepare the data for transformation into a table (using the table() function). It makes the structure of the data more compact and thus clearer for extracting values from the data. The structure of results for both functions, normalizedResult and summarizedResult, is described more in detail in the section Structure of the results.

The input of the normalizeAtiResult is the data from a single query of a request. In general the function has the following format: normalizeAtiResult(results[<position-of-query>]). For illustration, for the first query of a request, the function can be specified as normalizeAtiResult(results[0]). This creates an Array of data rows where each contains key-value pairs for all dimensions and metrics.

The function summarizeAtiResult() allows you to calculate summary of the extracted data from the query output. The data are inputted in the same way as using the normalizeAtiResult() function. The output of the function is an Object with summary. In the summary there is the sum of all values (called 'total'), maximum and minimum value (called 'maximum' and 'minimum') and the type (marked as 'type') for every metric included in the data.

Methods for normalizedResult

There are several methods that can be applied on the normalized data once they are transformed into the unified structure of normalizedResult. These methods can be used to adjust the display of the results using the Waaila table() function or to complement some more difficult queries transformations and combinations. For example using sorting and selecting number of rows you can find the hostname with the highest number of sessions or compare last week with last four weeks of data.

In order to sort the data by values of some column, you can use the .order() method on the normalized data. You need to specify the name of the column to order by and the direction in which the column should be sorted (by specifying 'true' you are selecting the ascending manner, i.e. from smaller to largest, and by specifying 'false' you are selecting the descending manner). While you can sort the data already in the query to the data source, you can use the method after filtering or when you want to display the top rows first by one column and then by another column.

To select only first several rows, the normalized data has the method .head() where inside the parentheses you need to specify the number of rows from the top of the data that you want to select (the default is first 5 rows). This is very helpful when you construct the test using the whole data and then you want to display a few top rows as an example.

The method .select() allows you to select only a subset of columns to be included in the data. You specify the names of the columns inside square brackets. This is useful for displaying a table when you have many metrics included for the evaluation of the test but you only need to display a subset of the columns.

Below is an example where the data are loaded, transformed to unified structure and the following transformation is performed:

  1. select which columns to display (date and sessions)
  2. sort the data by sessions, descending
  3. keeps only first 10 rows of the sessions data
// for GA data
const sessions = normalizeGaResult(results['sessions'][0])
const sessionsTransformed = sessions.select(['date', 'sessions']).order(['sessions'], false).head(10)
waaila.table(sessionsTransformed)
// for ATI data
const visits = normalizeAtiResult(results[0])
const visitsTransformed = visits.select(['Date', 'Visits']).order(['Visits'], false).head(10)
waaila.table(visitsTransformed)

Test types and results statuses

Test type is an indicator of test’s potential outputs and of its importance. It is directly derived from the combination of the main Waaila functions used in the tests. There are five different types of tests. The following table summarizes the 5 types along with their definition and a list of potential outputs the test can result in.

Test typeDefinitionPotential outputs
Assertion + Warningwarn() and assert() functionsPassed, Failed, Warning
Assertionassert() function (but no warn() function)Passed, Failed
Warningwarn() function (but no assert() function)Passed, Warning
Infoonly message() or table() functionInfo
Emptyno functionUnresolved

The default test type is Empty. This means that the test does not contain any of the main Waaila functions. It can serve as a preparation of a query for some other test or for checking raw data in the console. The output is marked as Unresolved.

The simplest test type with some Waaila functions is an Info. A test is of type Info if it contains only the informative Waaila functions, one or more message() or table() function. The purpose of tests of this type is to provide the user with an overview of aggregate data or a list of specific examples that require human interpretation of the results. The user can determine based on the results whether any action is required. For example, you can output a list of GET parameters to know the parameters used in your URLs. This allows you to exclude some of them according to your objectives in order to reduce granularity of pages in Google Analytics reports. Additionally, an Info can be used as the first phase of a customizable test, such as a test to verify that no unwanted hostname is covered in the measurement. In a customizable test, you may first receive a list of hostnames that are measured and based on the list you add an assert() function which should fail if a hostname occurs that is not in the specified list. The output of a test with type info is called Info.

The most common type of a test is Assertion which is based on evaluation of at least one assert() function. As it does not include any warn() function, it can result in Passed or Failed result status. Such a test is typically applied for verification of a condition which if not satisfied, indicates a problem and needs to be noticed and acted upon. An example can be verifying uniqueness of transaction ID value. If there are duplicate transactions in your data, the same data will be evaluated more than once and will bias your measurement. The severity of the issue tested by an Assertion is indicated by the height of the Max Score in comparison to other tests.

Tests of type Warning work similarly as the Assertion but they are considered less critical. If you check a condition only to be informed and it does not (have to) indicate a problem, you want the test to issue only a Warning, not a Failed result. As an example of a Warning, you can check whether demographic data are measured. The output of a Warning can be either Passed (if all conditions are satisfied) or Warning (if at least one condition is not satisfied).

The difference in the use between an Assertion and a Warning is that the assert() function typically ends with .break so the test exits directly when a condition of the assert() function is not satisfied. Further difference to an Assertion is the fact that a Warning is marked differently in the result report which you can you to filter the results in report based on the output status.

The most complex type of a test is a Assertion with Warning. This type contains both assert() and warn() functions, that means that there is both more important and less important conditions to evaluate in the test. Typically, a combination of assert() and warn() functions is useful for first verifying that data is present using a warn() function, then using an assert() function evaluating whether there is some wrongly measured user or purchase or whether the aggregate values correspond to other measures.

A more specific example of an Assertion with Warning can be a test which checks the total number of hits against two thresholds. Free version of Google Analytics has a limit on number of hits it can collect. Ingesting more than this limit has impact on how your data are processed, stored and sampled. To test it, you can set the assert() function on a higher threshold. If you reach this threshold, the test ends as Failed and you should do some action. Additionally, you can also set additional warn() function for a lower threshold. If this threshold is reached, the test results in a Warning and you can prepare in advance. If not even the smaller threshold is reached, the test Passed.

Adding an informative Waaila function, such as message() or table() function, to an Assertion, a Warning or an Assertion with Warning does not change the type of the test or the output result. It only produces an informative message in additional to potential assert or warning message.