Test Suite Specification#
Introduction#
This page shows you the details of test suite files.
Test Suite Specification#
The following sections describe the configuration and control inside of test suites.
Note
As per the JSON specification you can include comments in your test suite by using keys that are otherwise unused or nops. Any unused key is valid, but in the examples we show, the keyword commentX is typically used, where x is blank or a number indicating sequence.
Test Suite File Naming#
You must use the following naming specification for test file names:
ts_<name>.json
Where, <name> is any freeform test string that makes sense in your system and adheres to your environment’s file naming spec.
List of Tests Definition#
A test case definition is the fundamental definition used in the three sections that contain a list of tests – setup, testcases, and teardown. When listing tests, the following specification is applicable.
Keyword: tests#
"tests": [
{
"name": "<tc_name>",
"loops": [1:N],
"reboot": [true/false],
"macro_subs": {
"macro[0]": "<substitute_value>",
... ""
}
}
...
]
Where,
testscontains a list dict values for each test casenameis string that contains the test case nameloopsis an integer that contains the number of times this test case should execute.rebootboolean true/false indicating if Fuze™ Test should perform a reboot prior to executing the test casemacro_subsis a dict of key:value pairs indicating substitution values inside the test case for this specific execution. See Test Case Specification for details about macros.
Warning
All macro substitutions should be defined in either the test suite or a test case. If you define a macro substitution in both the test suite, Fuze™ Test will consider this ambiguous and it will choose the substitution from the test case.
Scope#
Fuze™ Test includes an automated “test content discovery” mechanism so that test suites can automatically be found and executed when they are placed in configured directory location.
The scoping section of the test suite configures how Fuze™ Test autodiscovers test content to be executed. The scope sections applies to the entire test suite.
Keyword: scope#
"scope": {
"key[0]": {
"include": [""],
"exclude": [""]
},
...
}
Where,
scopecontains a dict of key valueskey[0]is a dict that contains a list of includes and excludesincludeis a list of TBDexcludeis a list of TBD
Loops#
You can configure how many times the entire collection of test cases executes. The loops value applies to the entire test suite.
Note
If the test cases are fanned out across a parallel DUT farm, you cannot guarantee location or order of execution. Your test case and suite design must consider this.
Keyword: loops#
"loops": <value>
Where <value> is an integer specifying the number of loops for the entire test suite
Setup#
The setup section contains a list of test cases that execute on all target DUTs prior to executing the main section of test cases. These test cases execute in the order they are listed.
Note
The setup section may be used for parallel cross-device testing.
Keyword: setup#
"setup": {
"tests": [
{
"name": "<tc_name>",
"loops": [1:N],
"reboot": [true/false],
"macro_subs": {
"macro[0]": "",
... ""
}
}
...
]
}
Where,
testscontains a list of dict values for each test case. See list-of-tests for details on listing tests.
Testcases#
The test case section is the main list of test cases to be executed. If Fuze™ Test is configured to parallelize test cases across a DUT farm with > 1 DUT, this section defines the test cases that will be fanned out across.
Note
You cannot guarantee location or order of execution. Your test case and suite design must consider this.
Keyword: testcases#
"testcases": {
"tests": [
{
"name": "<tc_name>",
"loops": [1:N],
"reboot": [true/false],
"macro_subs": {
"macro[0]": "",
... ""
}
}
...
]
}
Where,
testscontains a list of dict values for each test case. See list-of-tests for details on listing tests.
Teardown#
The teardown section contains a list of test cases that execute on all target DUTs after executing the main section of test cases. These test cases execute in the order they are listed.
Note
The setup section may be used for parallel cross-device testing.
Keyword: teardown#
"teardown": {
"tests": [
{
"name": "<tc_name>",
"loops": [1:N],
"reboot": [true/false],
"macro_subs": {
"macro[0]": "",
... ""
}
}
...
]
}
Where,
testscontains a list of dict values for each test case. See list-of-tests for details on listing tests.
Exclude#
The exclude section provides a convenience feature that allows you to remove test cases from execution without deleting them from the sections above. This is particularly useful when you want to temporarily remove a test case from execution (because of failures, updates, etc) but will be reinstating it in the future.
All test cases listed in this section will be skipped from execution in the setup, teardown, and testcases sections.
Keyword: exclude#
"exclude": {
"tests": [
"tc_name[0]",
...
]
}
Where,
testscontains a list of test case names to skip
Per Test Case#
For each test case named in the suite, you may specify the following configuration and control parameters for each test case:
Number of loops
Reboot before running the test case (True or False)
Macro substitutions for this test case
Test Suite File Format#
{
"name": "",
"comment": "",
"scope": {
"key[0]": {
"include": [""],
"exclude": [""]
},
...
},
"loops": [1:N],
"setup": {
"tests": [
{
"name": "<tc_name>",
"loops": [1:N],
"reboot": [true/false],
"macro_subs": {
"macro[0]": "",
... ""
}
}
...
]
},
"testcases": [
{
"name": "tc_name",
"loops": [1:N],
"reboot": [true/false],
"macro_subs": {
"macro[0]": "",
... ""
}
}
...
],
"teardown": {
"tests": [
{
"name": "tc_name",
"loops": [1:N],
"reboot": [true/false],
"macro_subs": {
"macro[0]": "",
... ""
}
}
...
]
},
"exclude": {
"tests": [
"tc_name[0]", ...
]
}
}
Test Suite Example#
{
"name": "ts_bvt_basic_tests_contact",
"comment": "Execute smoke tests via PCSC-C (contact)",
"scope": {
"comment": "Test cases that can execute in contact mode using windows tools",
"sensor": {
"include": [ "kepler" ],
"exclude": []
},
"device": {
"include": [ "cs7" ],
"exclude": [ "-CL"]
},
"hosttoolsversion": {
"include": [ "windows" ],
"exclude": []
},
"programmer": {
"include": [],
"exclude": []
},
"se": {
"include": [ "slc38"],
"exclude": []
}
},
"loops": 1,
"setup": {
"comment": "Test cases that get run on all participating DUTs prior to running the main test cases",
"tests": [
{
"name": "tc_contactless_mode_disable_etc",
"loops": 1,
"reboot": false,
"macro_subs": {
"__Port__": "--port PCSC-C",
"__PowerLevel__": "",
"__Security__": "",
"__Reader__": "omnikey5422"
}
},
{
"comment" : "Ensure secure mode is cleared",
"name": "tc_secure_channel_disable_etc",
"loops": 1,
"reboot": false,
"macro_subs": {
"__Port__": "--port PCSC-C",
"__PowerLevel__": "",
"__Security__": "",
"__Param130-Default__": ""
}
}
]
},
"testcases": [
{
"name": "tc_check_params_209_210_not_zero",
"loops": 1,
"reboot": false,
"macro_subs": {
"__Port__": "--port PCSC-C",
"__PowerLevel__": "",
"__Security__": ""
}
},
{
"name": "tc_get_component_version_bvt",
"loops": 1,
"reboot": false,
"macro_subs": {
"__Port__": "--port PCSC-C",
"__PowerLevel__": "",
"__Security__": ""
}
},
{
"name": "tc_get_uid",
"loops": 1,
"reboot": false,
"macro_subs": {
"__Port__": "--port PCSC-C",
"__PowerLevel__": "",
"__Security__": ""
}
}
],
"teardown": {
"comment": "Test cases that get run on all participating DUTs after running the main test cases",
"tests": [
{
"comment" : "Ensure secure mode is cleared",
"name": "tc_secure_channel_disable_etc",
"loops": 1,
"reboot": false,
"macro_subs": {"__Port__": "--port PCSC-C", "__PowerLevel__": "", "__Security__": "", "__Param130-Default__": ""}
}
]
},
"exclude": {
"comment": "We skip these temporarily",
"tests": ["tc_check_params_209_210_not_zero"]
}
}