Description of BS ISO/IEC TR 13233:1995 1996

0.1 This Technical Report provides guidance for assessors and testing laboratories on the specific interpretation of the accreditation requirements applicable to testing (including validation of means of testing and test tools) in the field of Information Technology and Telecommunications (IT&T), specifically in relation to software and protocol testing services. This Technical Report does not apply to the accreditation of inspection, certification and quality assurance assessment activities.

0.2 However, ISO/IEC Guide 25 and any other applicable ISO/IEC Guides take precedence over the interpretation given in this Technical Report.

0.3 This Technical Report covers the use by accredited testing laboratories of services for the validation of means of testing (MOT) and test tools, and also applies to the possibility of accreditation of MOT and test tool validation services, because such a validation service is just a specialised form of software testing service.

NOTE — In many areas of IT&T, it may be impractical to require the use of accredited MOT and test tool validation services, both economically and given the state of the art in the particular area. It is important to recognise that the mere existence of an applicable accredited validation service does not mean that relevant accredited testing laboratories should be required to use it, as other suitable forms of MOT and test tool validation may exist. Other factors outside the scope of this Technical Report will determine if and when use of accredited MOT and test tool validation services might become a requirement.

0.4 The aim is that it should be generally applicable across the whole software and protocol testing area, whenever accreditation to ISO/IEC Guide 25 applies. However, it does not cover all the requirements of ISO/IEC Guide 25. Laboratories are reminded that, in order to obtain and maintain accreditation, they shall fully comply with ISO/IEC Guide 25. This Technical Report interprets the ISO/IEC Guide 25 requirements in this field; it does not in any way replace them. Furthermore, there may be other interpretations of ISO/IEC Guide 25 which are sector independent, maybe focusing on just one aspect of accreditation, in which case such generally applicable interpretations continue to apply, and are not replaced by this interpretation.

0.5 This interpretation applies to conformance testing and other types of objective testing of software. Specific guidance is provided for OSI, telecommunications protocols, product data exchange (as defined by ISO TC184), graphics, POSIX and compilers. The testing of physical properties of hardware is outside the scope of this interpretation, but may be covered elsewhere. Evaluation of systems and products, as in IT&T Security and Software Quality evaluation (ISO/IEC 9126), is also not included in the scope of this interpretation. Safety-critical software and general application software testing are also not included in this edition.

0.6 Specific text is given in this interpretation for conformance testing. However, the general interpretations given in this Technical Report are applicable to all types of objective testing, including measuring some objective aspects of performance (e.g. as in compiler testing for some programming languages) and types of testing that are particular to a single area within the IT&T field. Analysis by the test operator in order to produce the final result for a test case, in accordance with procedures that lead to objective results, is included in this interpretation.

NOTES

1. Normally, each individual test case in a test suite (set of test cases) will be designed to yield a test verdict, that is a statement of pass, fail or inconclusive.

2. Conformance testing involves testing the implementation against the conformance requirements specified in one or more standards (or other normative specifications). The standards against which implementations are tested for conformance will often be International Standards, although they may be ITU-T Recommendations, regional or national standards, or even a manufacturer's specification when the manufacturer is seeking independent confirmation that the implementation conforms.

3. The test cases to be used in conformance testing may also be standardized, but (in the fields of software and protocol testing) are usually distinct from the standards which specify the requirements to which implementations are supposed to conform.

4. Each test verdict should be made with respect to the purpose of the test case and the requirements of the relevant standard(s). Optionally, a particular test suite may specify various classes of pass, fail or inconclusive test verdict (e.g. fail class 1 : severe non-conformance; fail class 2: invalid behaviour but satisfied the test purpose), but this does not alter the general points about test verdicts.

Requirements in ISO/IEC Guide 25,IT&T Interpretations and Definitions		Guidance and Examples
1	Scope
	No IT&T specific interpretation is required for clause 1 of ISO/IEC Guide 25.	See clause 0 for the scope of this Technical Report. Note that this Technical Report applies to testing laboratories but not to calibration laboratories. The relevant laboratories, however, include validation laboratories that offer validation services for means of testing and/or test tools to be used by testing laboratories; in this case, the item to be validated is to be regarded as a system or implementation under test.
2	References
	No IT&T specific interpretation is required for the references of ISO/IEC Guide 25.
	The following standards contain provisions which, through reference in this text, constitute provisions of this Technical Report. At the time of publication, the editions indicated were valid. All standards are subject to revision, and parties to agreements based on this Technical Report are encouraged to investigate the possibility of applying the most recent editions of the standards indicated below. Members of IEC and ISO maintain registers of currently valid International Standards.
	ISO/IEC Guide 25: 1990, General requirements for the competence of calibration and testing laboratories.
	ISO 9000-3: 1991, Quality management and quality assurance standards - Part 3: Guidelines for the application of ISO 9001 to the development, supply and maintenance of software.
	These are the only normative references required by this interpretation. Informative references used in this Technical Report are given in Annex B.
3	Definitions
	No IT&T specific interpretation is required for the definitions of ISO/IEC Guide 25. However, ISO/ IEC Guide 25 subclause 3.7 is referenced in 10.A.1 and subclause 3.15 is referenced in 4.B.
	Additional definitions are required in this Technical Report; for these, see Annex A.	As far as possible standard definitions are used. Even where this is not possible, the intent is not to standardize new definitions but rather to explain the meaning of terms as used in this Technical Report.
	The distinction between a means of testing (MOT) and a test tool is important in this interpretation.	The complexity of MOT and test tools varies from one area of software testing to another. For example, in OSI and telecommunications protocol testing, each MOT is a very complex hardware and software system which plays a major part in the testing, whereas in compiler testing, in addition to the test suite (of programs) itself, only a few ancillary software test tools are used.
	For the purposes of this Technical Report, a means of testing is hardware and/or software, and the procedures for its use, including the executable test suite itself, used to carry out the testing required. In an accredited testing service, the MOT is run under the control of the testing laboratory.
	For the purposes of this Technical Report, a test tool is hardware and/or software, excluding the test suite itself, used to carry out or assist in carrying out the testing required. It may be concerned with running the test cases, analysing the results, or both. Those concerned with running the test cases may also involve parameterization, selection or even generation of the test cases.
4	Organization and management
4.1	No IT&T specific interpretation is required for sub clause 4.1 of ISO/IEC Guide 25.
4.2	ISO/IEC Guide 25 subclause 4.2 is interpreted in 4.A and 4.B.
4.A	Use of commercial reference implementations
	In ISO/IEC Guide 25, subclause 4.2, requires the following:
	'The laboratory shall...
	have arrangements to ensure its personnelare free from any commercial, financial andother pressures which might adversely affectthe quality of their work; be organized in such a way that confidencein its independence of judgement and integrityis maintained at all times;...'
	In IT&T, these requirements shall be interpreted as follows.
4.A.1	If a commercial implementation (not designed to be a reference implementation) is used by a testing laboratory or validation laboratory as a reference implementation for the purposes of MOT validation within the laboratory, then the adequacy of the technical coverage with respect to the other implementations that are available shall be kept under review by the laboratory, in consultation with the accreditation body. If it is agreed that the technical coverage has become inadequate compared to other implementations, then the commercial implementation should be replaced, within a time period to be agreed with the accreditation body, by one of the following, as appropriate: another implementation with better coverage; a set of implementations from different suppliers; or an implementation which is designed to be a reference implementation.	See 9.B for a description of the use of reference implementations in MOT validation. It is recognised that for some IT&T standards there may be no alternative to using a normal commercial implementation as a reference implementation against which to validate the MOT. In such cases, the publication of the identity of the reference implementation (in order to be open about the nature of the MOT validation conducted) may inadvertently give commercial advantage to the supplier. The decision to use a normal commercial implementation as a reference implementation, and the choice of which commercial Implementation to use in this way, are decisions to be made by the laboratory. In some cases, it may be necessary to use multiple reference implementations for MOT validation, in order to ensure that adequate coverage of the MOT behaviour is checked. This arises because a given commercial implementation may only support a subset (or "profile") of the relevant specification(s). This may be acceptable as a temporary solution, particularly if the market Is primarily interested in that subset, but is inadequate as a longer term solution if the testing service is to cover the specification(s) in full.
	If a set of implementations is chosen, the set shall be chosen to give better technical coverage of the relevant specification(s) and not for commercial reasons.
4.B	Proficiency testing
	In ISO/IEC Guide 25, subclause 4.2 j) requires the following:
	'The laboratory shall, where appropriate,participate in interlaboratory comparisons andproficiency testing programmes. '
	The definition of proficiency testing given in subclause 3.15 of ISO/IEC Guide 25 is as follows:
	'Determination of the laboratory ... testingperformance by means of interlaboratorycomparisons. '
	In ISO/IEC Guide 25, subclause 5.6 requires the following:
	'In addition to periodic audits the laboratory shallensure the quality of results ...by implementingchecks. These checks ... shall include, asappropriate, but not limited to:
	..
	participation in proficiency testing or otherinterlaboratory comparisons;'
	In IT&T, these requirements shall be interpreted as follows.
4.B.1	If a laboratory claims to offer a harmonised testing or validation service, it shall provide evidence of its participation in the relevant inter-laboratory comparisons to ensure that the declared harmonisation is achieved and maintained.	There may be numerous inter-laboratory comparison schemes organised for IT&T. IT&T Agreement Groups have been and are being formed to operate mutual recognition agreements whereby the group of testing laboratories establish the means to recognise the mutual equivalence of their corresponding testing services. Such Agreement Groups provide one formalized way of participating in inter-laboratory comparisons. They may require that testing service harmonization and demonstrations of equivalence are carried out, and that all participating testing laboratories become accredited for the services they offer (within a reasonable period of time). Agreement Groups may also provide inter-laboratory comparison schemes for MOT validation services. If it is not practical or economic for the laboratory to participate in inter-laboratory comparisons, then the laboratory shall not claim that the service is harmonised. A laboratory may decide not to join an Agreement Group and therefore not to claim to provide a harmonised testing or validation service. It may nevertheless be required by the accreditation body to participate in some informal inter-laboratory comparison exercises, in order to overcome any doubts there may be about the objectivity of its test or validation results.
5	Quality system, audit and review
5.1	No IT&T specific interpretation is required for this subclause.
5.2	ISO/IEC Guide 25 subclause 5.2 is interpreted in 5.A and 5.B.
5.3	No IT&T specific interpretation is required for this subclause.
5.4	No IT&T specific interpretation is required for this subclause.
5.5	No IT&T specific interpretation is required for this subclause.
5.6	ISO/IEC Guide 25 subclause 5.6 is interpreted in 4.B.
5.A	Maintenance procedures
	In ISO/IEC Guide 25, subclause 5.2 requires the following:
	'The quality manual and related qualitydocumentation shall also
	...
	reference procedures for... verification andmaintenance of equipment;
	...
	In ISO/IEC Guide 25, subclause 8.2 requires the following:
	'All equipment shall be properly maintained.... '
	In ISO/IEC Guide 25, subclause 9.1 requires the following:
	'All... testing equipment having an effect on theaccuracy or validity of... tests shall be... verifiedbefore being put into service. The laboratory shallhave an established programme for the ...verification of its... test equipment.'
	In IT&T, these requirements shall be interpreted as follows.
5.A.1	A testing laboratory shall have procedures defining the checking to be performed whenever major or minor changes are made to the MOT or other test tools, in order to ensure that harmonisation is maintained as appropriate with other testing laboratories and that correctness is maintained with respect to the relevant standard(s) or specification(s).
5.A.2	A validation laboratory, or a testing laboratory which conducts its own MOT or test tool validations, shall have procedures defining the checking to be performed whenever major or minor changes are made to the reference implementation or other means of validation, in order to ensure that harmonisation is maintained as appropriate with other validation laboratories and that correctness is maintained with respect to the relevant standard (s) or specification (s).
5.B	Documentation of MOT and test tool validation
	In ISO/IEC Guide 25, subclause 5.2 requires the following:
	'The quality manual and related qualitydocumentation shall also contain:
	. . .
	m) reference to procedures for... verificationand maintenance of equipment;
	. . .
	In ISO/IEC Guide 25, subclause 10.1 requires the following:
	'The laboratory shall have documentedinstructions on the use and operation of allrelevant equipment...'
	In ISO/IEC Guide 25, subclause 9.2 requires the following:
	'The overall programme of ... validation ofequipment shall be designed and operated so asto ensure that, wherever applicable,measurements made by the laboratory aretraceable to national standards of measurementwhere available:
	In the context of MOT and test tool validation, these requirements shall be interpreted as follows.
5.B.1	The procedures for carrying out MOT and test tool validations shall be documented by the laboratory.	If an accredited external validation service is used, then the procedures merely need to refer to the use of that service. If a non-accredited external validation service is used, then the laboratory should provide adequate procedures for the selection and monitoring of the results of the service (see 15.A.2). If the laboratory carries out its own validations, then the procedures should include those for selecting which test cases to run.
5.B.2	If, for a given MOT or test tool, there is no suitable validation service available outside the testing laboratory to which accreditation is applicable, and there is no suitable reference implementation that could be used by the testing laboratory to validate the MOT or test tool, then the testing laboratory shall define and document the procedures and methods that it uses to check on the correct operation of the MOT or test tool, and provide evidence that these procedures and methods are applied whenever the MOT or test tool is modified.	The suitability of an external validation service may depend not only on its relevance to the given MOT or test tool, but also on the cost-effectiveness of using the service compared to alternative means of validation that may be available and acceptable. The locally defined procedures could involve arbitrarily complex arrangements of other hardware and software tools. They could also involve some checking of the MOT or test tool by one or more other testing laboratories. ISO/ IEC Guide 25, subclause 9.3, cites inter-laboratory comparison as one of the means of providing satisfactory evidence of correlation of results.
	Such checking is required to fulfil the requirements in ISO/IEC Guide 25 subclauses 8.2 and 9.	See 9.A and 9.B on the validation of the MOT and test tools.
6	Personnel
6.1	ISO/IEC Guide 25 subclause 7.2 is interpreted in 6.A.
6.2	ISO/IEC Guide 25 subclause 7.2 is interpreted in 6.A.
6.3	No IT&T specific interpretation is required for this subclause.
6.A	Maintaining Competence
	In ISO/IEC Guide 25, subclause 6.1 requires the following:
	'The testing laboratory shall have sufficientpersonnel having the necessary education,training, technical knowledge and experience fortheir assigned functions. '
	In ISO/IEC Guide 25, subclause 6.2 requires the following:
	'The testing laboratory shall ensure that thetraining of its personnel is kept up-to-date. '
	In IT&T, these requirements shall be interpreted as follows.
6.A.1	The testing laboratory shall have a procedure to define how to maintain competence of its test operators, particularly in the absence of clients for a given testing or validation service.	It is recommended that in order to maintain their competence in operating a particular testing or validation service, the relevant test operators should be required to conduct at least one test campaign or validation exercise per year, going through all the steps in the process, but not necessarily running all the test cases. Thus, in the absence of clients for a given testing or validation service, special training exercises should be arranged for this purpose.
7	Accommodation and environment
7.1	No IT&T specific interpretation is required for this subclause.
7.2	ISO/IEC Guide 25 subclause 7.2 is interpreted in 7.A, 7.B, 7.C, 7.D, 11.A.2 and 12.A.
7.3	No IT&T specific interpretation is required for this subclause.
7.4	ISO/IEC Guide 25 subclause 7.4 is interpreted in 7.A.
7.5	No IT&T specific interpretation is required for this subclause.
7.6	ISO/IEC Guide 25 subclause 7.6 is interpreted in 11 .A.
7.A	Separation of data partitions
	In ISO/IEC Guide 25, subclause 7.4 requires the following: 'There shall be effective separation betweenneighbouring areas when the activities therein are incompatible. '	Although subclause 7.4 normally refers to separation between different testing services, in some IT&T testing or validation services there could be also the need for clear separation of the software components used within a single testing or validation service.
	In ISO/IEC Guide 25, subclause 7.2 requires the following:
	'The environment in which these activities areundertaken shall not invalidate the results. '
	In ISO/IEC Guide 25, subclause 11.1 requires the following:
	'The laboratory shall have a documented systemfor uniquely identifying the items to be... tested, toensure that there can be no confusion regardingthe identity of such items at any time. '
	In IT&T, these requirements shall be interpreted as follows.
7.A.1	There shall be a clear separation of test tools, the implementation under test (IUT) and the rest of the environment, even when part of that environment is within the same computer system as the IUT.	This is particularly relevant when there is not a clear physical separation between test tools, IUT and the rest of the environment, as is the case in compiler, graphics, portable operating system interfaces (POSIX) and application portability testing. This is also the case when using the Distributed and Coordinated test methods in Open Systems Interconnection (OSI) or telecommunications protocol testing.
7.A.2	The testing laboratory and client shall agree in writing what constitutes the IUT and what constitutes the environment within the system under test (SUT).
7.A.3	Insofar as the IUT is integrated in a system also provided by the client, the laboratory shall ensure that there is no interference from other activities in the environment part of the SUT which could affect the test or validation results, and shall instruct the client to take the necessary actions to achieve this.	As far as is practical, it should be ensured that the SUT is not supporting any activities while the tests or validation are being conducted, other than those required for the testing or validation. For example, in the case of POSIX testing, it has to be ensured that the SUT is not running any task which could interfere with the correct installation, configuration or execution of the test suite, and that no other user is logged onto the system. A 'clean', empty and secure directory should be set up for the test campaign.
7.B	Identification of environment
	In ISO/IEC Guide 25, subclause 7.2 requires the following:
	'The environment in which these activities areundertaken shall not invalidate the results oradversely affect the required accuracy ofmeasurement.'
	In ISO/IEC Guide 25, subclause 11.1 requires the following:
	'The laboratory shall have a documented systemfor uniquely identifying the items to be... tested, toensure that there can be no confusion regardingthe identity of such items at any time. '
	In IT&T, these requirements shall be interpreted as follows.
7.B.1	The laboratory shall obtain from the client and record, as far as possible, the correct and complete identification of the test environment within the SUT (or the validation environment within the MOT). The client shall take responsibility for this identification. In addition, the laboratory shall record the values of any parameters that are used to control the test environment within the SUT during testing (or to control the validation environment within the MOT during validation).	In the context of IT&T testing and validation, the test (or validation) environment means the software and hardware operating environment in which testing or validation takes place. Part of this may be within the SUT (or MOT) and part may be outside the SUT (or MOT). The record of the software test (or validation) environment commences with its definition. Depending on the type of testing or validation to be undertaken, the test (or validation) environment within the SUT (or MOT) may consist of a number of parameters (e.g. processor, disk drive, operating system, compiler, compiler switches, linker, run-time system — this list is not comprehensive). All such parameters should be precisely defined, including as a minimum the name, model and version number. In some specific cases, it is known that even individual components need to be identified.
7.B.2	The test (or validation) environment outside the SUT (or MOT) shall be sufficiently recorded and controlled by the laboratory to ensure, as far as possible, correct and complete identification of the test (or validation) environment outside the SUT (or MOT) at any time.
7.C	Remote testing over a network
	In ISO/IEC Guide 25, subclause 7.2 requires the following:
	'The environment in which these activities areundertaken shall not invalidate the results oradversely affect the required accuracy ofmeasurement. Particular care shall be takenwhen such activities are undertaken at sites otherthan the permanent laboratory premises. '
	In IT&T, these requirements shall be interpreted as follows.
7.C.1	For the purposes of this Technical Report, remote testing is testing which is conducted over a network, with the SUT connected to but separate from the network, and the network being part of the test environment.	Note that the interpretation in 7.C.2 applies whether or not the SUT is in a different physical location from the MOT. It is also possible that both the main part of the MOT and the SUT are physically remote from the test operators. It is sometimes impractical for testing laboratory staff to be physically present with each separate part of the MOT and with the SUT, but this need not undermine the trustworthiness of the test results.
7.C.2	In the context of remote testing over a network, the laboratory shall use test methods and procedures that ensure that the behaviour of the network does not invalidate the test or validation results.	Special consideration needs to be given to the requirements on environmental control when testing remotely over a network. In particular, any environmental conditions which might affect the correct operation of the network should be noted. If the requirements are not met, then no testing or validation should take place.
		Prior to running conformance or performance tests, a number of specified tests should be executed in order to establish that the network is operating correctly. If a failure is encountered, diagnostic codes should be checked to determine whether the failure was in the network or in the IUT. If a network error is detected, then checks should be carried out to ensure that no previously run tests have been affected by the error.
		Problems may be encountered during remote testing if the network is being heavily used. In such situations, it may be necessary to reschedule testing or validation to a time when usage of the network is not so high.
		If, during test execution, interference is caused by network behaviour (e.g. a network generated reset or disconnect) then the verdict should be considered to be inconclusive and the test case should be rerun.
		In order for the laboratory to be assured that the IUT is the one stated by the client, the laboratory will normally need to rely on contractual commitment. In some special cases, it may be appropriate for the laboratory to require the use of software authentication techniques. If all the observations of the SUT behaviour required for the purpose of determining test results can be made remotely from the SUT, and the SUT is at a separate physical location from the at least part of MOT, then it should not be necessary for a test operator to be present at the SUT location as well as the main MOT location.
7.D	Checking the testing or validation environment
	In ISO/IEC Guide 25, subclause 7.2 requires the following:
	'The environment in which these activities areundertaken shall not invalidate the results oradversely affect the required accuracy ofmeasurement. Particular care shall be takenwhen such activities are undertaken at sites otherthan the permanent laboratory premises. '
	In IT&T, these requirements shall be interpreted as follows.
7.D.1	The laboratory shall make it clear to the client that it is their responsibility to ensure that all equipment supplied by them is available and operating correctly. The laboratory shall ensure that pre-test or pre-validation checks are performed on the equipment, to establish its suitability for purpose.	This is particularly important when testing at a site other than the permanent laboratory premises, because the client may be asked to provide equipment on which test software and/or test suites are to be mounted.
7.D.2	A testing laboratory shall ensure that the correct version of each MOT and each test tool is used and that they have not been modified in any way that might lead to incorrect test results.	This is particularly important when the client has received a copy of the MOT or test tool prior to the date on which the testing laboratory begins the testing. The best way to be sure that the MOT or test tool is the correct version and is unmodified is to use a new copy of the MOT or test tool, brought by the testing laboratory. If it is impractical to use a new copy of the MOT or test tool (e.g. because of the time needed to configure it for use with the SUT) then an integrity check will need to be performed on the MOT or test tool, either by using some checking tool or by comparing the MOT or test tool with a reference copy brought by the testing laboratory.
7.D.3	A validation laboratory, or a testing laboratory which conducts its own MOT or test tool validations, shall ensure that the correct version of each reference implementation or other means of validation is used and that they have not been modified in any way that might lead to incorrect validation results.
7.D.4	For all software testing and validation, it shall be ensured that any files containing old results or old test programs on the SUT cannot be confused with the current test programs and test or validation results.
8	Equipment and reference materials
8.1	ISO/IEC Guide 25 subclause 8.1 is interpreted in 9.B.1.
8.2	ISO/IEC Guide 25 subclause 8.2 is interpreted in 5.A, 5.B.2, 8.B, 8.C, 9.A.1 and 9.B.1.4.
8.3	No IT&T specific interpretation is required for this subclause.
8.4	ISO/IEC Guide 25 subclause 8.4 is interpreted in 8.A, 8.C, 8.D, and 9.A.2.
8.A	Means of testing and test tool validation records
	In ISO/IEC Guide 25, subclause 8.4 requires all equipment records to include 'details ofmaintenance carried out...;and 'history of any...modification or repair.
	In IT&T, these requirements shall be interpreted as follows.
8.A.1	A testing laboratory shall maintain a record of all MOT validations and re-validations carried out.	See 9.A and 9.B for a description of the MOT validation concept and the use of reference implementations in MOT validation.
8.A.2	If the MOT validations are carried out by the testing laboratory itself, its records shall include the reasons for the test cases being run, date, environmental information if appropriate, and a summary of the results obtained plus the detail of any discrepancies from the expected results. When MOT validation is made by the testing laboratory itself using a reference implementation, the requirements also mean that the testing laboratory shall document fully the expected results (i.e. previously obtained results) from using the full conformance test suite to test the nominated reference implementation.	When an implementation is first being considered as a possible reference implementation, there will need to be a period In which it is tested using the full conformance test suite and the results obtained are scrutinized to check their validity. Once they have been checked in this way, those results (possibly as amended as a consequence of the scrutiny) become the 'expected results' to be looked for on a subsequent validation or re-validation of the MOT. Records of MOT validation should include clarification of the coverage of the validation with respect to the test suite being used.
8.A.3	If the MOT validations are carried out by one or more external organisations, the testing laboratory shall keep in its records the validation reports, each including the identity of the organisation(s) carrying out the validation, date(s), and results of the validation.	It is important for traceability that the validation of each MOT or test tool results in a validation report, provided to the MOT supplier to be copied to all testing laboratories that use that MOT or test tool.
8.B	Procedures for handling errors in the MOT and test tools
	In ISO/IEC Guide 25 subclause 8.2 requires the following: 'Any item of equipment which has been subjectedto overloading or mishandling, or which givessuspect results, or has been shown by verificationor otherwise to be defective, shall be taken out ofservice... until it has been repaired and shown by... verification or test to perform satisfactorily. '	In order to Interpret these requirements in the IT&T field, it needs to be realised that an 'item of equipment' for a testing laboratory does not need to be a complete MOT. The interpretation of 'item of equipment' depends on the extent to which the location of the fault can be isolated from the rest of the MOT.
	In IT&T, these requirements shall be interpreted as follows.
8.B.1	If a defect in an item of equipment is such that it impacts only a few specific test cases, then those test cases shall be taken out of service. These test cases shall not be put into service again unless revised procedures are introduced for analysing the results of those test cases to overcome the defect.	In particular, this may well be the case for defects In results analysis tools (i.e. test tools which analyse the recorded observations made during testing in order to produce test or validation reports). If a testing service is such that there are specified test cases that are run, then if the discrepancies seem to be Isolated to particular test cases, then only those test cases need be suspended from use. If, however, the concept of test cases does not apply or if the discrepancies indicate a general problem with the MOT or test tool, then the MOT or test tool needs to be suspended.
8.B.2	Thus, the testing laboratory shall document and use adequate procedures, for use whenever any MOT or test tool is suspected or found to contain errors which make it defective or unfit for use. These procedures shall include establishing that there is a genuine error, reporting the error to the appropriate maintenance authority, withdrawing the MOT, test tool or test case(s) from service, as appropriate, correcting the errors, and then re-validating the MOT or test tool, as appropriate.
8.B.3	Similarly, a validation laboratory shall document and use adequate procedures, for use whenever any reference implementation or other means of validation is suspected or found to contain errors which make it defective or unfit for use. These procedures shall include establishing that there is a genuine error, reporting the error to the appropriate maintenance authority, withdrawing the reference implementation or other means of validation from service, as appropriate, correcting the errors, and then re-checking the reference implementation or other means of validation, as appropriate.	it Is important to recognise the difference between errors in a reference implementation which make it unfit for use, and deliberate error generation capabilities built into the reference implementation to make its use in MOT validation more effective. It is only the former that need to be removed.
8.C	MOT and test tool maintenance
	In ISO/IEC Guide 25, subclause 8.2 requires the following:
	'All equipment shall be properly maintained.Maintenance procedures shall be documented. '
	In ISO/IEC Guide 25, subclause 8 4 requires that equipment records shall include the following:
	the manufacturer's name... '
	In IT&T, these requirements shall be interpreted as follows.
8.C.1	A testing laboratory shall keep a record of the design and maintenance authority for each MOT, test tool and executable test suite.	For some MOT, the executable test suite may be maintained by a different organisation from the test tool on which it runs. It is important that test case errors can be reported directly to the appropriate maintenance authority.
8.C.2	A validation laboratory, or a testing laboratory which conducts its own MOT or test tool validations, shall keep a record of the design and maintenance authority for each reference implementation and each tool used in other means of validation.
8.D	Identification of equipment
	In ISO/IEC Guide 25, subclause 8.4 specifies requirements for equipment records, including identification.	In the IT&T context, the use of the term 'equipment' in this clause refers only to 'test equipment'. The 'equipment under test' is referred to as the 'test item' in ISO/IEC Guide 25 and is the subject of clause 11 rather than 8. In each equipment record for an MOT, the MOT should be identified by its name, version number, and supplier; and a reference should be given to the latest validation report, if relevant, for that MOT.
9	Validation and traceability (Measurement traceability and calibration)
9.1	ISO/IEC Guide 25 subclause 9.1 is interpreted in 5.A, 9.A.2, 5.B.2 and 9.F.1.
9.2	ISO/IEC Guide 25 subclause 9.2 is interpreted in 5.B and 9.D.1.
9.3	ISO/IEC Guide 25 subclause 9.3 is interpreted in 9.A.2 and 9.B.1, and referenced in 5.B.2.
9.4	No IT&T specific interpretation is required for this subclause.
9.5	No IT&T specific interpretation is required for this subclause.
9.6	ISO/IEC Guide 25 subclause 9.6 is interpreted in 9.A.2, 9.B.1 and 9.E.
9.7	No IT&T specific interpretation is required for this subclause.
9.A	The validation concept
9.A.1	In IT&T, the concept of MOT validation is used to meet the ISO/IEC Guide 25 requirements for verification of equipment.	Since measurement traceability and calibration are not generally directly relevant to software and protocol testing, the title of clause 9 in this interpretation has been changed to 'Validation and traceability'. However, one exception Is given in 9.F.
	In ISO/IEC Guide 25, subclause 8.2 requires the following: 'All equipment shall be properly maintained.... ' In ISO/IEC Guide 25, subclause 8.2 also requires the following: 'Any item of equipment which has been subject to... mishandling, or which gives suspect results, orhas been shown by verification or otherwise to bedefective shall be taken out of service, ... until ithas been repaired and shown by... verification ortest to perform satisfactorily. The laboratory shallexamine the effect of this defect on the previous...tests. '	The general accreditation requirements (as specified by ISO/IEC Guide 25, clause 9) include requirements concerning calibration and traceability which are applicable to the use of measuring equipment in testing. These concepts are not directly applicable in this interpretation to software and protocol testing. This is because the relevant MOT and test tools are not what is meant by 'measuring equipment', because they observe and control the operation of the tests, rather than making any physical measurements. The one obvious exception is in the case of Physical layer communications testing, which is mostly outside the scope of this interpretation. Furthermore, there are no relevant 'primary standards' of measurement to which traceability, via a chain of calibrations, can refer.
	Nevertheless, the requirements from ISO/IEC Guide 25 quoted here are applicable to software and protocol testing. When taken together it can be seen that they require some means of checking the validity of the MOT and test tool Standard Number BS ISO/IEC TR 13233:1995 Title Information technology. Interpretation of accreditation requirements in ISO/IEC Guide 25. Accreditation of information technology and telecommunications testing laboratories for software and protocol testing services Status Definitive Publication Date 1996-11-15 Identical National Standard Of ISO/IEC TR 13233:1995 Descriptors Laboratory testing, Conformity, Calibration, Telecommunication, Laboratory accreditation, Definitions ICS 35.020 Information technology (IT) in general, 03.120.20 Product and company certification. Conformity assessment Committee ICT/1 ISBN 0 580 26595 1 Publisher BSI Pages 56 About BSI BSI Group, also known as the British Standards Institution is the national standards body of the United Kingdom. BSI produces technical standards on a wide range of products and services and also supplies certification and standards-related services to businesses.	GROUPS Book Keywords US Local Codes BOOK KEYWORDS climate electric code electric power energy code existing building fire code fire door fire extinguishers fire fighting fire hydrants fuel gas code geotechnical green guide green tips health care hurricane hvac hydroelectric mechanical code plumbing plumbing code plumbing standard residential code school seismic snow solar energy swimming pool tsunami wind power building code civil earthquake structural load US LOCAL CODES US Territory Code Adoptions / Amendments District of Columbia Guam Northern Marianas Islands Puerto Rico US Virgin Islands US State Code Adoptions / Amendments AL AK AZ AR CA CO CT DE FL GA HI ID IL IN IA KS KY LA ME MD MA MI MN MS MO MT NE NV NH NJ NM NY NC ND OH OK OR PA RI SC SD TN TX UT VT VA WA WV WI WY             ICS 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100