External quality assessment performance in ten countries: an IFCC global laboratory quality project

Introduction

As part of total quality management (TQM), external quality assessment (EQA) is a vital component to ensure validity of the laboratory results so as to ensure confidence in patient results.

To assure analytical reliability of the results, internal quality control (IQC) is the primary tool used by medical laboratories. While IQC is critical for daily operation to evaluate changes in instrumentation and reagent lots, provides a regular check on the continuous precision of the analytical system, and identifies probable analytical errors, it is of limited utility in evaluating result accuracy and it often does not encompass the entire testing process.

EQA is a key component of a laboratory’s TQM system, required by many national regulations, for accreditation according to ISO EN 15189 [1], and recommended by the Clinical and Laboratory Standards Institute (CLSI) [2, 3]. When EQA schemes are not available for a specific analyte or methodology, alternative assessment procedures are necessary, and these vary significantly between laboratories and regions [3, 4]. EQA provides an external assessment of laboratory performance over time. Depending on the design, EQA can provide an assessment of trueness, intra and interlaboratory variation, linearity, identify differences between methods, and monitor continued efforts at harmonization 5], [6], [7], [8.

The IFCC inaugurated a Task Force on Global Lab Quality (TF-GLQ) in 2021. The initial scope and mandate were to provide tools for the evaluation of laboratories’ performance in developing countries using IQC and EQA programs in order to improve the medical diagnostics and disease treatment of patients. The intention was to identify vendors capable of providing quality control materials and associated software for on-going monitoring and to develop training programs for participating laboratories.

For the EQA program, a call for application to participate were issued to all IFCC member societies from which ten countries, (Georgia, Serbia, Bosnia/Herzegovina, Bolivia, Columbia, Peru, Indonesia, Sri Lanka, Malawi and Zambia) were selected to participate in the pilot program. Working in collaboration with the respective member societies, five candidate participating laboratories within each country were identified. A repertoire of common clinical biochemistry laboratory tests was agreed and suggested for the EQA program.

A bidding process was undertaken in 2021 to identify a commercial vendor for the EQA program and subsequently, OneWorld Accuracy (1WA), Vancouver, Canada, was selected to provide EQA materials, related software and respective online training to participating laboratories. All costs were borne by the IFCC. The TF-GLQ reviewed all reports, discussed the results, and any performance issues on a monthly basis.

The EQA program started in 2022 with 50 labs (i.e., five per country). Based on the positive results of a survey conducted at the end of 2022 (participants indicated satisfaction with the EQA program), participating countries were encouraged to extend the program from five to ten laboratories at the beginning of 2023. The results reported here are for the program run in 2023.

Materials and methods

Results

Figure 1 shows the percentage of survey participation for the ten individual countries for 2023. Although not shown in Figure 1, Country A and Country F did not return any results for October, November and December and Country J for November and December 2023. Country G had zero participation for April, July and October 2023. Country J averaged only 2.4 laboratories per month submitting results.

Figure 1:

The rate of participation of countries in the survey for 2023. The Figure shows overall participation for each of the ten countries in % for 2023.

As part of enrolling in the 1WA results data base, the laboratories were required to list the manufacturer and instrument model for each analyte. Table 1 list the instrument manufacturers that were used in two or more laboratories within the 100 included in the survey. In total, it was found that at least 50 manufacturers and over 100 instruments were listed as being used by these laboratories, including some used for one analyte only.

For some of the countries there were considerable differences in the number of results reported by different laboratories (Figure 2). As can be seen, the maximum number of results reported by an individual laboratory for Country A is greater than the number of analytes in the material provided which is due to one laboratory returning results for the same analyte assayed on multiple instruments. Similarly, one laboratory from Country F measured the same analytes on two different instruments. There was large variability in the instruments used by laboratories within any one country.

Figure 2:

Results returned per month by laboratories in the 10 survey countries. Some laboratories returned more results than the number of analytes in the material provided as it was assayed on multiple instruments within one laboratory. Minimum and maximum number of applied analytes per laboratory per country are shown. Note that each laboratory could choose from the 80 provided lyophilised serum containing analytes according to their laboratory’s routine. In case that the graph shows a number greater than 80 (i.e., country A) same analytes were assayed on multiple instruments

The percentage of acceptable results submitted was calculated using the z-score (Figure 3).

Figure 3:

Acceptability per month for individual countries. The acceptability was determined from the reports provided by 1WA based on CLIA criteria or±2 SD if not available in CLIA’88. The four countries shown in red were considered not to perform to an acceptable standard. Acceptability per month for each country. Of the ten countries surveyed, six averaged greater than 90 % acceptable results over the whole 12 months cycle, one had unacceptable results for two of the nine months they returned results and the other four were considered to not perform to an acceptable standard (shown in red). See text for additional interpretation.

Of the ten countries surveyed, six averaged greater than 90 % acceptable results over the whole 12-months cycle, one had unacceptable results for two of the nine months they returned results and the other four were considered to not perform to an acceptable standard (shown in red). For instance, the October result of 4 % is from two laboratories in Country J, with one laboratory having all 15 results unacceptable and the other having 10 of 11 results being unacceptable. Country A and F had no submission for October, November and December, Country J for November and December and Country G for April, July and October.

The results in Table 2 show the various in acceptability of results for individual laboratories within each country.

As can be seen, the number of analyte results submitted for some laboratories varied significantly, e.g., laboratory 5, country F and laboratory 8, country H. On further investigation, this was found to be lack of reagents which is a major issue in many developing countries.

The performance for individual analytes measured in the survey are shown in Table 3.

There are 80 available analytes in the material that was provided and the breakdown of these that were assayed by the participants is as follows:

1. 80 % of the participants assayed albumin, alkaline phosphatase, ALT, AST, bilirubin total and conjugated, cholesterol, creatinine, glucose, potassium, total protein, sodium, urea, triglycerides and uric acid.

2. 20 % of the participant laboratories assayed amylase, CO2 total, DHEA sulphate, lithium, and transferrin.

3. Less than 20 % of the participant laboratories assayed 11 deoxy-cortisol, 17 OH progesterone, aldosterone, androstenedione, cholinesterase, fructosamine, homocysteine, Lp(a), osmolality, prostatic acid phosphatase, total acid phosphatase, SHBG, T3 uptake, TIBC, acetaminophen, carbamazepine, digoxin, gentamicin, phenobarbital, phenytoin, salicylate, theophylline, tobramycin, valproic acid and vancomycin.

As expected, analytes requiring immunoassay for measuring were generally the least performed.

Discussion

The scope of this study was to evaluate laboratories’ EQA program performances in developing countries in order to improve the medical diagnostics and disease treatment of patients. EQA materials and associated software for on-going monitoring were provided free of charge to participating countries. We wanted to understand daily performance routines and their issues, acceptability rates of applied assays (pass or fail) and develop training programs for participating laboratories.

The simple EQA design applied fulfilled our aim of identifying poor performing laboratories in low to middle income countries that required further help and support. We didn’t need commutable, metrological traceable material with APS based on clinical or biological goals to achieve our aim. We needed cheap, stable material from a provider that had global EQA presence where we could use simple peer review using pragmatic APS criteria.

As an ISO 17043 accredited provider, 1WA assesses the homogeneity and stability of the EQA samples provided in their program. They may assess commutability of the samples based on CLSI EP14 “Evaluation of commutability of processed samples” but that is not a requirement. There is no mention in 1WA EQA collateral suggesting commutable samples are utilized. Note that an assessment of commutability would be done on the broader data set for each sample, not just that associated with labs in the IFCC pilot program.

As the standardization/harmonization of test methods was not the intent of the EQA Pilot program, commutability was not assessed by the TF-GLQ nor by 1WA on the processed samples. We know that EQA samples supporting large (global) programs will likely have matrix effects (biases) associated with their supplemented additives and processing. That is why consensus-based peer group assessments (instrument/reagent), where possible, are valuable in determining acceptable performance of participants.

Commutability was not assessed and considered unlikely. For this reason, peer view assessment was used rather than a comparison against a reference method or overall mean. The method peer group mean from participants results after exclusion of outliers was therefore used as the target value for each sample.

There are several different APSs that could be used (some country specific), but the program selected already included the recently revised CLIA criteria. These were selected in preference to the Milan Model biological variation used within Europe and Australia as the latter was considered unrealistic for low- and middle-income countries.

This EQA performance study covered the years 2022 and 2023. In preparation of this study, ten IFCC member countries were recruited in 2021, liaison managers in these countries were identified and assigned with the help of the respective local Clinical Chemistry organisations. A suitable commercial vendor of EQA materials was identified (i.e., 1WA) and a proposal for a panel of 15 commonly and widely used Biochemistry-and Immunoassays for routine analysis was developed. However, not every laboratory measured all of these, and they were expected to only measure analytes which were already part of their routine service.

In 2022, 50 laboratories in ten developing countries participated. Based on the initial acceptance of the program (as evidenced by a survey conducted at the end of 2022 in which participating countries expressed interest to expand the program), the program was extended to 100 laboratories in the same ten participating countries. The results reported here are for the program run in 2023.

We observed distinct differences in performance: Six out of the ten participating countries showed a consistently solid acceptance rate of >90 % over the observed time period. In contrast, four out of the ten participating countries showed poor acceptance rates between 60 and 90 %. This is below generally accepted EQA standards and requires improvement. The TF-GLQ intends to focus their activities to continue aid by online and on-site training particular with these countries. The six countries with solid acceptance rates of >90 % do not need to be further monitored and do not require additional assistance.

Some countries encountered shipping and customs clearance problems, thus being unable to submit results in a timely fashion. As lyophilized EQA materials are stable, it was concluded that for many developing countries in which logistics and customs clearance may be an issue, a single annual shipment is the most appropriate way to distribute the EQA material.

During the survey period and from subsequent discussions with participating laboratories, a number of common issues were identified as affecting performance:

1. Material awaiting custom clearance causes delays in laboratories receiving samples and not meeting the submission deadlines, as was the case for laboratories in Country G which were unable to submit results for April, July and October.

2. The availability of reagents and other materials due to a lack of money is a significant problem for laboratories in some countries. Also, often governments in developing countries prioritise resources and biochemistry may not be seen as being as urgent as the measurement of infectious diseases. For instance, in Africa, there is a major drive to control the spread of malaria, HIV and tuberculosis.

3. In some countries, there was a lack of understanding of the importance of quality control and external quality assessment and the associated procedures in ensuring acceptable laboratory performance leading to a non-commitment by the laboratory. This was reflected in the results for Country J where only 4 out of the 10 laboratories that were enrolled in the program submitted at least 4 sets of results. In this scenario, previous experience has shown that laboratories will participate if there is oversight of the program by the relevant authority such as the Ministry of Health.

In October and November of 2023, scientific support teams from the TF-GLQ in collaboration with the national societies organized workshops, laboratory visits and conducted country-specific training in Malawi, Zambia, Peru and Columbia. Workshops included training and case studies in IQC along with interpretation of EQA reports using real case studies and performance data from those countries. These visits were also used to identify issues and barriers to improving the quality of laboratory diagnostics in these and similar countries and the measures needed to address these concerns.

This study shows that not all countries perform to globally accepted EQA performance standards (note: all participating countries of this study are IFCC member countries) and that additional future efforts should be devoted to improve this unsatisfactory situation to the good of improved medical diagnostics and disease treatment of patients.