Measuring accuracy of the neutralizing activity of COVID-19 convalescent plasma

Massimo Franchini; Carlo Mengoli; Beatrice Caruso; Roberto Petilino; Alessia Ballotari; Claudia Glingani

doi:10.1515/cclm-2021-0810

Article Publicly Available

Measuring accuracy of the neutralizing activity of COVID-19 convalescent plasma

Massimo Franchini , Carlo Mengoli , Beatrice Caruso , Roberto Petilino , Alessia Ballotari and Claudia Glingani

Published/Copyright: September 2, 2021

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information

From the journal Clinical Chemistry and Laboratory Medicine (CCLM) Volume 60 Issue 1

Keywords: COVID-19; neutralizing antibodies; SARS-CoV-2

To the Editor,

Plasma collected from individuals recovered from severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, called hyperimmune or convalescent plasma (CP), has been utilized worldwide during the pandemic Coronavirus Disease 2019 (COVID-19) period with mixed results [1]. Assuming that the main determinant of CP effectiveness is its level of anti-SARS-CoV-2 neutralizing antibodies, it is reasonable that most investigators focused their research on the identification of the optimal neutralization tests. For instance, Bal and colleagues recently evaluated the neutralizing activity of nine high-throughput SARS-CoV-2 commercial serological assays and concluded that, although with some differences, they cannot replace the reference virus neutralization test (VNT) [2]. Several studies have tried to validate various serological tests as surrogate of VNT [3, 4], which is currently considered the gold standard for assessing the neutralization potency of CP but which is quite demanding (utilizes live SARS-CoV-2 virus and thus require a biosafety level 3 laboratory) and time-consuming (5–7 days for results) [5]. This argument is particularly critical in the setting of COVID-19 CP donation, where there is the need to collect rapidly high-titer CP units. To further elucidate this still unsolved issue, we have compared the traditional VNT₅₀ (performed at the Molecular Virology Unit of the University Hospital of Pavia and based on the determination of the in vivo cytopathic effect, as previously described) [6] and the chemiluminescent immunoassay (CLIA) DiaSorin (LIAISON SARS-CoV-2 S1/S2 IgG, DiaSorin, Vercelli, Italy) in 495 consecutive CP donors (median age 42.6 years, range 19–65 years; 401 males and 94 females).

The VNT₅₀ was used as “gold standard” and a titer ≥80 was considered positive (negative if < 80). The Pearson’s rho as measure of the correlation and the non-parametric Spearman rho were calculated. Moreover, the relationship between the two tests was investigated by linear regression, adjusting for age and gender. Adequacy of linear model was verified by visual inspection of data and linear prediction. Then, the non-parametric receiver operating characteristic (ROC) curve method was used to assess the predictive value of Diasorin SARS-CoV-2 S1/S2 IgG assay as a potential substitute of the more demanding neutralization assay. For the search of the cut-point, the following methods were used: (1) maximization of the sum sensitivity + specificity, (2) maximization of the product sensitivity × specificity, and (3) minimization of the absolute difference sensitivity – specificity. Area under curve (AUC), sensitivity, specificity, accuracy, true positives (TP), false negatives (FN), false positives (FP), true negatives (TN) and diagnostic odds ratio (DOR) at each cut-point were reported as well. Stata 16.1 and R version 4.0.3 were used to perform calculations.

The in vitro neutralization test detected 340 subjects with a titer ≥80 (neutralization-positives), and 155 subjects with a lower titer (neutralization-negatives). By linear multiple regression method on 495 observations, the linearity of the relationship between the quantitative VNT₅₀ and the DiaSorin SARS-CoV-2 S1/S2 IgG assay was far from ideal (Figure 1, top). A locally weighted scatterplot smoother (LOWESS) was added to the figure, for comparison to the linear prediction. Using the ROC curve method, the area under the curve was 0.6471 (95% CI 0.5957, 0.6985) (Figure 1, bottom). The optimal cut-point was 89 with methods (1), 70 with (2), and 63 with (3). As showed in Table 1, at the first cut-point (89), the sensitivity was 43%, the specificity was 81%, and the fraction of the correctly classified patients (accuracy) was 55%. At the third cut-point (63), the sensitivity, specificity and accuracy were 59%.

Figure 1:

Correlation between viral neutralization and Diasorin tests by linear multiple regression (top) and ROC curve (bottom) methods.

Table 1:

Performances of the three candidate cut-points applied to the ROC curve procedure.

	Max (sensitivity + specificity)	Max (sensitivity × specificity)	Min (sensitivity − specificity)^a
Method	1	2	3
Cut-point	89	70	63
Accuracy	0.55	0.58	0.59
Sensitivity	0.43	0.53	0.59
Specificity	0.81	0.68	0.59
TP	145	180	201
FN	195	160	139
FP	30	50	64
TN	126	106	92
DOR	3.12	2.39	2 0.08

TP, true positive; FN, false negative; FP, false positive; TN, true negative: DOR, diagnostic odds ratio; MAX, maximization; Min, minimization. ^aAbsolute difference.

The ROC curve method appeared fit to evaluate the predictive value of the DiaSorin test. Based on the AUC, the DiaSorin assay showed a modest diagnostic accuracy using the traditional VNT₅₀ as the reference test. Its capability to predict the neutralization assay cannot be considered adequate due to the overlap of negative (<80) and positive (≥80) results. This imperfect separation explains the low AUC value.

In conclusion, the suboptimal sensitivity, specificity and accuracy of the Diasorin SARS-CoV-2 S1/S2 IgG assay with respect to VNT₅₀ observed in our study is in agreement with the results of the study by Bal and colleagues [2] and confirms that this method cannot replace viral neutralization for the correct determination of the neutralizing activity of CP.

Corresponding author: Massimo Franchini, MD, Department of Hematology and Transfusion Medicine, Carlo Poma Hospital, Mantova, Italy, Phone: +0039 0376 201234, Fax: +0039 0376 220144, E-mail: massimo.franchini@asst-mantova.it

Research funding: None declared.
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
Competing interests: Authors state no conflict of interest.
Informed consent: Informed consent was obtained from all individuals included in this study.
Ethical approval: The local Institutional Review Board deemed the study exempt from review.

References

1. Franchini, M, Liumbruno, GM, Piacentini, G, Glingani, C, Zaffanello, M. The three pillars of COVID-19 convalescent plasma therapy. Life (Basel) 2021;11:354. https://doi.org/10.3390/life11040354.Search in Google Scholar

2. Bal, A, Pozzetto, B, Trabaud, MA, Escuret, V, Rabilloud, M, Langlois-Jacques, C, et al.. Evaluation of high-throughput SARS-CoV-2 serological assays in a longitudinal cohort of patients with mild COVID-19: clinical sensitivity, specificity and association with virus neutralization test. Clin Chem 2021;67:742–52. https://doi.org/10.1093/clinchem/hvaa336.Search in Google Scholar

3. Tang, MS, Case, JB, Franks, CE, Chen, RE, Anderson, NW, Henderson, JP, et al.. Association between SARS-CoV-2 neutralizing antibodies and commercial serological assays. Clin Chem 2020;66:1538–47. https://doi.org/10.1093/clinchem/hvaa211.Search in Google Scholar

4. Suhandynata, RT, Hoffman, MA, Huang, D, Tran, JT, Kelner, MJ, Reed, SL, et al.. Commercial serology assays predict neutralization activity against SARS-CoV-2. Clin Chem 2021;67:404–14. https://doi.org/10.1093/clinchem/hvaa262.Search in Google Scholar

5. Wouters, E, Steenhuis, M, Schrezenmeier, H, Tiberghien, P, Harvala, H, Feys, HB, et al.. Evaluation of SARS-CoV-2 antibody titers and potency for convalescent plasma donation: a brief commentary. Vox Sang 2021;116:493–6. https://doi.org/10.1111/vox.13060.Search in Google Scholar

6. Perotti, C, Baldanti, F, Bruno, R, Del Fante, C, Seminari, E, Casari, S, et al.. Mortality reduction in 46 severe Covid-19 patients treated with hyperimmune plasma. A proof of concept single arm multicenter trial. Haematologica 2020;105:2834–40. https://doi.org/10.3324/haematol.2020.261784.Search in Google Scholar

Received: 2021-07-17

Accepted: 2021-08-17

Published Online: 2021-09-02

Published in Print: 2022-01-26

Articles in the same Issue

https://doi.org/10.1515/cclm-2021-0810

Keywords for this article

COVID-19; neutralizing antibodies; SARS-CoV-2