Title: | Multi-Reader Multi-Case Analysis of Binary Diagnostic Tests |
---|---|
Description: | The goal of 'MRMCbinary' is to compare the performance of diagnostic tests (i.e., sensitivity and specificity) for binary outcomes in multi-reader multi-case (MRMC) studies. It is based on conditional logistic regression and Cochran’s Q test (or McNemar’s test when the number of modalities is equal to 2). |
Authors: | Seungjae Lee [aut, cre]
|
Maintainer: | Seungjae Lee <[email protected]> |
License: | MIT + file LICENSE |
Version: | 1.0.5 |
Built: | 2025-02-20 08:25:46 UTC |
Source: | https://github.com/seungjae2525/mrmcbinary |
R package MRMCbinary is a package aimed at comparing the performance of diagnostic tests (i.e., sensitivity and specificity) for binary outcomes in multi-reader multi-case (MRMC) studies.
Seungjae Lee [email protected] and Woojoo Lee [email protected]
Lee, S., Jang, S., and Lee, W. (2025). Evaluating Diagnostic Accuracy of Binary Medical Tests in Multi-reader Multi-case Study.
Useful links:
Report bugs at https://github.com/seungjae2525/MRMCbinary/issues
MRMCbinary()
is the main function of MRMCbinary
package and
can be used to compare sensitivity and specificity of diagnostic tests for binary outcomes in multi-reader multi-case (MRMC) studies.
MRMCbinary( data, Modality, Reader, Case, D, Y, measure, effect, interaction = NULL, reference.Modality = NULL, reference.Reader = NULL )
MRMCbinary( data, Modality, Reader, Case, D, Y, measure, effect, interaction = NULL, reference.Modality = NULL, reference.Reader = NULL )
data |
A data frame in which contains the modality identifiers ( |
Modality |
Variable of the modality identifiers. |
Reader |
Variable of the reader identifiers. |
Case |
Variable of the case identifiers. |
D |
Variable of the true disease status. It should be set the value to 1 for cases diseased and to 0 for those non-diseased. |
Y |
Variable of the binary diagnostic test result. It should be set the value to 1 for cases diagnosed as positive and to 0 for those diagnosed as negative. |
measure |
Diagnostic accuracy measure (one of |
effect |
Effect to compare sensitivity and specificity (one of |
interaction |
When evaluating the interaction effect between modality and reader, |
reference.Modality |
Reference in variable of the modality identifiers. |
reference.Reader |
Reference in variable of the reader identifiers. |
There are three effects that can be evaluated:
effect = "Modality"
: This is used when the goal is to exclusively evaluate the effects of multiple modalities.
And, Cochran's Q test (when the number of modalities is greater than 2) or McNemar's test (when the number of modalities is equal to 2) result is reported.
When effect = "Modality"
, interaction
must be set to NULL.
effect = "Reader"
: This is used when the goal is to exclusively evaluate the effects of multiple readers.
And, Cochran's Q test (when the number of modalities is greater than 2) or McNemar's test (when the number of modalities is equal to 2) result is reported.
When effect = "Reader"
, interaction
must be set to NULL.
effect = "Both"
: This is used when the goal is to simultaneously evaluate the effects of multiple modalities and multiple readers.
In this case, interaction
must be specified (TRUE or FALSE).
If one want to evaluate the interaction effect between modality and reader in the conditional logistic regression, set interaction = TRUE
, otherwise interaction = FALSE
.
When interaction = TRUE
, Cochran's Q test result is reported.
However, when interaction = FALSE
, Cochran's Q test or McNemar's test result is not reported.
See Lee et al. (2025) for details.
An object of class MRMCbinary
. The object is a data.frame with the following components:
CLR_sen |
Conditional logistic regression results for sensitivity. |
CLR_LRT_sen |
Likelihood ratio test from the conditional logistic regression results for sensitivity. |
CLR_Score_sen |
Score test from the conditional logistic regression results for sensitivity. |
CLR_Wald_sen |
Wald test from the conditional logistic regression results for sensitivity. |
Q_MN_sen |
Cochran's Q test (when the number of modalities is greater than 2) or McNemar's test (when the number of modalities is equal to 2) result for sensitivity. This is only reported if (1) |
CLR_spe |
Conditional logistic regression results for specificity. |
CLR_LRT_spe |
Likelihood ratio test from the conditional logistic regression results for specificity. |
CLR_Score_spe |
Score test from the conditional logistic regression results for specificity. |
CLR_Wald_spe |
Wald test from the conditional logistic regression results for specificity. |
Q_MN_spe |
Cochran's Q test (when the number of modalities is greater than 2) or McNemar's test (when the number of modalities is equal to 2) result for specificity. This is only reported if (1) |
formula |
Formula used in the conditional logistic regression. |
args |
List of arguments used in the |
n.modality |
Total number of modalities. |
n.reader |
Total number of readers. |
n.case |
Total number of cases. |
effect |
Effect to compare sensitivity and specificity. |
measure |
Diagnostic accuracy measure. |
interaction |
This is only included in the |
reference.Modality |
Reference in variable of the modality identifiers. |
reference.Reader |
Reference in variable of the reader identifiers. |
n.diseased |
The number of diseased cases. If |
n.nondiseased |
The number of non-diseased cases. If |
n.pos.diseased |
The number of test positive cases among diseased cases. If |
n.pos.nondiseased |
The number of test positive cases among non-diseased cases. If |
The results for the MRMCbinary
are printed with the print.MRMCbinary
function.
Also, the results for the MRMCbinary
are summarized with the summary.MRMCbinary
function.
Lee, S., Jang, S., and Lee, W. Evaluating Diagnostic Accuracy of Binary Medical Tests in Multi-reader Multi-case Study.
print.MRMCbinary
, summary.MRMCbinary
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## See unique readers unique(VanDyke$reader) ## See unique modalities unique(VanDyke$treatment) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3) ## Example usage of MRMCbinary function: # When comparing the sensitivities and specificities between modalities modality_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Modality", interaction = NULL, reference.Modality = "1", reference.Reader = NULL) # When comparing the sensitivities and specificities between readers reader_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Reader", interaction = NULL, reference.Modality = NULL, reference.Reader = "1") # When comparing the sensitivities and specificities # between modalities and between readers together # not considering interaction between modalities and readers both_result_wo_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = FALSE, reference.Modality = "1", reference.Reader = "1") # When comparing the sensitivities and specificities # between modalities and between readers together # considering interaction between modalities and readers both_result_with_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = TRUE, reference.Modality = "1", reference.Reader = "1")
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## See unique readers unique(VanDyke$reader) ## See unique modalities unique(VanDyke$treatment) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3) ## Example usage of MRMCbinary function: # When comparing the sensitivities and specificities between modalities modality_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Modality", interaction = NULL, reference.Modality = "1", reference.Reader = NULL) # When comparing the sensitivities and specificities between readers reader_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Reader", interaction = NULL, reference.Modality = NULL, reference.Reader = "1") # When comparing the sensitivities and specificities # between modalities and between readers together # not considering interaction between modalities and readers both_result_wo_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = FALSE, reference.Modality = "1", reference.Reader = "1") # When comparing the sensitivities and specificities # between modalities and between readers together # considering interaction between modalities and readers both_result_with_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = TRUE, reference.Modality = "1", reference.Reader = "1")
MRMCbinary
objectsPrint the results for object of class MRMCbinary
.
## S3 method for class 'MRMCbinary' print(x, ...)
## S3 method for class 'MRMCbinary' print(x, ...)
x |
An object for class |
... |
Further arguments (currently not used). |
Print the results for object of class MRMCbinary
.
From the conditional logistic regression results, "Estimate" corresponds to the log odds ratio and "SE" corresponds to the standard error of the log odds ratio.
No return value, called for side effects.
MRMCbinary
, summary.MRMCbinary
, print
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## Extract unique modalities unique(VanDyke$treatment) ## Extract Unique readers unique(VanDyke$reader) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3) ## Example usage of MRMCbinary function: # When comparing the sensitivities and specificities between modalities modality_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Modality", interaction = NULL, reference.Modality = "1", reference.Reader = NULL) print(modality_result) # When comparing the sensitivities and specificities between readers reader_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Reader", interaction = NULL, reference.Modality = NULL, reference.Reader = "1") print(reader_result) # When comparing the sensitivities and specificities # between modalities and between readers together # not considering interaction between modalities and readers both_result_wo_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = FALSE, reference.Modality = "1", reference.Reader = "1") print(both_result_wo_int) # When comparing the sensitivities and specificities # between modalities and between readers together # considering interaction between modalities and readers both_result_with_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = TRUE, reference.Modality = "1", reference.Reader = "1") print(both_result_with_int)
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## Extract unique modalities unique(VanDyke$treatment) ## Extract Unique readers unique(VanDyke$reader) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3) ## Example usage of MRMCbinary function: # When comparing the sensitivities and specificities between modalities modality_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Modality", interaction = NULL, reference.Modality = "1", reference.Reader = NULL) print(modality_result) # When comparing the sensitivities and specificities between readers reader_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Reader", interaction = NULL, reference.Modality = NULL, reference.Reader = "1") print(reader_result) # When comparing the sensitivities and specificities # between modalities and between readers together # not considering interaction between modalities and readers both_result_wo_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = FALSE, reference.Modality = "1", reference.Reader = "1") print(both_result_wo_int) # When comparing the sensitivities and specificities # between modalities and between readers together # considering interaction between modalities and readers both_result_with_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = TRUE, reference.Modality = "1", reference.Reader = "1") print(both_result_with_int)
SensSpec
objectsPrint the results for object of class SensSpec
.
## S3 method for class 'SensSpec' print(x, ...)
## S3 method for class 'SensSpec' print(x, ...)
x |
An object for class |
... |
Further arguments (currently not used). |
Print the results for object of class SensSpec
.
No return value, called for side effects.
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## Extract unique modalities unique(VanDyke$treatment) ## Extract Unique readers unique(VanDyke$reader) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3) ## Example usage of SensSpec function: senspe_result1 <- SensSpec(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, percentage = FALSE, digits = 3) print(senspe_result1) # Report results as percentage points senspe_result2 <- SensSpec(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, percentage = TRUE, digits = 1) print(senspe_result2)
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## Extract unique modalities unique(VanDyke$treatment) ## Extract Unique readers unique(VanDyke$reader) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3) ## Example usage of SensSpec function: senspe_result1 <- SensSpec(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, percentage = FALSE, digits = 3) print(senspe_result1) # Report results as percentage points senspe_result2 <- SensSpec(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, percentage = TRUE, digits = 1) print(senspe_result2)
SensSpec()
is the function that calculates overall sensitivity and specificity, modality-specific sensitivity and specificity, and modality- and reader-specific sensitivity and specificity.
SensSpec( data, Modality, Reader, Case, D, Y, percentage = FALSE, digits = max(1L, getOption("digits") - 3L) )
SensSpec( data, Modality, Reader, Case, D, Y, percentage = FALSE, digits = max(1L, getOption("digits") - 3L) )
data |
A data frame in which contains the modality identifiers ( |
Modality |
Variable of the modality identifiers. |
Reader |
Variable of the reader identifiers. |
Case |
Variable of the case identifiers. |
D |
Variable of the true disease status. It should be set the value to 1 for cases diseased and to 0 for those non-diseased. |
Y |
Variable of the binary diagnostic test result. It should be set the value to 1 for cases diagnosed as positive and to 0 for those diagnosed as negative. |
percentage |
Whether to report results as decimals or percentage points. Default: |
digits |
Number of significant digits. Default: |
An object of class SensSpec
. The object is a data.frame with the following components:
Overall Result |
Overall sensitivity and specificity |
Modality-specific Result |
Modality-specific sensitivity and specificity |
Reader-specific Modality-specific Result |
Modality- and reader-specific sensitivity and specificity |
digits |
The number of significant digits |
The results for the SensSpec
are printed with the print.SensSpec
function.
Yerushalmy, J. (1947). Statistical Problems in Assessing Methods of Medical Diagnosis, with Special Reference to X-Ray Techniques. Public Health Reports (1896-1970), 62(40), 1432–1449.
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## Extract unique modalities unique(VanDyke$treatment) ## Extract Unique readers unique(VanDyke$reader) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3) ## Example usage of SensSpec function: # Report results as decimals senspe_result1 <- SensSpec(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, percentage = FALSE, digits = 3) # Report results as percentage points senspe_result2 <- SensSpec(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, percentage = TRUE, digits = 1)
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## Extract unique modalities unique(VanDyke$treatment) ## Extract Unique readers unique(VanDyke$reader) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3) ## Example usage of SensSpec function: # Report results as decimals senspe_result1 <- SensSpec(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, percentage = FALSE, digits = 3) # Report results as percentage points senspe_result2 <- SensSpec(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, percentage = TRUE, digits = 1)
MRMCbinary
objectsSummary the results for object of class MRMCbinary
.
## S3 method for class 'MRMCbinary' summary(object, digits = max(1L, getOption("digits") - 3L), ...)
## S3 method for class 'MRMCbinary' summary(object, digits = max(1L, getOption("digits") - 3L), ...)
object |
An object for class |
digits |
Number of significant digits. Default: |
... |
Further arguments (currently not used). |
Summary the results for object of class MRMCbinary
.
From the conditional logistic regression results, the odds ratio, confidence interval of the odds ratio, and P value are reported.
No return value, called for side effects.
MRMCbinary
, print.MRMCbinary
, summary
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## Extract unique modalities unique(VanDyke$treatment) ## Extract Unique readers unique(VanDyke$reader) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3) ## Example usage of MRMCbinary function: # When comparing the sensitivities and specificities between modalities modality_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Modality", interaction = NULL, reference.Modality = "1", reference.Reader = NULL) summary(modality_result, digits = 3) # When comparing the sensitivities and specificities between readers reader_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Reader", interaction = NULL, reference.Modality = NULL, reference.Reader = "1") summary(reader_result, digits = 3) # When comparing the sensitivities and specificities # between modalities and between readers together # not considering interaction between modalities and readers both_result_wo_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = FALSE, reference.Modality = "1", reference.Reader = "1") summary(both_result_wo_int, digits = 3) # When comparing the sensitivities and specificities # between modalities and between readers together # considering interaction between modalities and readers both_result_with_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = TRUE, reference.Modality = "1", reference.Reader = "1") summary(both_result_with_int, digits = 3)
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## Extract unique modalities unique(VanDyke$treatment) ## Extract Unique readers unique(VanDyke$reader) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3) ## Example usage of MRMCbinary function: # When comparing the sensitivities and specificities between modalities modality_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Modality", interaction = NULL, reference.Modality = "1", reference.Reader = NULL) summary(modality_result, digits = 3) # When comparing the sensitivities and specificities between readers reader_result <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Reader", interaction = NULL, reference.Modality = NULL, reference.Reader = "1") summary(reader_result, digits = 3) # When comparing the sensitivities and specificities # between modalities and between readers together # not considering interaction between modalities and readers both_result_wo_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = FALSE, reference.Modality = "1", reference.Reader = "1") summary(both_result_wo_int, digits = 3) # When comparing the sensitivities and specificities # between modalities and between readers together # considering interaction between modalities and readers both_result_with_int <- MRMCbinary(data = VanDyke, Modality = treatment, Reader = reader, Case = case, D = truth, Y = Y, measure = "All", effect = "Both", interaction = TRUE, reference.Modality = "1", reference.Reader = "1") summary(both_result_with_int, digits = 3)
Example data from a study comparing the relative performance of cinematic presentation of magnetic resonance imaging (CINE MRI) to single spin-echo magnetic resonance imaging (SE MRI) for the detection of thoracic aortic dissection (Van Dyke et al., 1993).
VanDyke
VanDyke
A data frame with 1140 rows and 7 variables:
Reader identifier for the five radiologists
Treatment identifier for the two imaging modalities
Case identifiers for 114 cases
Case identifier (cases nested within readers)
Case identifier (cases nested within treatments)
Indicator for thoracic aortic dissection (i.e., true disease status): 1 = performed (i.e., patients with aortic dissection imaged with both SE MRI and CINE MRI) or 0 = not performed (i.e., patients without dissection imaged with both SE MRI and CINE MRI)
Five-point ratings given to case images by the radiologists (i.e., diagnostic test result): 1 = definitely no aortic dissection, 2 = probably no aortic dissection, 3 = unsure about aortic dissection, 4 = probably aortic dissection, or 5 = definitely aortic dissection
This example compares the relative performance of SE MRI with the CINE MRI in detecting thoracic aortic dissection.
There are 45 patients with an aortic dissection and 69 patients without a dissection imaged with both SE MRI and CINE MRI.
One can directly use this data from MRMCaov
package. See Source.
This data are available at https://perception.lab.uiowa.edu and https://github.com/brian-j-smith/MRMCaov/tree/master/data.
Van Dyke, C. W., White, R. D., Obuchowski, N. A., Geisinger, M. A., Lorig, R. J., & Meziane, M. A. (1993). Cine MRI in the diagnosis of thoracic aortic dissection. 79th RSNA Meetings. Chicago, IL, 28.
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## Extract unique modalities unique(VanDyke$treatment) ## Extract Unique readers unique(VanDyke$reader) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3)
## Load example data data(VanDyke) ## Return the first parts of an object head(VanDyke) ## Extract unique modalities unique(VanDyke$treatment) ## Extract Unique readers unique(VanDyke$reader) ## Create binary test results (Y_ijk) VanDyke$Y <- as.numeric(VanDyke$rating >= 3)