-
Notifications
You must be signed in to change notification settings - Fork 11
/
README.Rmd
184 lines (132 loc) · 4 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
knitr::opts_chunk$set(warning = FALSE, message = FALSE)
```
# runway
<!-- badges: start -->
[](https://www.tidyverse.org/lifecycle/#maturing)
<!-- badges: end -->
The goal of runway is to generate statistics and plots to calculate discrimination, calibration, and decision curves for prediction models.
## Why is it called runway?
Because you can use it to visually compare models.
Sometimes your models look quite different.
Other times your models look the same...
## Installation
You can install `runway` from GitHub with:
```{r eval=FALSE}
remotes::install_github('ML4LHS/runway')
```
## Load the package
First, load the package.
```{r}
library(runway)
```
## Sample datasets
Runway comes with two sample datasets.
```{r}
data(single_model_dataset)
head(single_model_dataset)
data(multi_model_dataset)
head(multi_model_dataset)
```
## Evaluating a single model
### Threshold-performance plot (single model)
```{r}
threshperf_plot(single_model_dataset,
outcome = 'outcomes',
positive = '1',
prediction = 'predictions')
```
### Calibration plot with 10 bins (single model)
Note: 10 bins is the default.
```{r}
cal_plot(single_model_dataset,
outcome = 'outcomes',
positive = '1',
prediction = 'predictions')
```
### Calibration plot with 5 bins (single model)
```{r}
cal_plot(single_model_dataset,
outcome = 'outcomes',
positive = '1',
prediction = 'predictions',
n_bins = 5)
```
### Calibration plot with loess curve (single model)
```{r}
cal_plot(single_model_dataset,
outcome = 'outcomes',
positive = '1',
prediction = 'predictions',
n_bins = 0,
show_loess = TRUE)
```
## Comparing multiple models
### Threshold-performance plot (multiple models)
```{r}
threshperf_plot_multi(multi_model_dataset,
outcome = 'outcomes',
positive = '1',
prediction = 'predictions',
model = 'model_name')
```
### Calibration plot with 10 bins (multiple models)
Note: 10 bins is the default.
```{r}
cal_plot_multi(multi_model_dataset,
outcome = 'outcomes',
positive = '1',
prediction = 'predictions',
model = 'model_name')
```
## Calibration plot with 5 bins (multiple models)
```{r}
cal_plot_multi(multi_model_dataset,
outcome = 'outcomes',
positive = '1',
prediction = 'predictions',
model = 'model_name',
n_bins = 5)
```
## Calibration plot with loess curve (multiple models)
Unlike single calibration plots, the choice of binned calibration and loess calibration are mutually exclusive. To show less curves, you must set `show_loess` to `TRUE` *and* `n_bins` to `0`.
```{r}
cal_plot_multi(multi_model_dataset,
outcome = 'outcomes',
positive = '1',
prediction = 'predictions',
model = 'model_name',
n_bins = 0,
show_loess = TRUE)
```
## ROC curve w/CI
```{r}
roc_plot(single_model_dataset,
outcome = 'outcomes',
positive = '1',
prediction = 'predictions',
ci = TRUE,
plot_title = 'Single ROC curve w/CI ribbon')
```
## Multiple ROC curves w/CI ribbons
```{r}
roc_plot_multi(multi_model_dataset,
outcome = 'outcomes',
positive = '1',
prediction = 'predictions',
model = 'model_name',
ci = TRUE,
plot_title = 'Multiple model ROC curves w/CI ribbons')
```