This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
wiki:autolit:screening:inclusionpredictionmodel [2023/05/17 22:20] jthurnham |
wiki:autolit:screening:inclusionpredictionmodel [2024/03/06 02:59] (current) kevinkallmes [Robot Screener] |
||
---|---|---|---|
Line 3: | Line 3: | ||
The Screening Model uses AI to learn from screening decisions within a specific nest, predicting inclusion (standard screening) or abstract advancement (two pass screening) probabilities based on configuration. Then it automatically re-orders studies in Screening so that the most likely to be included/ | The Screening Model uses AI to learn from screening decisions within a specific nest, predicting inclusion (standard screening) or abstract advancement (two pass screening) probabilities based on configuration. Then it automatically re-orders studies in Screening so that the most likely to be included/ | ||
- | It also works to power [[: | + | ===== Robot Screener ===== |
+ | |||
+ | The Screening Model can be used to power AI-assisted screening, replacing one expert in Dual Screening processes: | ||
+ | |||
+ | {{youtube> | ||
+ | |||
+ | When selecting a mode, note that in most cases, when employing Dual Two Pass Mode, **the Robot Screener should replace an expert reviewer only for the Abstract stage of screening **, as the model itself is trained on and screens based on Abstract content. Using the model in this way provides Advancement probabilities (in effect, relevancy scores) to each record. | ||
+ | |||
+ | See here for full [[: | ||
===== User Guide ===== | ===== User Guide ===== | ||
Line 31: | Line 39: | ||
You can see the accuracy in the modal after the model is trained. In the Cross Validation tab, several statistics are shown. Scores of Recall and Accuracy can be used to interpret how the model will perform on the remaining records. High recall (0.7/70%+) indicates that the model will less frequently exclude relevant records, meaning higher performance. Similarly, accuracy indicates how correct the model' | You can see the accuracy in the modal after the model is trained. In the Cross Validation tab, several statistics are shown. Scores of Recall and Accuracy can be used to interpret how the model will perform on the remaining records. High recall (0.7/70%+) indicates that the model will less frequently exclude relevant records, meaning higher performance. Similarly, accuracy indicates how correct the model' | ||
- | {{ : | + | {{ : |
==== Implications for Screening ==== | ==== Implications for Screening ==== | ||
Line 57: | Line 65: | ||
==== Testing out the model ==== | ==== Testing out the model ==== | ||
+ | |||
+ | In an internal study, Nested Knowledge ran the model across several hundred SLR projects, finding the following cumulative accuracy statistics: | ||
=== Standard Screening === | === Standard Screening === | ||
- | * AUC: 0.88 | + | * Area Under the [Receiver Operating Characteristic] Curve (AUC): 0.88 |
* Classification Accuracy: 0.92 | * Classification Accuracy: 0.92 | ||
* Recall: 0.76 | * Recall: 0.76 | ||
Line 119: | Line 129: | ||
Often some of this data will be missing for records; it is imputed as if the record is approximately typical to other records in the nest. | Often some of this data will be missing for records; it is imputed as if the record is approximately typical to other records in the nest. | ||
- | |||