Differences

This shows you the differences between two versions of the page.

--- wiki:autolit:screening:inclusionpredictionmodel [2024/07/24 16:49]
jthurnham [Screening Model]
+++ wiki:autolit:screening:inclusionpredictionmodel [2024/09/23 11:59] (current)
jthurnham [Interpreting Robot Screener]
@@ Line 10: / Line 10: @@
 The below guidance is specifically for using Robot Screener, for information on training the model for probability generation only and general information on how the model works see [[wiki:autolit:screening:model|Using and Interpreting the Screening Model.]]
+----
 ===== Robot Screener =====
@@ Line 16: / Line 19: @@
 **Robot Screener may only be turned on in Dual Screening modes** and it's important to note at what stage they are generated and the language used:
-  * Standard Mode: Robot Screener replaces a reviewer in the singular round of Screening based on //Inclusion Probabilities.//
+  * Dual Standard Mode: Robot Screener replaces a reviewer in the singular round of Screening based on //Inclusion Probabilities.//
-  * Two Pass Mode: Robot Screener replaces a reviewer in the Title/Abstract round of Screening only, based on //Advancement Probabilities.//
+  * Dual Two Pass Mode: Robot Screener replaces a reviewer in the Title/Abstract round of Screening only, based on //Advancement Probabilities.//
 {{youtube>9bsA4DMF4aE}}
+----
-===== Robot Screener Validation Studies =====
-Robot Screener has been validated in several published studies assessing its decisions in comparison to human decisions across multiple reviews and review types.
-  * [[https://www.ispor.org/heor-resources/presentations-database/presentation/intl2024-3900/136092|Internal]] validation: Nested Knowledge assessed the Recall and Precision of Robot Screener in 19 projects with over 100,000 cumulative decisions, finding significantly lower Precision than humans (that is, humans correctly exclude studies more often) but significantly higher Recall-- meaning that the Robot Screener misses fewer includable records. In this analysis, Robot Screener was found to have 97.1% Recall.
+===== User Guide =====
-  * [[https://www.ispor.org/heor-resources/presentations-database/presentation/intl2024-3896/139709|External]] validation: Cichewicz et al. assessed diagnostic accuracy across many metrics in 15 projects, finding Robot Screener had significantly lower Precision than humans and no statistical differences in Recall between Robot Screener and humans. Robot Screener had fewer overall False Negatives, but no significant differences were found.
-  * Estimates of [[https://about.nested-knowledge.com/2023/11/10/the-data-is-in-deciding-when-to-automate-screening-in-your-slr/|time savings]] using different modes of Robot Screener have been previously published online.
-You can see a deeper summary of the Validation Studies and their implications [[https://about.nested-knowledge.com/2024/05/28/validation-summary-robot-screeners-performance-in-screening-records-for-systematic-literature-review-and-health-technology-assessment/|here]].
+==== Settings ====
-===== User Guide =====
+To turn on Robot Screener, head to Nest Settings --> Screening Model, toggle on Robot Screener.
-==== Running the Screening Model ====
+{{ :undefined:screenshot_2024-07-24_at_17.53.25.png?nolink |}}
-To learn about configuration settings, which enable you to toggle Manual updating vs. Automatic and Displayed vs. Hidden, see the [[:wiki:autolit:admin:configure#inclusion_prediction_model|Settings page]].
+Not displayed? You must be in a Dual Screening mode to use Robot Screener.
-In its default setting, the Screening Model must be run manually. To do so, click "Train Screening Model" on the Screening panel:
+Want to use Automatic Training for use in manual screening instead? [[wiki:autolit:screening:model|Learn more here.]]
-{{  :undefined:4screen.png?nolink&  }}
+----
-Once the modal opens, click "Train New Model."
+==== Meeting the Threshold ====
+When toggling Robot Screener, you'll be presented with an instructional modal:
+{{ :undefined:screenshot_2024-07-24_at_17.57.44.png?nolink |}}
+Highlighted in red are the requirements for training the screening model, and the actual numbers based on the progress made in your nest. You will not be able to turn on Robot Screener until these minimum requirements are met.
 <WRAP center round important 60%>
-To provide the model with sufficient information to begin understanding your review, we require **50 total adjudicated screening decisions with 10 advancements or inclusions**  before the model can be trained. If there is insufficient evidence to train the model, complete more adjudicated screening (2 reviewers and 1 adjudicator) until the "Train New Model" button becomes available.
+Before Robot Screener can be turned on, 50 adjudicated screening decisions with 10 advancements/inclusions must be made. After this is met and the Robot is turned on, it will continue to train on further adjudicated screening decisions made.
 </WRAP>
+Once trained and turned on, the Robot is assigning both inclusion probabilities and actual screening decisions to the remainder of records in the queue. Currently, Robot Screener does not assign exclusion reasons, so decisions are displayed as "Advance"/"Include" or "Robot Excluded". The records that Robot Screener makes a decision on will still need an additional human to screen these records as a second reviewer, and a human adjudicator to make the final decision on these records. This means that each record will always have two pairs of eyes to review.
-It may take a minute to train, after which it will populate a histogram on the left. From then on, each record will show a probability of inclusion or advancement:
+----
+==== Interpreting Robot Screener ====
+At any time, you may wish to view how the screening model is performing. To view the model performance, navigate to Nested Settings --> Screening model --> View Screening model.
+{{ :undefined:screenshot_2024-07-24_at_18.13.41.png?nolink |}}
+This will display a histogram under the "Predictions" tab, a table of various Cross Validation statistics displaying history of previous trainings of the model, and an explanation as to how to interpret these values. Note: the history of trained models is displayed for informational purposes only, and not versions that can be reverted back to. Retraining the model does not guarantee improved statistics and performance.
+You can also view the Robot Screener recommendations in the Screening model modal. Select "Advance"/"Include" to view studies the Robot has advanced/included, or "Exclude" for excluded studies-- these are both shortcuts that take you to Study Inspector to show you the corresponding subsets of studies. Otherwise, the filter can be manually added from Study Inspector. From this modal, you can also delete the model if you wish to start again from scratch.
+With Predictions toggled:
+{{ :undefined:screenshot_2024-09-23_at_12.56.51.png?nolink |}}
+With Cross Validation toggled:
+{{ :undefined:screenshot_2024-09-23_at_12.57.00.png?nolink |}}
+[[wiki:autolit:screening:model#interpreting_the_model|Learn more about interpreting the model, its performance and how it works here.]]
+----
+==== Improving Robot Screener ====
+The best way to improve Robot Screener, is to adjudicate records, since these are the decisions it trains on. We recommend, if you can, have your adjudicator make their final decisions on the Adjudicate Screening page after every 50 studies are screened for best model performance. For reference, the following is what adjudicators will see for records that have one human and one Robot Screener decision applied:
+{{ :undefined:screenshot_2024-07-24_at_18.18.39.png?nolink |}}
+----
+===== Robot Screener Validation Studies =====
+Robot Screener has been validated in several published studies assessing its decisions in comparison to human decisions across multiple reviews and review types.
+  * [[https://www.ispor.org/heor-resources/presentations-database/presentation/intl2024-3900/136092|Internal]] validation: Nested Knowledge assessed the Recall and Precision of Robot Screener in 19 projects with over 100,000 cumulative decisions, finding significantly lower Precision than humans (that is, humans correctly exclude studies more often) but significantly higher Recall-- meaning that the Robot Screener misses fewer includable records. In this analysis, Robot Screener was found to have 97.1% Recall.
+  * [[https://www.ispor.org/heor-resources/presentations-database/presentation/intl2024-3896/139709|External]] validation: Cichewicz et al. assessed diagnostic accuracy across many metrics in 15 projects, finding Robot Screener had significantly lower Precision than humans and no statistical differences in Recall between Robot Screener and humans. Robot Screener had fewer overall False Negatives, but no significant differences were found.
+  * Estimates of [[https://about.nested-knowledge.com/2023/11/10/the-data-is-in-deciding-when-to-automate-screening-in-your-slr/|time savings]] using different modes of Robot Screener have been previously published online.
+You can see a deeper summary of the Validation Studies and their implications [[https://about.nested-knowledge.com/2024/05/28/validation-summary-robot-screeners-performance-in-screening-records-for-systematic-literature-review-and-health-technology-assessment/|here]].
-{{  :undefined:2screen.png?nolink&  }}

Nested Knowledge

User Tools

Site Tools

Differences

Page Tools