This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
wiki:autolit:screening:inclusionpredictionmodel [2024/07/24 17:08] jthurnham [Running the Screening Model] |
wiki:autolit:screening:inclusionpredictionmodel [2024/09/23 11:59] (current) jthurnham [Interpreting Robot Screener] |
||
---|---|---|---|
Line 10: | Line 10: | ||
The below guidance is specifically for using Robot Screener, for information on training the model for probability generation only and general information on how the model works see [[wiki: | The below guidance is specifically for using Robot Screener, for information on training the model for probability generation only and general information on how the model works see [[wiki: | ||
+ | |||
+ | |||
+ | ---- | ||
===== Robot Screener ===== | ===== Robot Screener ===== | ||
Line 40: | Line 43: | ||
Want to use Automatic Training for use in manual screening instead? [[wiki: | Want to use Automatic Training for use in manual screening instead? [[wiki: | ||
+ | |||
+ | ---- | ||
==== Meeting the Threshold ==== | ==== Meeting the Threshold ==== | ||
Line 53: | Line 58: | ||
</ | </ | ||
- | Once trained and turned on, the Robot is assigning both inclusion probabilities and actual screening decisions to the remainder of records in the queue. The remainder of records will need an additional human to screen these records as a second reviewer, and a human adjudicator to make the final decision on these records. This means that each record will always have two pairs of eyes to review. | + | Once trained and turned on, the Robot is assigning both inclusion probabilities and actual screening decisions to the remainder of records in the queue. Currently, Robot Screener does not assign exclusion reasons, so decisions are displayed as " |
+ | |||
+ | ---- | ||
+ | |||
+ | |||
+ | ==== Interpreting Robot Screener ==== | ||
+ | |||
+ | At any time, you may wish to view how the screening model is performing. To view the model performance, | ||
+ | |||
+ | {{ : | ||
+ | |||
+ | This will display a histogram under the " | ||
+ | |||
+ | You can also view the Robot Screener recommendations in the Screening model modal. Select " | ||
+ | |||
+ | With Predictions toggled: | ||
+ | |||
+ | {{ : | ||
+ | |||
+ | With Cross Validation toggled: | ||
+ | |||
+ | {{ : | ||
+ | |||
+ | [[wiki: | ||
+ | |||
+ | ---- | ||
+ | |||
+ | |||
+ | ==== Improving Robot Screener ==== | ||
+ | |||
+ | The best way to improve Robot Screener, is to adjudicate records, since these are the decisions it trains on. We recommend, if you can, have your adjudicator make their final decisions on the Adjudicate Screening page after every 50 studies are screened for best model performance. For reference, the following is what adjudicators will see for records that have one human and one Robot Screener decision applied: | ||
+ | |||
+ | {{ : | ||
+ | |||
+ | ---- | ||
===== Robot Screener Validation Studies ===== | ===== Robot Screener Validation Studies ===== | ||