This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
wiki:autolit:screening:inclusionpredictionmodel [2024/07/24 16:23] jthurnham |
wiki:autolit:screening:inclusionpredictionmodel [2024/09/23 11:59] (current) jthurnham [Interpreting Robot Screener] |
||
---|---|---|---|
Line 3: | Line 3: | ||
The Screening Model uses AI to learn from screening decisions within a specific nest, generating inclusion probabilities based on configuration. | The Screening Model uses AI to learn from screening decisions within a specific nest, generating inclusion probabilities based on configuration. | ||
- | You may use the screening model in two ways: | + | You may use the screening model in **two ways:** |
- | | + | * [[wiki: |
* //Dual modes only:// Turn on **Robot Screener** to replace an expert reviewer, which makes decisions based on these probabilities. | * //Dual modes only:// Turn on **Robot Screener** to replace an expert reviewer, which makes decisions based on these probabilities. | ||
Line 10: | Line 10: | ||
The below guidance is specifically for using Robot Screener, for information on training the model for probability generation only and general information on how the model works see [[wiki: | The below guidance is specifically for using Robot Screener, for information on training the model for probability generation only and general information on how the model works see [[wiki: | ||
+ | |||
+ | |||
+ | ---- | ||
===== Robot Screener ===== | ===== Robot Screener ===== | ||
- | The Screening Model can be used to power AI-assisted screening, replacing one expert in Dual Screening processes: | + | The Screening Model can be used to power AI-assisted screening, replacing one expert in Dual Screening processes. |
+ | |||
+ | **Robot Screener may only be turned on in Dual Screening modes** and it's important to note at what stage they are generated and the language used: | ||
+ | * Dual Standard Mode: Robot Screener replaces a reviewer in the singular round of Screening based on //Inclusion Probabilities.// | ||
+ | * Dual Two Pass Mode: Robot Screener replaces a reviewer in the Title/ | ||
{{youtube> | {{youtube> | ||
- | When selecting a mode, note that in most cases, when employing Dual Two Pass Mode, **the Robot Screener should replace an expert reviewer only for the Abstract stage of screening **, as the model itself is trained on and screens based on Abstract content. Using the model in this way provides Advancement probabilities (in effect, relevancy scores) to each record. | ||
- | When used in "Robot Screener" | + | ---- |
- | ===== Robot Screener Validation Studies ===== | ||
- | Robot Screener has been validated in several published studies assessing its decisions in comparison to human decisions across multiple reviews and review types. | ||
- | * [[https:// | ||
- | * [[https:// | ||
- | * Estimates of [[https:// | ||
- | You can see a deeper summary of the Validation Studies and their implications [[https:// | ||
===== User Guide ===== | ===== User Guide ===== | ||
- | ==== Running the Screening Model ==== | + | ==== Settings |
- | To learn about configuration settings, which enable you to toggle Manual updating vs. Automatic and Displayed vs. Hidden, see the [[: | + | To turn on Robot Screener, head to Nest Settings |
- | In its default setting, the Screening Model must be run manually. To do so, click "Train Screening Model" on the Screening panel: | + | {{ :undefined: |
- | {{ : | + | Not displayed? You must be in a Dual Screening mode to use Robot Screener. |
- | Once the modal opens, click "Train New Model." | + | Want to use Automatic Training for use in manual screening instead? [[wiki: |
+ | |||
+ | ---- | ||
+ | |||
+ | ==== Meeting | ||
+ | |||
+ | When toggling Robot Screener, you'll be presented with an instructional | ||
+ | |||
+ | {{ : | ||
+ | |||
+ | Highlighted in red are the requirements for training the screening model, and the actual numbers based on the progress made in your nest. You will not be able to turn on Robot Screener until these minimum requirements are met. | ||
<WRAP center round important 60%> | <WRAP center round important 60%> | ||
- | To provide the model with sufficient information to begin understanding your review, we require **50 total adjudicated screening decisions with 10 advancements | + | Before Robot Screener can be turned on, 50 adjudicated screening decisions with 10 advancements/inclusions |
</ | </ | ||
+ | Once trained and turned on, the Robot is assigning both inclusion probabilities and actual screening decisions to the remainder of records in the queue. Currently, Robot Screener does not assign exclusion reasons, so decisions are displayed as " | ||
- | It may take a minute | + | ---- |
+ | |||
+ | |||
+ | ==== Interpreting Robot Screener ==== | ||
+ | |||
+ | At any time, you may wish to view how the screening model is performing. To view the model performance, navigate to Nested Settings --> Screening model --> View Screening model. | ||
+ | |||
+ | {{ : | ||
+ | |||
+ | This will display | ||
+ | |||
+ | You can also view the Robot Screener recommendations in the Screening model modal. Select " | ||
+ | |||
+ | With Predictions toggled: | ||
+ | |||
+ | {{ : | ||
+ | |||
+ | With Cross Validation toggled: | ||
+ | |||
+ | {{ : | ||
+ | |||
+ | [[wiki: | ||
+ | |||
+ | ---- | ||
+ | |||
+ | |||
+ | ==== Improving Robot Screener ==== | ||
+ | |||
+ | The best way to improve Robot Screener, is to adjudicate records, since these are the decisions it trains | ||
+ | |||
+ | {{ : | ||
+ | |||
+ | ---- | ||
+ | ===== Robot Screener Validation Studies ===== | ||
+ | |||
+ | Robot Screener has been validated in several published studies assessing its decisions in comparison to human decisions across multiple reviews and review types. | ||
+ | |||
+ | * [[https:// | ||
+ | * [[https:// | ||
+ | * Estimates of [[https:// | ||
+ | |||
+ | You can see a deeper summary | ||
- | {{ : | ||