This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
wiki:autolit:screening:inclusionpredictionmodel [2024/07/24 16:17] jthurnham [Screening Model] |
wiki:autolit:screening:inclusionpredictionmodel [2024/09/23 11:59] (current) jthurnham [Interpreting Robot Screener] |
||
---|---|---|---|
Line 3: | Line 3: | ||
The Screening Model uses AI to learn from screening decisions within a specific nest, generating inclusion probabilities based on configuration. | The Screening Model uses AI to learn from screening decisions within a specific nest, generating inclusion probabilities based on configuration. | ||
- | You may use the screening model in two ways: | + | You may use the screening model in **two ways:** |
- | * **Generate inclusion probabilities** to be displayed on records and assist in your own manual screening. | + | * [[wiki: |
* //Dual modes only:// Turn on **Robot Screener** to replace an expert reviewer, which makes decisions based on these probabilities. | * //Dual modes only:// Turn on **Robot Screener** to replace an expert reviewer, which makes decisions based on these probabilities. | ||
Line 10: | Line 10: | ||
The below guidance is specifically for using Robot Screener, for information on training the model for probability generation only and general information on how the model works see [[wiki: | The below guidance is specifically for using Robot Screener, for information on training the model for probability generation only and general information on how the model works see [[wiki: | ||
- | ===== Robot Screener ===== | ||
- | The Screening Model can be used to power AI-assisted screening, replacing one expert in Dual Screening processes: | ||
- | {{youtube> | + | ---- |
- | When selecting a mode, note that in most cases, when employing Dual Two Pass Mode, **the Robot Screener | + | ===== Robot Screener |
- | When used in "Robot Screener" | + | The Screening Model can be used to power AI-assisted screening, replacing one expert |
- | ===== Robot Screener | + | **Robot Screener |
+ | * Dual Standard Mode: Robot Screener replaces a reviewer in the singular round of Screening based on //Inclusion Probabilities.// | ||
+ | * Dual Two Pass Mode: Robot Screener replaces a reviewer in the Title/ | ||
- | Robot Screener has been validated in several published studies assessing its decisions in comparison to human decisions across multiple reviews and review types. | + | {{youtube> |
- | * [[https:// | ||
- | * [[https:// | ||
- | * Estimates of [[https:// | ||
- | You can see a deeper summary of the Validation Studies and their implications [[https:// | + | ---- |
- | ===== User Guide ===== | ||
- | ==== Running the Screening Model ==== | ||
- | To learn about configuration settings, which enable you to toggle Manual updating vs. Automatic and Displayed vs. Hidden, see the [[: | ||
- | In its default setting, the Screening Model must be run manually. To do so, click "Train Screening Model" on the Screening panel: | ||
- | {{ : | ||
- | Once the modal opens, click "Train New Model." | + | ===== User Guide ===== |
- | <WRAP center round important 60%> | + | ==== Settings ==== |
- | To provide the model with sufficient information to begin understanding your review, we require **50 total adjudicated screening decisions with 10 advancements or inclusions** | + | |
- | </ | + | |
+ | To turn on Robot Screener, head to Nest Settings --> Screening Model, toggle on Robot Screener. | ||
- | It may take a minute to train, after which it will populate a histogram on the left. From then on, each record will show a probability of inclusion or advancement: | + | {{ :undefined: |
- | {{ : | + | Not displayed? You must be in a Dual Screening mode to use Robot Screener. |
- | ==== Interpreting the Model ==== | + | Want to use Automatic Training for use in manual screening instead? [[wiki: |
- | Once the Model is trained, you should see a graph where Included or Advanced, Excluded, and Unscreened records are represented by green, red, and purple curves, respectively: | + | ---- |
- | {{ : | + | ==== Meeting the Threshold ==== |
- | Odds of inclusion/ | + | When toggling Robot Screener, you'll be presented with an instructional modal: |
- | You can see the accuracy in the modal after the model is trained. In the Cross Validation tab, several statistics are shown. Scores of Recall and Accuracy can be used to interpret how the model will perform on the remaining records. High recall (0.7/70%+) indicates that the model will less frequently exclude relevant records, meaning higher performance. Similarly, accuracy indicates how correct the model' | + | {{ : |
- | {{ : | + | Highlighted in red are the requirements for training the screening model, and the actual numbers based on the progress made in your nest. You will not be able to turn on Robot Screener until these minimum requirements are met. |
- | ==== Implications for Screening ==== | + | <WRAP center round important 60%> |
+ | Before Robot Screener can be turned on, 50 adjudicated screening decisions with 10 advancements/ | ||
+ | </ | ||
- | Inclusion Probability generated from the Screening model is also available as a filter | + | Once trained and turned on, the Robot is assigning both inclusion probabilities and actual screening decisions to the remainder of records |
- | ===== Model Performance ===== | + | ---- |
- | ==== Our Philosophy ==== | ||
- | Screening is a complex task that relies on human expertise. Our model may stumble due to: | + | ==== Interpreting Robot Screener ==== |
- | * Insufficient training examples (usually included/ | + | At any time, you may wish to view how the screening |
- | * Data not available | + | |
- | * Weak signal amongst available predictors against protocol | + | |
- | **For these reasons, we recommend using the model to augment your screening workflow, not fully automate it. ** | + | {{ : |
- | How can it augment your screening? | + | This will display a histogram under the " |
- | * Excluding clearly low-relevancy records | + | You can also view the Robot Screener recommendations in the Screening model modal. Select " |
- | * Raising high relevancy records | + | |
- | **Our model errs towards including/ | + | With Predictions toggled: |
- | ==== Testing out the model ==== | + | {{ : |
- | In an internal study, Nested Knowledge ran the model across several hundred SLR projects, finding the following cumulative accuracy statistics: | + | With Cross Validation toggled: |
- | === Standard Screening === | + | {{ : |
- | * Area Under the [Receiver Operating Characteristic] Curve (AUC): 0.88 | + | [[wiki:autolit:screening:model# |
- | * Classification Accuracy: 0.92 | + | |
- | * Recall: 0.76 | + | |
- | * Precision: 0.40 | + | |
- | * F1: 0.51 | + | |
- | === Two Pass Screening === | + | ---- |
- | In two pass screening, the model predicts advancement of a record from abstract screening to full text screening. Given that advancement rates are typically higher than inclusion rates, the model has more positive training examples, and demonstrates improved recall. | ||
- | * AUC: 0.88 | + | ==== Improving Robot Screener ==== |
- | * Classification Accuracy: 0.93 | + | |
- | * Recall: 0.81 | + | |
- | * Precision: 0.44 | + | |
- | * F1: 0.56 | + | |
- | Following our philosophy, recall | + | The best way to improve Robot Screener, is to adjudicate |
- | For comparison purposes, our study found human reviewer recall (relative to the adjudicated decision) was 85% in the average nest. Our models are within 4 & 9 points of human performance on this most critical measure. | + | {{ : |
- | ==== Analyzing Your Nest ==== | + | ---- |
+ | ===== Robot Screener Validation Studies ===== | ||
- | When you train a new model, we generate k-fold cross validation performance measures using the same model hyperparameters the final model is trained with. These performance measures typically provide a lower bound on the performance you can expect from the model on records not yet screened | + | Robot Screener has been validated |
- | While we cannot guarantee performance improvement, below is some rough empirical data for how you might expect performance measures to improve as you screen | + | * [[https:// |
+ | * [[https:// | ||
+ | * Estimates of [[https:// | ||
- | === Timing of Model Training === | + | You can see a deeper summary |
- | + | ||
- | In general, as you screen more records, the better the model will perform. Of course, you want to use the model before you’ve screened every record! | + | |
- | + | ||
- | To provide the model with sufficient information to begin understanding your review, we require 50 total screens and 10 inclusions/ | + | |
- | + | ||
- | As the graph below shows, AUC and recall can grow on a relatively sharp curve early in your review. The curve begins to flatten around 20-30% | + | |
- | + | ||
- | {{ | + | |
- | + | ||
- | ===== How the Screening Model Works ===== | + | |
- | + | ||
- | At a high level, the model is a Decision Tree- a series of Yes/No questions | + | |
- | + | ||
- | In more detail, the model is a gradient-boosted decision tree ensemble. Its hyperparameters, | + | |
- | + | ||
- | ==== What data does the model use? ==== | + | |
- | + | ||
- | The model uses the following data from your records | + | |
- | + | ||
- | * Bibliographic data | + | |
- | * Time since publication of the record | + | |
- | * Page count | + | |
- | * Keywords/ | + | |
- | * Abstract Content | + | |
- | * N-grams | + | |
- | * OpenAI text embedding (ada-002) | + | |
- | * Citation Counts from Scite, accessed using the DOI | + | |
- | * Number of citing publications | + | |
- | * Number of supporting citation statements | + | |
- | * Number of contrasting citation statements | + | |
- | + | ||
- | Often some of this data will be missing for records; it is imputed as if the record is approximately typical to other records in the nest. | + | |