Differences

This shows you the differences between two versions of the page.

--- wiki:autolit:screening:robot [2023/04/10 17:09]
jthurnham [Robot Screener]
+++ wiki:autolit:screening:robot [2024/03/06 02:40] (current)
kevinkallmes
@@ Line 1: / Line 1: @@
-====== Robot Screener ======
+===== Guidance on Robot Screener in Dual Two Pass Mode =====
-Robot Screener is //type// of Dual Screening where an automated substitute for one of the human reviewers traditionally involved in dual screening.
+To clarify, the recommended practice is to **use Robot Screener as a reviewer in Abstract Screening** when in Dual Two Pass mode, and use fully manual Full Text screening.
-===== Enabling Robot Screener =====
+For projects with large numbers of Full Text records, Robot Screener //can// be used for both Abstract Screening and Full Text Screening but the requirements must be met separately.
-Robot Screener may be enabled from Screening Settings, under "Inclusion Modeling". Before Robot Screener can be enabled, the nest must:
+   * Abstract Screening requires **50**  abstract adjudicated records, either advanced or excluded under Abstract Screening –> Adjudicate Screening, and **10**  advanced records
+      * These actions must be performed before Robot Screener can be turned on for Abstract Screening
-  * Be configured for dual screening
+  * Full Text Screening requires **50**  full text adjudicated records, either included or excluded under Full Text Screening –> Adjudicate Screening, and **10**  included records
-  * And contain **50**  adjudicated //records //  (i.e. having a final screening decision)
+      * These actions must be performed before Robot Screener can be turned on for Full Text Screening
-  * And contain **5**  included records
-The latter two requirements help ensure a minimum model accuracy; however, the model will usually be suboptimal with this volume of training data, and typically improves as more records are screened.
-By enabling Robot Screener, automatic updating to the inclusion model is made non-optional, as is the hiding of inclusion probabilities. This avoids biasing human reviewers with data already being used by Robot Screener.
-===== In Action =====
-==== Which Records ====
-Robot Screener automatically adds a reviewer-level screening decision to records with:
-  * Fewer than 2 reviewer-level screening decisions
-  * No adjudicated screening decision
-Records it excludes will assigned the exclusion reason ''Robot Excluded''  in all cases.
-==== When ====
-Robot Screener adds/updates its screening decisions when:
-  * Robot Screener is enabled
-  * A new inclusion model is trained
-  * New records are imported into your nest
-  * 10 addditional records have been adjudicated, since the model was last trained
-As pointed out above, Robot Screener will not modify its decision on a record after that record has been adjudicated. Prior to adjudication, its decision on a record may be modified, reflecting more information available to it & improvement in its accuracy.
-===== Usage Guidance =====
-Robot Screener, by accuracy, is not a replacement for a human reviewer. It should typically be used for reviews where budget, team size, or time available are restricted. Some additional usage tips:
-  * While Robot Screener may be enabled at 50 adjudicated / 5 included records, we typically advise not enabling it until a cross validation AUC of 0.7 or greater is achieved.
-  * Avoid use of the "Auto-adjudicate" button, especially early on. The model (and human reviewer) should be subject to scrutiny. This will improve the quality of your review, reduce later reworks, and produce better training data for Robot Screener.

Nested Knowledge

User Tools

Site Tools

Differences

Page Tools