Nested Knowledge

Bringing Systematic Review to Life

User Tools

Site Tools

Trace:
wiki:autolit:screening:robot

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
wiki:autolit:screening:robot [2023/05/17 23:13]
jthurnham [Enabling Robot Screener] 		wiki:autolit:screening:robot [2024/03/06 02:40]
kevinkallmes
Line 1: Line 1:
-====== Robot Screener ======+===== Guidance on Robot Screener in Dual Two Pass Mode =====
  
-Robot Screener is //type// of Dual Screening where one of the traditionally human reviewers is replaced by an automated substitute.+To clarify, the recommended practice is to **use Robot Screener as a reviewer in Abstract Screening** when in Dual Two Pass mode, and use fully manual Full Text screening.
  
-===== Enabling Robot Screener ===== +For projects with large numbers of Full Text records, Robot Screener //can// be used for both Abstract Screening and Full Text Screening but the requirements must be met separately.
- +
-Robot Screener may be enabled from Screening Settings. When Dual Screening mode (standard or two pass) is switched on, Robot Screener can be switched on under "Screening Model". Before Robot Screener can be enabled, the nest must: +
- +
-  * Be configured for [[wiki:autolit:screening:dual|Dual Screening]] or [[wiki:autolit:screening:dualtwopass|Dual Two Pass Screening]] +
-  * And contain **50**  adjudicated //records //  (i.e. having an adjudicated screening decision, abstract or full text) +
-  * And contain **10**  included records (or abstract advanced for two pass mode) +
- +
-The latter two requirements help ensure a minimum model accuracy; however, the model will usually be suboptimal with this volume of training data, and typically improves as more records are screened. +
- +
-By enabling Robot Screener, automatic updating to the screening model is made non-optional, as is the hiding of inclusion probabilities. This avoids biasing human reviewers with data already being used by Robot Screener. +
- +
-When switching on Robot Screener, you will see the below modal. In red are the actual numbers of adjudicated and included or advanced records compared to the numbers you need to meet in order to Enable Robot Screener: +
- +
-{{ :undefined:robo.png?nolink |}} +
- +
-===== In Action ===== +
- +
-==== Which Records ==== +
- +
-Robot Screener automatically adds a single reviewer-level screening decision to records with: +
- +
-  * Fewer than 2 reviewer-level screening decisions +
-  * No adjudicated screening decision +
- +
-Records it excludes will assigned the exclusion reason ''Robot Excluded''  in all cases. +
- +
-==== When ==== +
- +
-Robot Screener adds/updates its screening decisions when: +
- +
-  * Robot Screener is enabled +
-  * A new Screening model is trained +
-  * New records are imported into your nest +
-  * 10 additional records have been adjudicated, since the model was last trained +
- +
-As pointed out above, Robot Screener will not modify its decision on a record after that record has been adjudicated. Prior to adjudication, its decision on a record may be modified, reflecting more information available to it & improvement in its accuracy. +
- +
-===== Usage Guidance ===== +
- +
-Robot Screener, by accuracy, is not a replacement for a human reviewer. It should typically be used for reviews where budget, team size, or time available are restricted. Some additional usage tips: +
- +
-  * While Robot Screener may be enabled at 50 adjudicated / 10 included records, we typically advise not enabling it until a cross validation recall of 0.7 or greater is achieved. +
-  * Only use the "Auto-adjudicate" button when you are confident in trusting a single human review with only the Robot as a 'check'. This may mean manually reviewing records that the human and Robot agree on, especially early on. This will improve the quality of your review, reduce later reworks, and produce better training data for Robot Screener. +
- +
-Learn more about [[wiki:autolit:screening:inclusionpredictionmodel|the Screening model that powers Robot Screener here.]] +
- +
-===== Why can't I turn on Robot Screener yet? ===== +
- +
-It may be due to not having enough screening decisions in your nest. [[wiki:autolit:screening:robot#enabling_robot_screener|See the requirements above.]] +
- +
-If you are in Dual Two Pass Screening mode, Robot Screener can be used for //both// Abstract Screening and Full Text Screening but the requirements must be met separately.+
  
    * Abstract Screening requires **50**  abstract adjudicated records, either advanced or excluded under Abstract Screening –> Adjudicate Screening, and **10**  advanced records    * Abstract Screening requires **50**  abstract adjudicated records, either advanced or excluded under Abstract Screening –> Adjudicate Screening, and **10**  advanced records
wiki/autolit/screening/robot.txt · Last modified: 2024/03/06 02:40 by kevinkallmes

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki