Nested Knowledge

Bringing Systematic Review to Life

User Tools

Site Tools

Trace:
wiki:autolit:tagging:tag_recommendations

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
wiki:autolit:tagging:tag_recommendations [2023/10/17 18:16]
kevinkallmes 		wiki:autolit:tagging:tag_recommendations [2024/01/27 19:28] (current)
jthurnham [Internal Validation Testing of Abstract Smart Tag Recommendations]
Line 2: Line 2:
  
 There are two Tag Recommendation modes: Standard and Smart (Beta). In short, **standard tag recommendations** use keyword searches and is available to all users while **Smart Tag Recommendations** uses OpenAI, a Large Language Model that can read and answer questions, but is //only available to enterprise customers.// There are two Tag Recommendation modes: Standard and Smart (Beta). In short, **standard tag recommendations** use keyword searches and is available to all users while **Smart Tag Recommendations** uses OpenAI, a Large Language Model that can read and answer questions, but is //only available to enterprise customers.//
- 
 ===== Standard Tag Recommendations ===== ===== Standard Tag Recommendations =====
  
-Whether your nest is in Standard or Form-based tagging mode, Nested Knowledge automatically generates Standard Tag Recommendations for every included study. Once you have uploaded the Full Text, you can apply tags; however, you can also open the Tag Recommendations in the panel to the right of the full text.+Nested Knowledge automatically generates Standard Tag Recommendations for every included study. These are available for Full Texts (by default) but you may also turn on Abstract Tag Recommendations in Settings. 
 + 
 +{{  :undefined:screenshot_2023-11-02_at_14.14.57.png?nolink&  }} 
 + 
 +==== Standard Tag Recs: Full Text ==== 
 + 
 +Once you have uploaded the Full Text, standard tag recommendations will become available. To view them, open the "Full Text Tag Recs" tab on the right hand side, and Standard should be toggled by default. Doing so will open a list of tag recommendations, which are the results for an automatic search **by exact keyword** look up across the whole full text. If in form-based tagging mode (default) a recommendation will be shown as a potential answer to the tag question, whereas in standard mode, a list of recommendations for all tags will be shown. 
 + 
 +Form-based appearance: 
 + 
 +{{  :undefined:screenshot_2023-11-02_at_14.30.05.png?nolink&  }}
  
-Doing so will open a list of tag recommendations, which are the results for an automatic search **by exact keyword** look up across the whole full text:+Standard appearance:
  
-{{:wiki:autolit:tagging:untitled_12.png}}+{{  :undefined:screenshot_2023-11-02_at_14.30.49.png?nolink&  }}
  
 To view the text that is associated with a given Tag Recommendation, click on the recommendation you're interested in. This will: To view the text that is associated with a given Tag Recommendation, click on the recommendation you're interested in. This will:
Line 26: Line 35:
 Once added, the Tag Recommendation will disappear from the Tag Recommendations list and appear as an applied Tag. Once added, the Tag Recommendation will disappear from the Tag Recommendations list and appear as an applied Tag.
  
-===== Smart (Beta) Tag Recommendations (for Enterprise subscriptions only) =====+==== Standard Tag Recs: Abstracts ====
  
-<WRAP center round box 80%> +Similarly, when standard tag recommendations for abstracts is turned on, any recommendations will be highlighted in blue when clicked on. 
-Introducing Nested Knowledge's Smart Tag Recommendations beta, which utilizes OpenAI's GPT-4 model to read and understand your tags, search within your full texts, and offer both smart recommendations and automatic excerpts of text. + 
-</WRAP>+{{  :undefined:screenshot_2023-11-02_at_14.33.02.png?nolink&  }} 
 + 
 +==== Aliases ==== 
 + 
 +If a tag has an alias and the original tag name does not generate any recommendations, the alias will then be searched in order to generate a tag recommendation. To add an alias to a tag, navigate to the tag hierarchy, click on the tag, and add an alias. In the below example, "Gender" was added as an alias to the "Sex" tag. As a result, when "Sex" is unavailable, the standard tag recommendations will search the either the abstract or full text for "Gender" instead and this will be displayed in the tab. 
 + 
 +{{  :undefined:screenshot_2023-11-02_at_14.35.55.png?nolink&  }} 
 + 
 +===== Smart (Beta) Tag Recommendations (for Enterprise subscriptions only) =====
  
 +<WRAP center round box 80%> Introducing Nested Knowledge's Smart Tag Recommendations beta, which utilizes OpenAI's GPT-4 model to read and understand your tags, search within your full texts, and offer both smart recommendations and automatic excerpts of text. </WRAP>
  
-Smart Tag Recommendations are an upgraded version of the above tag recommendations that utilizes GPT 4, a language model provided by OpenAI, to search for tags and automatically highlight corresponding excerpts. To see a full disclosure of what data Smart Tag Recommendations use and how it works, see the [[wiki:support:ai_disclosure#smart_tag_recommendations|Nested Knowledge AI Disclosure]].+Smart Tag Recommendations are an upgraded version of the above tag recommendations that utilizes GPT 4, a language model provided by OpenAI, to search for tags and automatically highlight corresponding excerpts. To see a full disclosure of what data Smart Tag Recommendations use and how it works, see the [[:wiki:support:ai_disclosure#smart_tag_recommendations|Nested Knowledge AI Disclosure]].
  
 Unlike "standard" tags, instead of searching the full text for key words alone, Smart Tag Recommendations uses artificial intelligence to search for similar or applicable terms and/or specific answers to your form-based questions. Compatible with Standard and Form-based Tagging modes, you can toggle on this feature in Admin Settings, provided you have an Enterprise subscription. Unlike "standard" tags, instead of searching the full text for key words alone, Smart Tag Recommendations uses artificial intelligence to search for similar or applicable terms and/or specific answers to your form-based questions. Compatible with Standard and Form-based Tagging modes, you can toggle on this feature in Admin Settings, provided you have an Enterprise subscription.
  
-{{ :undefined:screenshot_2023-10-02_at_16.44.44.png?nolink |}}+{{  :undefined:screenshot_2023-10-02_at_16.44.44.png?nolink&  }}
  
-After toggling on Smart Tag Recommendations, the feature runs in the background to update all recommendations for included studies with full texts in the Tagging queue. This currently takes around 1-2 minutes per study, so it can take some time for recommendations to be generated if you have a large number of included studies. It will also automatically update when new studies are added to the queue.+After toggling on Smart Tag Recommendations, the feature runs in the background to update all recommendations for included studies with full texts in the Tagging queue. This currently takes around 1-2 minutes per study, so it can take some time for recommendations to be generated if you have a large number of included studies. It will also automatically update when new studies are added to the queue. Note: the current limit for generating recommendations is up to 250 studies.
  
 Once updated, a list of tag recommendations will be shown. Similarly to Standard Tag Recommendations, clicking on any recommendation will auto-fill the excerpt box, auto-jump you to the area of the pdf based on the AI's findings and you can apply tags as normal. Below, the recommendation for the tag "Disease Symptom Scores" was selected and the associated text evidence was automatically highlighted in the full text. From here, you can apply tag or clear the selection if you do not wish to apply with this excerpt. Once updated, a list of tag recommendations will be shown. Similarly to Standard Tag Recommendations, clicking on any recommendation will auto-fill the excerpt box, auto-jump you to the area of the pdf based on the AI's findings and you can apply tags as normal. Below, the recommendation for the tag "Disease Symptom Scores" was selected and the associated text evidence was automatically highlighted in the full text. From here, you can apply tag or clear the selection if you do not wish to apply with this excerpt.
Line 49: Line 67:
 ==== Abstract Tag Recommendations ==== ==== Abstract Tag Recommendations ====
  
 +In addition to Full Text Smart Tags, for further assistance, you also have the option to turn on Tag Recommendations (both Standard and Smart) for included Abstracts (see Settings). The format is the same as Full Text Smart Tags, but labelled "Abstract Tag Recs" when the Abstract tab is toggled. It is powered by GPT 4 Turbo.
  
-In addition to Full Text Smart Tags, for further assistance, you also have the option to turn on Tag Recommendations (both Standard and Smart) for included Abstracts (see Settings). The format is the same as Full Text Smart Tags, but labelled "Abstract Tag Recs" when the Abstract tab is toggled.+{{  :undefined:screenshot_2023-10-02_at_16.51.19.png?nolink&  }}
  
-{{ :undefined:screenshot_2023-10-02_at_16.51.19.png?nolink |}}+==== Internal Validation Testing of Abstract Smart Tag Recommendations ====
  
 +In an internal validation, when compared against expert tagging, Smart Tag Recommendations had recall of approximately 50%-60% across three diverse Systematic Literature Review topics when employing OpenAI's GPT 4 Turbo to tag Abstracts from underlying records. Given the customizability of tags and the distinct content in different reviews, recall and accuracy of Smart Tag Recommendations (for both Abstracts and Full Texts) will vary by project, driving home the need for expert confirmation of any Smart Tag Recommendation.
 +
 +==== Guidelines for Optimizing Smart Tag Recommendations ====
 +
 +Check out our [[wiki:autolit:tagging:smart tags guidelines|guidelines for optimizing Smart Tag Recommendations]] and its ability to assist with your review.
 ===== Standard vs Smart Tag Recommendations Example ===== ===== Standard vs Smart Tag Recommendations Example =====
  
 In Standard mode, only the tag name is searched in the full text. In Smart mode, GPT 4 uses similarly reported terms to find the most applicable excerpt. In the below example, for the question "Where are the lesions located?", Standard mode only searches the tag answers: "ICA terminus" "MCA" etc. Clicking on MCA, the exact name is searched in the full text only: In Standard mode, only the tag name is searched in the full text. In Smart mode, GPT 4 uses similarly reported terms to find the most applicable excerpt. In the below example, for the question "Where are the lesions located?", Standard mode only searches the tag answers: "ICA terminus" "MCA" etc. Clicking on MCA, the exact name is searched in the full text only:
  
-{{ :undefined:mca.png?nolink |}}+{{  :undefined:mca.png?nolink&  }}
  
 However, in Smart mode, a deeper dive is conducted. The AI was able to correctly identify the section where all the lesions were located, auto-selecting "Other" as the answer to encompass the multiple locations: However, in Smart mode, a deeper dive is conducted. The AI was able to correctly identify the section where all the lesions were located, auto-selecting "Other" as the answer to encompass the multiple locations:
  
-{{ :undefined:smarttags.png?nolink |}}+{{  :undefined:smarttags.png?nolink&  }} 
 ==== Regenerating Smart Tag Recommendations and Error Reporting ==== ==== Regenerating Smart Tag Recommendations and Error Reporting ====
  
Line 71: Line 96:
 <WRAP center round info 60%> Wondering why Smart Tag Recommendations is unavailable or blank? You may not currently have access due to your subscription type. Upgrade to an Enterprise subscription to access Smart Tag Recommendations and toggle on in Settings. If this is not the case, it may be because it hasn't finished loading yet, check background jobs in the top right. Lastly, the feature may just not have any recommendations for that particular tag or question, try a clicking on a few more and see if it works! If none of these are applicable, contact us [[support@nested-knowledge.com|support@nested-knowledge.com]] </WRAP> <WRAP center round info 60%> Wondering why Smart Tag Recommendations is unavailable or blank? You may not currently have access due to your subscription type. Upgrade to an Enterprise subscription to access Smart Tag Recommendations and toggle on in Settings. If this is not the case, it may be because it hasn't finished loading yet, check background jobs in the top right. Lastly, the feature may just not have any recommendations for that particular tag or question, try a clicking on a few more and see if it works! If none of these are applicable, contact us [[support@nested-knowledge.com|support@nested-knowledge.com]] </WRAP>
  
 +Additionally, when generating new recommendations for identical or similar questions (within or between nests), a certain level of change in the recommendations should be expected. This is because GPT-4 is nondeterministic, due to how it parallelizes floating point arithmetic for performance.
 +
 +===== Tag Recommendations in Screening Module =====
 +
 +Tag recommendations are also displayed in the Screening module. When viewing the abstract, abstract tag recommendations are displayed if turned on in Settings. When viewing the full text, full text tag recommendations are displayed if turned on.
 +
 +{{ :undefined:screenshot_2024-01-17_at_17.36.19.png?nolink |}}
  
wiki/autolit/tagging/tag_recommendations.1697566616.txt.gz · Last modified: 2023/10/17 18:16 by kevinkallmes

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki