Differences

This shows you the differences between two versions of the page.

--- wiki:autolit:tagging:tag_recommendations [2023/11/21 11:46]
jthurnham [Smart (Beta) Tag Recommendations (for Enterprise subscriptions only)]
+++ wiki:autolit:tagging:tag_recommendations [2024/01/27 19:28] (current)
jthurnham [Internal Validation Testing of Abstract Smart Tag Recommendations]
@@ Line 2: / Line 2: @@
 There are two Tag Recommendation modes: Standard and Smart (Beta). In short, **standard tag recommendations** use keyword searches and is available to all users while **Smart Tag Recommendations** uses OpenAI, a Large Language Model that can read and answer questions, but is //only available to enterprise customers.//
 ===== Standard Tag Recommendations =====
 Nested Knowledge automatically generates Standard Tag Recommendations for every included study. These are available for Full Texts (by default) but you may also turn on Abstract Tag Recommendations in Settings.
-{{ :undefined:screenshot_2023-11-02_at_14.14.57.png?nolink |}}
+{{  :undefined:screenshot_2023-11-02_at_14.14.57.png?nolink&  }}
 ==== Standard Tag Recs: Full Text ====
@@ Line 15: / Line 14: @@
 Form-based appearance:
-{{ :undefined:screenshot_2023-11-02_at_14.30.05.png?nolink |}}
+{{  :undefined:screenshot_2023-11-02_at_14.30.05.png?nolink&  }}
 Standard appearance:
-{{ :undefined:screenshot_2023-11-02_at_14.30.49.png?nolink |}}
+{{  :undefined:screenshot_2023-11-02_at_14.30.49.png?nolink&  }}
 To view the text that is associated with a given Tag Recommendation, click on the recommendation you're interested in. This will:
@@ Line 40: / Line 39: @@
 Similarly, when standard tag recommendations for abstracts is turned on, any recommendations will be highlighted in blue when clicked on.
-{{ :undefined:screenshot_2023-11-02_at_14.33.02.png?nolink |}}
+{{  :undefined:screenshot_2023-11-02_at_14.33.02.png?nolink&  }}
 ==== Aliases ====
@@ Line 47: / Line 45: @@
 If a tag has an alias and the original tag name does not generate any recommendations, the alias will then be searched in order to generate a tag recommendation. To add an alias to a tag, navigate to the tag hierarchy, click on the tag, and add an alias. In the below example, "Gender" was added as an alias to the "Sex" tag. As a result, when "Sex" is unavailable, the standard tag recommendations will search the either the abstract or full text for "Gender" instead and this will be displayed in the tab.
-{{ :undefined:screenshot_2023-11-02_at_14.35.55.png?nolink |}}
+{{  :undefined:screenshot_2023-11-02_at_14.35.55.png?nolink&  }}
 ===== Smart (Beta) Tag Recommendations (for Enterprise subscriptions only) =====
@@ Line 68: / Line 67: @@
 ==== Abstract Tag Recommendations ====
-In addition to Full Text Smart Tags, for further assistance, you also have the option to turn on Tag Recommendations (both Standard and Smart) for included Abstracts (see Settings). The format is the same as Full Text Smart Tags, but labelled "Abstract Tag Recs" when the Abstract tab is toggled.
+In addition to Full Text Smart Tags, for further assistance, you also have the option to turn on Tag Recommendations (both Standard and Smart) for included Abstracts (see Settings). The format is the same as Full Text Smart Tags, but labelled "Abstract Tag Recs" when the Abstract tab is toggled. It is powered by GPT 4 Turbo.
 {{  :undefined:screenshot_2023-10-02_at_16.51.19.png?nolink&  }}
+==== Internal Validation Testing of Abstract Smart Tag Recommendations ====
+In an internal validation, when compared against expert tagging, Smart Tag Recommendations had recall of approximately 50%-60% across three diverse Systematic Literature Review topics when employing OpenAI's GPT 4 Turbo to tag Abstracts from underlying records. Given the customizability of tags and the distinct content in different reviews, recall and accuracy of Smart Tag Recommendations (for both Abstracts and Full Texts) will vary by project, driving home the need for expert confirmation of any Smart Tag Recommendation.
+==== Guidelines for Optimizing Smart Tag Recommendations ====
+Check out our [[wiki:autolit:tagging:smart tags guidelines|guidelines for optimizing Smart Tag Recommendations]] and its ability to assist with your review.
 ===== Standard vs Smart Tag Recommendations Example =====
@@ Line 90: / Line 96: @@
 <WRAP center round info 60%> Wondering why Smart Tag Recommendations is unavailable or blank? You may not currently have access due to your subscription type. Upgrade to an Enterprise subscription to access Smart Tag Recommendations and toggle on in Settings. If this is not the case, it may be because it hasn't finished loading yet, check background jobs in the top right. Lastly, the feature may just not have any recommendations for that particular tag or question, try a clicking on a few more and see if it works! If none of these are applicable, contact us [[support@nested-knowledge.com|support@nested-knowledge.com]] </WRAP>
+Additionally, when generating new recommendations for identical or similar questions (within or between nests), a certain level of change in the recommendations should be expected. This is because GPT-4 is nondeterministic, due to how it parallelizes floating point arithmetic for performance.
-==== Abstract Tag Recommendations ====
+===== Tag Recommendations in Screening Module =====
+Tag recommendations are also displayed in the Screening module. When viewing the abstract, abstract tag recommendations are displayed if turned on in Settings. When viewing the full text, full text tag recommendations are displayed if turned on.
-In addition to Full Text Smart Tags, for further assistance, you also have the option to turn on Tag Recommendations (both Standard and Smart) for included Abstracts (see Settings). The format is the same as Full Text Smart Tags, but labelled "Abstract Tag Recs" when the Abstract tab is toggled.
+{{ :undefined:screenshot_2024-01-17_at_17.36.19.png?nolink |}}
-{{ :undefined:screenshot_2023-10-02_at_16.51.19.png?nolink |}}
-===== Standard vs Smart Tag Recommendations Example =====
-In Standard mode, only the tag name is searched in the full text. In Smart mode, GPT 4 uses similarly reported terms to find the most applicable excerpt. In the below example, for the question "Where are the lesions located?", Standard mode only searches the tag answers: "ICA terminus" "MCA" etc. Clicking on MCA, the exact name is searched in the full text only:
-{{ :undefined:mca.png?nolink |}}
-However, in Smart mode, a deeper dive is conducted. The AI was able to correctly identify the section where all the lesions were located, auto-selecting "Other" as the answer to encompass the multiple locations:
-{{ :undefined:smarttags.png?nolink |}}
-==== Regenerating Smart Tag Recommendations and Error Reporting ====
-If you make changes to your tag hierarchy or switched tagging modes after turning on Smart Tag Recommendations, you may wish to update the recommendations accordingly. Due to expense of generating smart tag recommendations, a limited number of regenerations are allowed in a nest at this time. This limit is 1 regeneration.
-Since the use of OpenAI for tag recommendations is in its beta phase, meaning it is technically feature complete but still in its early stages, there is a possibility you may run into errors. Be sure to [[https://nested-knowledge.nolt.io/|let us know if this occurs.]]
-<WRAP center round info 60%> Wondering why Smart Tag Recommendations is unavailable or blank? You may not currently have access due to your subscription type. Upgrade to an Enterprise subscription to access Smart Tag Recommendations and toggle on in Settings. If this is not the case, it may be because it hasn't finished loading yet, check background jobs in the top right. Lastly, the feature may just not have any recommendations for that particular tag or question, try a clicking on a few more and see if it works! If none of these are applicable, contact us [[support@nested-knowledge.com|support@nested-knowledge.com]] </WRAP>

Nested Knowledge

User Tools

Site Tools

Differences

Page Tools