Answers.Body: <p>From the insight of prime-target word pairs experiments in cognitive sciences that usually the participants are asked to do a fairly simple lexical decision or categorization task for the target word once seen the prime, your Latency Time metric is apparently more appropriate to align with priming. Of course you can further measure the Response Time since every LLM's internal response mechanism may be different, it can capture any potential effects on overall processing <em>assuming</em> the content of your LLM's responses are already appropriate. Finally It's important to note that this approach is an analogy to human semantic priming so your result should be interpreted cautiously.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-16T09:27:42.647Z
Answers.Text: I do agree with that, really a lot actually. Im 100% convinced that we shouldn't look at cognition in humans and just transfer it to LLMs, we there's an ocean of reasons for not doing that. On the other hand, we must acknowledge that like GPT, as next-word-predictor, is not simply a next-word-predictor: there's a ton of evidence of really cool emerging properties. I think study them it's important to not misjudge such as "cognitive capabilities", at least not in human-like manner. 
  Second, as you said RT and latency are sides of the coin. Are you suggesting measuring both then?
Answers.UserDisplayName: Filippo Colombi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-16T18:40:04.890Z
Answers.Text: @FilippoColombi The most important metric is LT as motivated in my answer above from the insight of priming experiment from psychology per your request, sure, if your resource permits it'll be beneficial to have the additional RT metric for correlation or distribution comparison as you rightfully argued that LLM's cognition may not be same as human's structurally, so more metrics will certainly offer more information. Hope this clarifies your lingering concerns,
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-15T19:46:07.247Z
Answers.LastActivityDate: 2024-11-15T19:46:07.247Z
Answers.UserDisplayName: cinch
Body: <p>Premise: here I take Latency as the time delay from when a prompt is submitted to the model until it begins generating a response, and Response Time as the end-to-end interval from the moment the prompt is submitted until the model completes generating its response.</p>
  <p>The point here is to have a look at LLMs (could be GPT-4) and extract a quantitive measure of semantic retrieval in a common priming experiment (prime-target word pairs). Does anyone have experience with similar research? Would you suggest using Latency or Response Time? Please motivate your response, any insight is very much appreciated!</p>
  
Tags: <neural-networks><large-language-models><chatgpt><cognitive-science>
Title: Latency or Response Time to measure semantic activation in an LLM?
UserDisplayName: Filippo Colombi
ThreadText: # Latency or Response Time to measure semantic activation in an LLM?
  
  (Asked by: username_0 on 2024-11-15)
  
  Premise: here I take Latency as the time delay from when a prompt is submitted to the model until it begins generating a response, and Response Time as the end-to-end interval from the moment the prompt is submitted until the model completes generating its response.
  
  The point here is to have a look at LLMs (could be GPT-4) and extract a quantitive measure of semantic retrieval in a common priming experiment (prime-target word pairs). Does anyone have experience with similar research? Would you suggest using Latency or Response Time? Please motivate your response, any insight is very much appreciated!
  
  **Tags:** neural-networks, large-language-models, chatgpt, cognitive-science
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-15. Score: 0 | Accepted answer
  
  From the insight of prime-target word pairs experiments in cognitive sciences that usually the participants are asked to do a fairly simple lexical decision or categorization task for the target word once seen the prime, your Latency Time metric is apparently more appropriate to align with priming. Of course you can further measure the Response Time since every LLM's internal response mechanism may be different, it can capture any potential effects on overall processing *assuming* the content of your LLM's responses are already appropriate. Finally It's important to note that this approach is an analogy to human semantic priming so your result should be interpreted cautiously.
  
  > **Comments:**
  > **username_0**: I do agree with that, really a lot actually. Im 100% convinced that we shouldn't look at cognition in humans and just transfer it to LLMs, we there's an ocean of reasons for not doing that. On the other hand, we must acknowledge that like GPT, as next-word-predictor, is not simply a next-word-predictor: there's a ton of evidence of really cool emerging properties. I think study them it's important to not misjudge such as "cognitive capabilities", at least not in human-like manner.  Second, as you said RT and latency are sides of the coin. Are you suggesting measuring both then?  
  > **username_1**: @FilippoColombi The most important metric is LT as motivated in my answer above from the insight of priming experiment from psychology per your request, sure, if your resource permits it'll be beneficial to have the additional RT metric for correlation or distribution comparison as you rightfully argued that LLM's cognition may not be same as human's structurally, so more metrics will certainly offer more information. Hope this clarifies your lingering concerns,  
  
  ---
  
  