Answers.Body: <blockquote>
  <p>to incorporate the uncertainty... I'm not able to understand the above extract,</p>
  </blockquote>
  <p>usually, uncertainty is difined by variance... therefore you can add some noise - see <a href="https://pdf.sciencedirectassets.com/280203/1-s2.0-S1877050917X00148/1-s2.0-S187705091732001X/main.pdf?X-Amz-Security-Token=IQoJb3JpZ2luX2VjENn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIGCV1dUXjqgpkPJqS1MqW4j6DQp5ZZF4k68%2FMeYYEWIIAiEA8hqlTNxiTG4CVRFZEhZtHE99HfNbXyDU9eciN%2FyDeFcqswUIYhAFGgwwNTkwMDM1NDY4NjUiDGyZswk8YDNwgmq0bSqQBc6zEDwp6gyZ0mUSYV4FPRQ8bQcBLiT7QAhJkA47%2BafthO2RXqA%2FgYmlqt9FZtCpB735kHcpHIhBHWTJ7bwAdyD5sjOfHphjY4OWArsgcHNrPmroICQVEnelCU6%2Bq6vptDKSsQ%2FEDUb4UAkja4UEU87Ys0AN47hvvPDBLBbDQ7SjeqLywPWPVz6sJKoi2Lm96nmbonYUmkD%2BTGoRaxgwBkIHEYk5vCvkFGmhw1xQQiVZ1UlxHDs0jK28DCvdVt08oQsjs%2FIuUJlhSDWft83UcibLpk9dlvNlH6iaFEg0VBDfGWnuoj3Uxo94Sk1UGh2Vnc8Ww1Umg%2FpuK4VJ68z%2Bl%2Fv8QQikB0uA8XFBNPVGWVGILbfass7EICgo1OsT6TeKLPhrYa9BdQXHwBgr9G3E3tbX7PeD9si%2FByFSLuUQObMzbZy7uDj71kPkV0lyEIRTJWGW%2BmOhfcgYIm5heE3BJQ2nezjLe59YlJMxAKBMRBo65b%2F1FdIXO8eC%2BZBImwWg1LwSsZgoD6T9Kt1sEoRgHav2WDe0MCZuQwIzHmzbNUR3xXGG3ZDNAovSAv1quDjSJ92OKcd6aE6RcmMlv1Lx%2Fo8NzAOqehe3%2FK1QuRW7FXn2T6dgULoPhpQchuhjQwv5XivByieFDe%2FKhpm6grmAOTkHQ9IqlBf1otVo7UGxpZR3F5akgf9Wjb3nnjcRqnC4v1Vli2dw9YmxgfDoUYTorp8fI5yQs9csWvCQR8j7SURik4vutLI%2B14h9%2FqoxiAPP9Pr6qQuaYcuI3goTlr2PnuEDXZ2dbeNQeRHzUVJ9s9LrVHUMmfTMmScokKLc4Je%2FIIp3R3Z%2F9vX4%2Bq0sGEjoyXE6D9Zj0sUH2OBWD9AAtDqEMIGk7bIGOrEBdVqbRH1mnzXaFZZlTUY7PtLRBbJ92dw7HljcacRydOsOn6DWO%2BN2%2Fto5QbG4h9IDdLpea9khKftGsPMyEOEoc%2FpOMj5F1jN%2F7jdaXWX1NZWVoXYywDZN6inxxa1S6ZeOJ6SYyaiApEUeMwpV%2ByrCDp%2By%2BnstXWsba3mvotqwUMr%2FP%2FDZk5AlhSFSGsoxKs8QCgzaTIeDcaS%2B2XhZm6bARdZeoeSCqOEoUU%2BWj7%2Bq0f9F&amp;X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20240601T172712Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=300&amp;X-Amz-Credential=ASIAQ3PHCVTYXO5BE54H%2F20240601%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=a09190a25ba1664f837b999e97a8fc02c97aa804c9c121d1731d4a15ee817b50&amp;hash=46918a502cfcc55392ed4b9e1f6e0cf07be058f0b31e625956dee44f91b6bcd3&amp;host=68042c943591013ac2b2430a89b270f6af2c76d8dfd086a07176afe7c76c2c61&amp;pii=S187705091732001X&amp;tid=spdf-91cd0fa6-5847-432b-9340-b6899c87c5fd&amp;sid=dc063577893eb544c049b6c7a345439b8f4bgxrqb&amp;type=client&amp;tsoh=d3d3LnNjaWVuY2VkaXJlY3QuY29t&amp;ua=061e5e5906570a560c&amp;rr=88d0e9be3f6aabfe&amp;cc=by" rel="nofollow noreferrer">here</a></p>
  <blockquote>
  <p>Moreover, the concept of natural noise is an emerging trend that is
  related to inconsistent behaviour of users</p>
  </blockquote>
  <p>-&gt; read Conclusion</p>
  <blockquote>
  <p>the results obtained from experimental section indicate that it is
  possible to improvethe accuracy of recommendation methods through the
  detection and correction of noisy ratings</p>
  </blockquote>
  <p>for details - read the whole...</p>
  <p>though algorithms can differ... another possible implementation is described <a href="https://buildingrecommenders.wordpress.com/2015/11/11/dithering/" rel="nofollow noreferrer">here</a>... it's up to data-analyst what tools for analysis to choose</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-01T17:40:05.047Z
Answers.LastActivityDate: 2024-06-01T17:40:05.047Z
Answers.UserDisplayName: JeeyCi
Answers.Body: <p>I found the answer further into <a href="https://arxiv.org/abs/1906.05237" rel="nofollow noreferrer">the paper</a> (section 3.2 <strong>Formulation as Markov Decision Process</strong>)! I'll post it here for everyone.</p>
  <blockquote>
  <p>Given any user, there is no pre-known targeted item in the KGRE-Rec problem, so it is unfeasible to consider binary rewards indicating whether the agent has reached a target or not. Instead, the agent is encouraged to explore as many “good” paths as possible. Intuitively, in the context of recommendations, a <strong>“good” path is one that leads to an item that a user will interact with, with high probability</strong>. To this end, we consider to give a soft reward only for the terminal state <span class="math-container">$s_{T}=\left(u, e_{T}, h_{T}\right)$</span> based on another scoring function <span class="math-container">$f(u, i)$</span>. The terminal reward <span class="math-container">$R_{T}$</span> is defined as</p>
  <p><span class="math-container">$$
  R_{T}= \begin{cases}\max \left(0, \frac{f\left(u, e_{T}\right)}{\max _{i \in I} f(u, i)}\right), &amp; \text { if } e_{T} \in \mathcal{I} \\ 0, &amp; \text { otherwise }\end{cases}
  $$</span></p>
  <p>where the value of <span class="math-container">$R_{T}$</span> is normalized to the range of <span class="math-container">$[0,1] . f(u, i)$</span> is also introduced in the next section.</p>
  </blockquote>
  <p>Details about <span class="math-container">$f(u,i)$</span>, the scoring function, can be found in the paper.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-03T05:55:24.980Z
Answers.Text: For clarification, a state is defined as a tuple $(u,e_t,h_t)$, where $u$,$e_t$,$h_t$ represent the starting user identity, entity the agent has reached at step $t$, and history prior to step $t$.
Answers.UserDisplayName: stoic-santiago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-03T05:50:05.840Z
Answers.LastActivityDate: 2021-12-26T13:44:00.090Z
Answers.LastEditDate: 2021-12-26T13:44:00.090Z
Answers.UserDisplayName: stoic-santiago
Body: <p>I've been reading <a href="https://arxiv.org/pdf/1906.05237.pdf" rel="nofollow noreferrer">this paper on recommendation systems</a> using reinforcement learning (RL) and knowledge graphs (KGs).</p>
  
  <p>To give some background, the graph has several (finitely many) entities, of which some are user entities and others are item entities. The goal is to recommend items to users, i.e. to find a recommendation set of items for every user such that the user and the corresponding items are connected by one reasoning path.</p>
  
  <p>I'm attaching an example of such a graph for more clarity (from the paper itself) -</p>
  
  <h1><a href="https://i.sstatic.net/zhdCC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zhdCC.png" alt="enter image description here"></a></h1>
  
  <p>In the paper above, they say</p>
  
  <blockquote>
    <p>First, we do not have pre-defined targeted items for any user, so it is <strong>not applicable to use a binary reward</strong> indicating whether the user interacts with the item or not. A better design of the reward function is to <em>incorporate the uncertainty</em> of how an item is relevant to a user based on the rich heterogeneous information given by the knowledge graph.</p>
  </blockquote>
  
  <p>I'm not able to understand the above extract, which talks about the reward function to use - binary, or something else. A detailed explanation of what the author is trying to convey in the above extract would really help. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-02T16:30:49.870Z
Comments.Text: Are you familiar with the basic concept of RL, including reward functions?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-02T16:31:54.543Z
Comments.Text: Yes, I am familiar with it.
Comments.UserDisplayName: stoic-santiago
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-02T16:36:58.380Z
Comments.Text: I didn't read this paper, but the authors probably mean that, given that you don't know _a priori_ (or beforehand) if a user will buy or not an item, it probably doesn't make sense to give a reward of 1 whenever the system recommends an item that the user eventually buys. It probably makes more sense to associate a stochastic reward to recommendations of the system based on your knowledge encoded in the KG. By "stochastic reward" I mean something that isn't deterministic. Maybe later I will investigate that paper and provide a more solid answer, but I can't assure you.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-03T05:52:37.317Z
Comments.Text: You're absolutely right, @nbro
  I've added an answer that I found convincing using the details I uncovered from the latter half of the paper.
Comments.UserDisplayName: stoic-santiago
Tags: <reinforcement-learning><papers><reward-functions><recommender-system><knowledge-graph>
Title: Which reward function works for recommendation systems using knowledge graphs?
UserDisplayName: stoic-santiago
ThreadText: # Which reward function works for recommendation systems using knowledge graphs?
  
  (Asked by: username_0 on 2020-05-02)
  
  I've been reading [this paper on recommendation systems](https://arxiv.org/pdf/1906.05237.pdf) using reinforcement learning (RL) and knowledge graphs (KGs).
  
  To give some background, the graph has several (finitely many) entities, of which some are user entities and others are item entities. The goal is to recommend items to users, i.e. to find a recommendation set of items for every user such that the user and the corresponding items are connected by one reasoning path.
  
  I'm attaching an example of such a graph for more clarity (from the paper itself) -
  
  # [enter image description here](https://i.sstatic.net/zhdCC.png)
  
  In the paper above, they say
  
  > First, we do not have pre-defined targeted items for any user, so it is **not applicable to use a binary reward** indicating whether the user interacts with the item or not. A better design of the reward function is to *incorporate the uncertainty* of how an item is relevant to a user based on the rich heterogeneous information given by the knowledge graph.
  
  I'm not able to understand the above extract, which talks about the reward function to use - binary, or something else. A detailed explanation of what the author is trying to convey in the above extract would really help.
  
  **Tags:** reinforcement-learning, papers, reward-functions, recommender-system, knowledge-graph
  > **Comments:**
  > **username_1**: Are you familiar with the basic concept of RL, including reward functions?  
  > **username_0**: Yes, I am familiar with it.  
  > **username_1**: I didn't read this paper, but the authors probably mean that, given that you don't know _a priori_ (or beforehand) if a user will buy or not an item, it probably doesn't make sense to give a reward of 1 whenever the system recommends an item that the user eventually buys. It probably makes more sense to associate a stochastic reward to recommendations of the system based on your knowledge encoded in the KG. By "stochastic reward" I mean something that isn't deterministic. Maybe later I will investigate that paper and provide a more solid answer, but I can't assure you.  
  > **username_0**: You're absolutely right, @nbro I've added an answer that I found convincing using the details I uncovered from the latter half of the paper.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2020-05-03. Score: 2 | Accepted answer
  
  I found the answer further into [the paper](https://arxiv.org/abs/1906.05237) (section 3.2 **Formulation as Markov Decision Process**)! I'll post it here for everyone.
  
  > Given any user, there is no pre-known targeted item in the KGRE-Rec problem, so it is unfeasible to consider binary rewards indicating whether the agent has reached a target or not. Instead, the agent is encouraged to explore as many “good” paths as possible. Intuitively, in the context of recommendations, a **“good” path is one that leads to an item that a user will interact with, with high probability**. To this end, we consider to give a soft reward only for the terminal state $s\_{T}=\left(u, e\_{T}, h\_{T}\right)$ based on another scoring function $f(u, i)$. The terminal reward $R\_{T}$ is defined as
  >
  > $$
  > R\_{T}= \begin{cases}\max \left(0, \frac{f\left(u, e\_{T}\right)}{\max \_{i \in I} f(u, i)}\right), & \text { if } e\_{T} \in \mathcal{I} \\ 0, & \text { otherwise }\end{cases}
  > $$
  >
  > where the value of $R\_{T}$ is normalized to the range of $[0,1] . f(u, i)$ is also introduced in the next section.
  
  Details about $f(u,i)$, the scoring function, can be found in the paper.
  
  > **Comments:**
  > **username_0**: For clarification, a state is defined as a tuple $(u,e_t,h_t)$, where $u$,$e_t$,$h_t$ represent the starting user identity, entity the agent has reached at step $t$, and history prior to step $t$.  
  
  ---
  
  ### Answer by username_2 on 2024-06-01. Score: 0
  
  > to incorporate the uncertainty... I'm not able to understand the above extract,
  
  usually, uncertainty is difined by variance... therefore you can add some noise - see [here](https://pdf.sciencedirectassets.com/280203/1-s2.0-S1877050917X00148/1-s2.0-S187705091732001X/main.pdf?X-Amz-Security-Token=IQoJb3JpZ2luX2VjENn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIGCV1dUXjqgpkPJqS1MqW4j6DQp5ZZF4k68%2FMeYYEWIIAiEA8hqlTNxiTG4CVRFZEhZtHE99HfNbXyDU9eciN%2FyDeFcqswUIYhAFGgwwNTkwMDM1NDY4NjUiDGyZswk8YDNwgmq0bSqQBc6zEDwp6gyZ0mUSYV4FPRQ8bQcBLiT7QAhJkA47%2BafthO2RXqA%2FgYmlqt9FZtCpB735kHcpHIhBHWTJ7bwAdyD5sjOfHphjY4OWArsgcHNrPmroICQVEnelCU6%2Bq6vptDKSsQ%2FEDUb4UAkja4UEU87Ys0AN47hvvPDBLBbDQ7SjeqLywPWPVz6sJKoi2Lm96nmbonYUmkD%2BTGoRaxgwBkIHEYk5vCvkFGmhw1xQQiVZ1UlxHDs0jK28DCvdVt08oQsjs%2FIuUJlhSDWft83UcibLpk9dlvNlH6iaFEg0VBDfGWnuoj3Uxo94Sk1UGh2Vnc8Ww1Umg%2FpuK4VJ68z%2Bl%2Fv8QQikB0uA8XFBNPVGWVGILbfass7EICgo1OsT6TeKLPhrYa9BdQXHwBgr9G3E3tbX7PeD9si%2FByFSLuUQObMzbZy7uDj71kPkV0lyEIRTJWGW%2BmOhfcgYIm5heE3BJQ2nezjLe59YlJMxAKBMRBo65b%2F1FdIXO8eC%2BZBImwWg1LwSsZgoD6T9Kt1sEoRgHav2WDe0MCZuQwIzHmzbNUR3xXGG3ZDNAovSAv1quDjSJ92OKcd6aE6RcmMlv1Lx%2Fo8NzAOqehe3%2FK1QuRW7FXn2T6dgULoPhpQchuhjQwv5XivByieFDe%2FKhpm6grmAOTkHQ9IqlBf1otVo7UGxpZR3F5akgf9Wjb3nnjcRqnC4v1Vli2dw9YmxgfDoUYTorp8fI5yQs9csWvCQR8j7SURik4vutLI%2B14h9%2FqoxiAPP9Pr6qQuaYcuI3goTlr2PnuEDXZ2dbeNQeRHzUVJ9s9LrVHUMmfTMmScokKLc4Je%2FIIp3R3Z%2F9vX4%2Bq0sGEjoyXE6D9Zj0sUH2OBWD9AAtDqEMIGk7bIGOrEBdVqbRH1mnzXaFZZlTUY7PtLRBbJ92dw7HljcacRydOsOn6DWO%2BN2%2Fto5QbG4h9IDdLpea9khKftGsPMyEOEoc%2FpOMj5F1jN%2F7jdaXWX1NZWVoXYywDZN6inxxa1S6ZeOJ6SYyaiApEUeMwpV%2ByrCDp%2By%2BnstXWsba3mvotqwUMr%2FP%2FDZk5AlhSFSGsoxKs8QCgzaTIeDcaS%2B2XhZm6bARdZeoeSCqOEoUU%2BWj7%2Bq0f9F&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240601T172712Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Credential=ASIAQ3PHCVTYXO5BE54H%2F20240601%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=a09190a25ba1664f837b999e97a8fc02c97aa804c9c121d1731d4a15ee817b50&hash=46918a502cfcc55392ed4b9e1f6e0cf07be058f0b31e625956dee44f91b6bcd3&host=68042c943591013ac2b2430a89b270f6af2c76d8dfd086a07176afe7c76c2c61&pii=S187705091732001X&tid=spdf-91cd0fa6-5847-432b-9340-b6899c87c5fd&sid=dc063577893eb544c049b6c7a345439b8f4bgxrqb&type=client&tsoh=d3d3LnNjaWVuY2VkaXJlY3QuY29t&ua=061e5e5906570a560c&rr=88d0e9be3f6aabfe&cc=by)
  
  > Moreover, the concept of natural noise is an emerging trend that is
  > related to inconsistent behaviour of users
  
  -> read Conclusion
  
  > the results obtained from experimental section indicate that it is
  > possible to improvethe accuracy of recommendation methods through the
  > detection and correction of noisy ratings
  
  for details - read the whole...
  
  though algorithms can differ... another possible implementation is described [here](https://buildingrecommenders.wordpress.com/2015/11/11/dithering/)... it's up to data-analyst what tools for analysis to choose
  
  
  ---
  
  