**Passive reinforcement learning**


Do not consider T and R → simplified task - **just evaluate policy by state value**
<details><summary>

**So goal is compute value of each state under policy (input is policy → (episodes training by observation) → output is value per state)**</summary>

<details><summary>

**direct evaluation** - just sum of after state's value and divide by episode number</summary>

</details>

<details><summary>

**Sample-based policy evaluation **- improve V per transition by estimate T and R by sample of outcome</summary>

<details><summary>

**Temporal difference learning** - 그상태만 고려</summary>

![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F91ec5e05-f868-4806-859d-35d00b69575a%2FUntitled.png?table=block&id=4b86d05f-b8db-49b9-894a-571262041d23&cache=v2)
</details>

<details><summary>

**Exponential moving average** - 최근꺼 더 중요하게 고려</summary>

</details>

<details><summary>

**Active reinforcement learning **- 실해로 행하면서</summary>

> Fundamental tradeoff: exploration vs. exploitation - since it is online playing

</details>

<details><summary>

**Q-learning **- exponential + 전부다 = Q-value-iteration</summary>

![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fe0cc087b-dcad-48e8-9419-56b5ce7df7c0%2FUntitled.png?table=block&id=4f27f063-8a1c-4599-947c-5126a8ed5029&cache=v2)
![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F486a1aca-a438-4011-932b-4bad8be94167%2FUntitled.png?table=block&id=af9e1f2f-801d-42bb-85c6-46ae58d1c65a&cache=v2)
![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fe69f841c-7864-4e5b-8ea4-cc3b33c91a37%2FUntitled.png?table=block&id=07d1519b-abfb-41d1-8818-48e46c6e47ce&cache=v2)
</details>

</details>


</details>

> Optimal value function means high Q value
> Q value is prediction of all future R by action
> V value is prediction of all future R in state
> Policy is action per state which maximize V
> Alpha means learning rate

![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F3e9870ff-dce0-4ac8-b380-9ab6bc418002%2FUntitled.png?table=block&id=46ea0752-d926-4a7f-bbf1-6adc2f1cd993&cache=v2)
![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fbfadfe49-2e0f-436f-afde-3a86a8f7bc17%2FUntitled.png?table=block&id=517faf71-946c-49af-8c3b-6ec539f68c2b&cache=v2)
> So most reinforce learning is Q - MDP is not used - just for understanding





