We analyze the Gambler's problem, a simple reinforcement learning problem where the gambler has the chance to double or lose their bets until the target is reached. This is an early example introduced in the reinforcement learning textbook by Sutton & Barto (2018), where they mention an interesting pattern of the optimal value function with high-frequency components and repeating nonsmooth points. It is however without further investigation. We provide the exact formula for the optimal value function for both the discrete and the continuous case. Though simple as it might seem, the value function is pathological: fractal, self-similar, zero-derivative almost everywhere, not smooth on any interval, and not written as elementary functions. It is in fact one of the generalized Cantor functions, where it holds a complexity that has been uncharted thus far. Our analyses could lead insights on improving value function approximation, gradientbased algorithms, and Q-learning, in real applications and implementations.
The paper derives the optimal value function for the continuous and discrete versions of the Gambler's problem. It also explores certain properties of this value function and finds that it has many fractal-like qualities.
This paper focuses on an unconventional but intriguing topic, bringing more insight into the possible shapes of value functions. The main result, which describes the value function for the gambler's problem, seems surprisingly complex and could have some practical implications for value-learning. Unfortunately, I find the paper is difficult to follow due its organization and the presence of many typos and unclear phrases. In summary, while I believe the topic is interesting and the work seems sound, the presentation would need to be improved significantly for me to recommend acceptance.
Here are certain points to be addressed:
1) On p.2, "The optimal value function presents its self-similar, fractal and non-rectifiable form". It is not clear from Thm. 11 that these properties hold. Some further explanation would be helpful 
2) I think it would be helpful to include some descriptions of certain concepts that may be less familiar to the reader in the appendix. E.g. the Cantor function
3) Fig.1 should have larger labels and dots. It is difficult to see currently.
4) p.3 "the similar fractal patterns observed empirically in other reinforcement learning tasks." Certain concrete examples would be helpful here since I was not aware that this was common.
5) p.4 The paragraph on 'self-similarity' was not clear to me. What is meant by 'chaos' or 'dimension' in this context? 
6) p.5 "Otherwise an arbitrary small amount of will have a fixed probability of reaching the target 1." I am unsure what this sentence means.
7) Perhaps starting with the intuitive description of v(s) (the recursive form) first before presenting the closed form solution would be easier to follow.
8) Why is there no discussion/conclusion section at the end?
There are numerous typos scattered throughout the paper. Here are some from the first page:
- abstract: "where they mention an interesting pattern of
the optimal value function with high-frequency components and repeating nonsmooth
points but without further investigation." Awkward phrase
- abstract: "With the analysis," Unnecessary phrase
- par. 1: "is described as below" -> "is described below"
- "which would hide its attractiveness." Awkward phrase
- "as an representative" -> "as a representative"
- "family of Markov decision process" -> "family of Markov decision processes"
- "the amount of bet." -> "the amount bet."
- "a round of bet." -> "a round of betting."
- "action-state value function" -> "state-action value function"
- "n as the starting capital (n denotes the state in the discrete setting)," The phrase is a bit confusing as it suggests n denotes both the discrete state and the starting capital.