Training a Constrained Natural Media Painting Agent using Reinforcement Learning
We present a novel approach to train a natural media painting using reinforcement learning. Given a reference image, our formulation is based on stroke-based rendering that imitates human drawing and can be learned from scratch without supervision. Our painting agent computes a sequence of actions that represent the primitive painting strokes. In order to ensure that the generated policy is predictable and controllable, we use a constrained learning method and train the painting agent using the environment model and follows the commands encoded in an observation. We have applied our approach on many benchmarks and our results demonstrate that our constrained agent can handle different painting media and different constraints in the action space to collaborate with humans or other agents.
The authors present the results of training a natural media painting agent using reinforcement learning for different types of strokes. The agent seems to be capable of learning how to pain under different types of constraints and produce visually interesting images. Comments: - Given that the authors give an implementation of the constrained RL agent as one of the key contributions of the paper, there is a glaring absence of mentioning related work on constrained reinforcement learning and reviewing the existing approaches in literature, in order to compare and contrast what the authors propose in this paper. This makes it hard for the readers to assess the novelty of the contribution. - Similarly, the authors should discuss in more detail the limitations on the types of constraints that are possible to easily express in this framework, given the simplicity of the constraints that are shown in the experiments - The readability of the paper seems like it could be improved. Apart from typos, there seem to be many long enumerations of approaches that other researchers have taken in this space, but for a reader it is not immediately obvious how these come together and relate to the work that is being presented. - With the above in mind, one thing that was conceptually unclear to me is that one of the main advantages of the proposed approach, according to the authors, when compared to some of the cited related work, is that this approach can generate intermediate representations and not just the final output with most resemblance to the reference picture. There is a mention of this opening up new artistic possibilities. Yet, this particular use case is not given central stage in evaluation. The authors should provide more examples of why this particular capability is relevant and how it leads to interesting outcomes. - The authors say: “we use 5 strokes to reproduce hand-written digits images, 20 strokes to reproduce character images, 100 strokes to reproduce face and object images” - which does seem reasonable, but the actual numbers (5, 20, 100) are not well motivated. After all, why not (10, 50, 200) or (5, 40, 100) or (5, 40, 200)? It would be good to show experimentally (for one of these) that the choice is justified. I’m not familiar with KanjiVG - but there do exist kanji characters with more than 50 strokes. I’m guessing they are not present in this dataset? - The authors should provide more details on the specifics of the model architecture and the hyperparameters that have been used / explored. - In the results/discussion of the paper, the authors should compare to prior work and highlight their novel contributions. Given that multiple papers out there that have had at first glance similar-looking results, it is hard to otherwise qualitatively judge whether what is proposed in this paper is better, without any form of side-by-side comparison.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Training a Constrained Natural Media Painting Agent using Reinforcement Learning
We present a novel approach to train a natural media painting using reinforcement learning. Given a reference image, our formulation is based on stroke-based rendering that imitates human drawing and can be learned from scratch without supervision. Our painting agent computes a sequence of actions that represent the primitive painting strokes. In order to ensure that the generated policy is predictable and controllable, we use a constrained learning method and train the painting agent using the environment model and follows the commands encoded in an observation. We have applied our approach on many benchmarks and our results demonstrate that our constrained agent can handle different painting media and different constraints in the action space to collaborate with humans or other agents.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors present the results of training a natural media painting agent using reinforcement learning for different types of strokes. The agent seems to be capable of learning how to pain under different types of constraints and produce visually interesting images. Comments: - Given that the authors give an implementation of the constrained RL agent as one of the key contributions of the paper, there is a glaring absence of mentioning related work on constrained reinforcement learning and reviewing the existing approaches in literature, in order to compare and contrast what the authors propose in this paper. This makes it hard for the readers to assess the novelty of the contribution. - Similarly, the authors should discuss in more detail the limitations on the types of constraints that are possible to easily express in this framework, given the simplicity of the constraints that are shown in the experiments - The readability of the paper seems like it could be improved. Apart from typos, there seem to be many long enumerations of approaches that other researchers have taken in this space, but for a reader it is not immediately obvious how these come together and relate to the work that is being presented. - With the above in mind, one thing that was conceptually unclear to me is that one of the main advantages of the proposed approach, according to the authors, when compared to some of the cited related work, is that this approach can generate intermediate representations and not just the final output with most resemblance to the reference picture. There is a mention of this opening up new artistic possibilities. Yet, this particular use case is not given central stage in evaluation. The authors should provide more examples of why this particular capability is relevant and how it leads to interesting outcomes. - The authors say: “we use 5 strokes to reproduce hand-written digits images, 20 strokes to reproduce character images, 100 strokes to reproduce face and object images” - which does seem reasonable, but the actual numbers (5, 20, 100) are not well motivated. After all, why not (10, 50, 200) or (5, 40, 100) or (5, 40, 200)? It would be good to show experimentally (for one of these) that the choice is justified. I’m not familiar with KanjiVG - but there do exist kanji characters with more than 50 strokes. I’m guessing they are not present in this dataset? - The authors should provide more details on the specifics of the model architecture and the hyperparameters that have been used / explored. - In the results/discussion of the paper, the authors should compare to prior work and highlight their novel contributions. Given that multiple papers out there that have had at first glance similar-looking results, it is hard to otherwise qualitatively judge whether what is proposed in this paper is better, without any form of side-by-side comparison.
assistant
The authors present the results of training a natural media painting agent using reinforcement learning for different types of strokes. The agent seems to be capable of learning how to pain under different types of constraints and produce visually interesting images. Comments: - Given that the authors give an implementation of the constrained RL agent as one of the key contributions of the paper, there is a glaring absence of mentioning related work on constrained reinforcement learning and reviewing the existing approaches in literature, in order to compare and contrast what the authors propose in this paper. This makes it hard for the readers to assess the novelty of the contribution. - Similarly, the authors should discuss in more detail the limitations on the types of constraints that are possible to easily express in this framework, given the simplicity of the constraints that are shown in the experiments - The readability of the paper seems like it could be improved. Apart from typos, there seem to be many long enumerations of approaches that other researchers have taken in this space, but for a reader it is not immediately obvious how these come together and relate to the work that is being presented. - With the above in mind, one thing that was conceptually unclear to me is that one of the main advantages of the proposed approach, according to the authors, when compared to some of the cited related work, is that this approach can generate intermediate representations and not just the final output with most resemblance to the reference picture. There is a mention of this opening up new artistic possibilities. Yet, this particular use case is not given central stage in evaluation. The authors should provide more examples of why this particular capability is relevant and how it leads to interesting outcomes. - The authors say: “we use 5 strokes to reproduce hand-written digits images, 20 strokes to reproduce character images, 100 strokes to reproduce face and object images” - which does seem reasonable, but the actual numbers (5, 20, 100) are not well motivated. After all, why not (10, 50, 200) or (5, 40, 100) or (5, 40, 200)? It would be good to show experimentally (for one of these) that the choice is justified. I’m not familiar with KanjiVG - but there do exist kanji characters with more than 50 strokes. I’m guessing they are not present in this dataset? - The authors should provide more details on the specifics of the model architecture and the hyperparameters that have been used / explored. - In the results/discussion of the paper, the authors should compare to prior work and highlight their novel contributions. Given that multiple papers out there that have had at first glance similar-looking results, it is hard to otherwise qualitatively judge whether what is proposed in this paper is better, without any form of side-by-side comparison.