Designing a support reward in reinforcement learning to help the agent reaching the goal when there is an obstacle between them I have a problem where an agent (a point) evolves in a 2D world and its goal is get sufficiently near to the goal (a circle) to consider is has reached it.
The action space is made of:

*

*the speed of the agent

*the direction of the agent

The observation space is made of:

*

*the agent coordinates (x_agent, y_agent)

*the goal coordinates (x_goal, y_goal)
where x_agent , y_agent (x_goal, y_goal respectively) are continuous values between 0 and 1.

*an image of the map is provided to the learning model.

The agent receives a reward > 1 upon reaching the goal.
If it collides with the obstacle or does not reach the goal within a given number of steps, the episode terminates with a 0 reward.
To fasten the learning process, I have designed a support reward function such that when the agent is getting closer (farther) to the goal, it receives a certain positive (negative) reward such that, at time step t, the support reward is:
$r_{t} = (\frac{dist_{t-1} - dist_{t}}{dist_{0}})$
Where $dist_{t-1}$ is the Euclidean distance between the agent and the goal at step t and $dist_{0}$ is the initial distance between the agent and the goal at the start of the episode.
The issue with this reward function is that the distance does not consider if there is an obstacle between the agent and the goal. In below example, the agent has to receive successive negative support reward to be able to reach the goal, but it seems that such a support reward hinders the learning process as the agent wants to get immediately closer to the goal and then it ends up colliding with the obstacle. This support reward is very helpful to accelerate the learning process when there is no obstacle but it seems it shows its limit in the given situation.
Is there a way to design a better reward function which might helps the agent learning the expected behavior ?

Note: My actual world is made of much more complicated obstacles in terms of shapes and numbers.

It sounds like you should compute the shortest path between any two points $x, y$ in your environment, and use that as your distance function, rather than euclidean distance.
Even if an exact shortest path is intractable in your actual, more complicated environment, even an approximate shortest path distance would be much better than euclidean distance.
