Is there a way to have dictinary-like field in Django model?
You should create an ForeignKey field between the user model and the voting model. To only allow a single vote, per user; per movie, you might want to create a unique key constraint, on the voting model, over the userid and the movieid.
The vote should then contain a relation to the movie and a rating. If a user withdraws his vote you remove it from database.
Using a dictionary will lead to problems for you as you will have multiple movies.
To increase the speed and performance counting the votes you might want to take a look at caching and simply cache the number of votes and update the number every time a vote was added/withdrawn by a user on the specific movie.
Suppose, I am making a movie rating app. A logged-in user should be able to rate the movie with stars (in the range 1 to 5).
I want to quickly access all the rater's name along with their rating. 
If a user rates the movie again, the rating should be updated. At the same time, if a user decides to withdraw his rating i.e. provide zero rating, I would like to remove the entry from the field.
I believe dictionary would be the best choice to achieve the same. However, I am open for suggestions.
I also want a user to see all the movies the he/she has rated along with the rating.