I have a written an app, with python acting as a simple web server (I am using bottle framework for this) and an HTML + JS client. The whole thing runs locally. The web page acts as GUI in this case.
In my code I have implemented a file browser interface so I can access local file structure from JavaScript.
The server accepts only local connections, but what bothers me is that: if for example somebody knows that I am running my app locally, and forges a site with AJAX request to localhost? and I visit his site in some way, will my local files be visible to the attacker? 
My main question is: is there any way to secure this? I mean that my server will know for sure that the request came from my locally served file?
The most direct way to protect against this attack is to just have a long complex secret key being required for every request. Just make your local code authenticate itself before processing the request. This is essentially how web services on the Internet are protected.
You might also want to consider having inter process communication in some other form like DBUS or unix sockets. I'm not sure which OS you are on but there are many options for inter process communication that wouldn't make you vulnerable in this way.
Secure localhost JavaScript ajax requests