Question: If I download a module which is not stable to run in Drupal (red color over download link), is this causing issues to my drupal installation even if it is not enabled ?

In other words, if I enable it, and use it.. could it cause issues to other modules or drupal core that remain there even after I've disabled it ?

thanks

Answer: 
If a bug in the module messes up configuration settings or other data in your database, these will remain even after you uninstalled the module. So yes, it could cause issues for other modules or Drupal core, even after removal.

Note that this could happen with 'stable' modules as well, as the decision on when to declare a module as 'ready for production' is more or less up to the maintainer!

This is one of many reasons to **never, ever** test new modules (stable or not) directly on a production install. Create a test/staging install first, install the new module(s) and test, test, test.

The same is valid for updates, even for Drupal core updates. **Always** do a test run on a separate install first (at the very least, make sure you have a working, restorable backup of your database and code that you can switch back to, in case anything goes wrong).
There are two things you need to do which may help

1. Clear the cache manually Under admin->performance
2. Uninstall the module (there is a tab at the top of the modules page to allow you to do this).

Modules which are well written should not affect your site after these steps are taken, but they may corrupt the data which other modules use. Which would lead to more difficult to fix issues.