Remove django sites app
If you are sure that there are not any dependencies from some other files in your project or in your apps you can safely remove them. 
First comment out one by one and every time check the project in your browser if it is running correctly. Also check the logs for warnings and errors.
I'd like to know if I can just ditch the Sites default app (comment it out from INSTALLED_APPS and so on) without breaking anything?
It's written in the doc that some other parts of django use it (redirects framework, comments, flatpages, syndic, auth, shortcut and view on site), but it's not explicitly said if it's going to break them. Is it?
Django 1.5