Is Django admin difficult to customize?
Django Admin is easy to customize if your requirements match what is customizable in Django. If you need fancy features that are not out-of-the-box functionality you either need to find a plugin that does it, or implement it yourself by subclassing the appropriate classes. Which can be difficult, and requires good understanding of the internals of Django.
So in summary: it makes it easier to setup an admin, but if you need anything special that's not out of the box: you got a steep learning curve.
Depending on your requirements you can choose to use django or not. Anything that requires a lot of functional speccing is better of implemented manually.
I have been playing for a couple of days with Django Admin to explore it, but I am still clueless of how it can be customized in the way we need. 
Every time I look for any help for the customization in the admin panel, what I find is, a bunch of articles on various communities and forums, explaining how to customize the template, the lists items, the the column views etc. But that should not be called Django Customization. 
If I need to make even a small change in the User functionality or any modification to the auth module. It takes a lots of efforts even in figuring out how that can be done.
Is Django that difficult to customize or its just lack of the help available over internet for this or its me who is moving in the wrong direction ?