What did Buddha say about dysfunctional families for novice practitioners? What did Buddha say about dysfunctional families for novice practitioners? What advice did he give for novice practitioners in order to have a good practice as being a novice such impediments as a dysfunctional and painful family can impact practice in a negative manner? What precautions should one take? Should one go no-contact, is that skillful?

Elsewhere, Buddha has mentioned that one cannot repay the debts of the parents and even in the Mangala Sutta it is mentioned that one should be helpful to relatives. However, if being in contact with dysfunctional families makes us lose our faith in ourselves and makes us angry and hateful all the time, this is a huge impediment to practice especially as a novice. I know that one should learn to have compassion for these people and see them as afflicted by delusions. However the transgressions are so huge and the fact that I am a novice makes me feel guilty that I cannot feel the compassion. I have tried Metta meditation and also analyzing them using the 5 ways to overcome hatred. At this point nothing is helping. I am just filled with this ball of anger and hatred inside me constantly. I am afraid that I am generating more negative mind-states by being in contact with this dysfunctional system. There is a lot of dysfunction and its so painful, to the point that I am losing my self-esteem and have started doubting my worth as a human -being. There is constant lies, manipulation, slandering... I am losing my faith in myself as a human being.

All advice to deal with this situation from the Suttas, and practical advice from personal experiences, will be appreciated. Please remember, this is for a novice. High spiritual ideals like compassion etc are great, but they are not helpful for me right now.
I can only agree with Dhammadhatu's last paragraph. Get yourself going first, for  if you are suffering inside you cannot help others and your inner rescources will become less and less. This is not about being selfish but I call it enlightened self interest.
Now, the Dhammapada is largely addressed to monks, but I think some of the quotes are still good reminders and something to strive for. 
To quote from the Dhammapada:


  Verse 160: One indeed is one's own refuge; how can others be a refuge to one? With oneself thoroughly tamed, one can attain a refuge (i.e., Arahatta Phala), which is so difficult to attain.


_


  Verse 61: If a person seeking a companion cannot find one who is better than or equal to him, let him resolutely go on alone; there can be no companionship with a fool.


and lastly verse 43, which targets your issue and echoes Dhatus last point: 


  Verse 43: Not a mother, nor a father, nor any other relative can do more for the well-being of one than a rightly-directed mind can.


Apart from Buddhism, I can also recommend REBT (Rational Emotive Behaviour Therapy) by Albert Ellis who was the "father" of cognitive behavioral therapy. Especially Windy Dryden published countless books in how to overcome obstacles (search on Google Books, I learned most of it through Book reviews there). One of the most important hallmarks in REBT is self acceptance, and if possible other and world acceptance, where you differentiate between personhood (or supposed "self/other worth") and the doer. Basically "condeming the sin but not the sinner". It would definitely help with your self esteem issue you raised. On YouTube, Albert Ellis is also to be found. There are old lectures, but still on point and have a no non-sense approach which I like and is usually quite rare to be found in comparison to Buddhism, where lectures and talks go up to 2-3 hours with a lot of non-sense and rambling.