PURPOSE: This phase I/II trial is studying the side effects and best dose of panobinostat when given together with letrozole and to see how well it works in treating patients with metastatic breast cancer.