PURPOSE: Phase I/II trial of topotecan, fludarabine, cytarabine, and filgrastim followed by peripheral stem cell transplantation or isotretinoin in treating patients who have acute myeloid leukemia, myelodysplastic syndrome, or recurrent or refractory acute lymphocytic leukemia.