This phase II trial studies how well ibrutinib and brentuximab vedotin work in treating patients with Hodgkin lymphoma that has returned (relapsed) or does not respond to treatment (refractory). Ibrutinib may stop the growth of cancer cells by blocking some of the enzymes needed for cell growth. Monoclonal antibodies, such as brentuximab vedotin, may interfere with the ability of cancer cells to grow and spread. Giving ibrutinib together with brentuximab vedotin may be a better treatment for Hodgkin lymphoma.