This phase Ib/II trial studies how well sorafenib tosylate and pembrolizumab work in treating patients with liver cancer that has spread to other parts of the body. Sorafenib tosylate may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Monoclonal antibodies, such as pembrolizumab, may interfere with the ability of tumor cells to grow and spread. Giving sorafenib tosylate and pembrolizumab may work better in treating patients with liver cancer.