This phase I/II trial studies the side effects and best dose of temozolomide and pazopanib hydrochloride when given together and to see how well they work in treating patients with advanced pancreatic neuroendocrine tumors (PNET) that cannot be removed by surgery. Drugs used in chemotherapy, such as temozolomide, work in different ways to stop the growth of tumor cells, either by killing the cells or by stopping them from dividing. Pazopanib hydrochloride may stop the growth of tumor cells by blocking some of the enzymes needed for tumor growth. Giving temozolomide together with pazopanib hydrochloride may be an effective treatment for patients with PNET.