The complex permittivity and return loss (RL) for the composites of silicone rubber filled with holey graphene nanosheets (HGNS, prepared by ultra-rapid heating during the step of thermal reduction/exfoliation of graphite oxide) were measured in the 3–18 GHz range. HGNS-based composites were found to have significantly higher microwave absorption than composites incorporating other types of graphene reduced at lower heating rates. Even with only 1 wt.% loading, its experimentally measured RL reached −32.1 dB at 13.2 GHz with a thickness of 2 mm, and simulation suggested that at a thickness of 3 mm its RL can be as low as −45.3 dB at 7.8 GHz. Material characterization indicated that the density of the holes increased with the temperature ramp rate, and the hole sizes ranged from 5 to 300 nm. Compared to other graphene samples, HGNS possessed significantly larger specific surface area and higher density of defects, suggesting that defect-induced losses, interfacial polarization, and multiple reflection/scattering at the interfaces are the major loss mechanisms. Our simple and low-cost process as well as the very low loading ratio of HGNS are advantageous for cost reduction in future applications.