  The ability to filter unwanted light signals is critical to the operation of
quantum memories based on neutral atom ensembles. Here we demonstrate an
efficient frequency filter which uses a vapor cell filled with $^{85}$Rb and a
buffer gas to attenuate both residual laser light and noise photons by nearly
two orders of magnitude with little loss to the single photons associated with
our cold $^{87}$Rb quantum memory. This simple, passive filter provides an
additional 18 dB attenuation of our pump laser and erroneous spontaneous
emissions for every 1 dB loss of the single photon signal. We show that the
addition of a frequency filter increases the non-classical correlations and
readout efficiency of our quantum memory by $\approx35\%$.

Quantum Physics