St. Anne's Church (facing Anne Street South) has some colorful 19th-century stained glass, and provides the setting for lunchtime concerts (look for the notices in the vestibule).
Concerts are only offered in St. Anne's Church in the evenings.