How is it wrong to believe that a self exists, or that it doesn't? the one place where the Buddha was asked point-blank whether or not
  there was a self, he refused to answer. When later asked why, he said
  that to hold either that there is a self or that there is no self is
  to fall into extreme forms of wrong view that make the path of
  Buddhist practice impossible.


It seems to me, and I may be wrong, that the skandhas can be identified as a self, as long as they aren't then taken to be in any way unchanged from moment to moment



Encyclopedia of Buddhism, By Damien Keown, Charles S. Prebish


So I was wondering if that above silence means that any dharma can fulfill the role of that empirical self.




An example would be: Adam tastes the apple

Supposing this "taste" can be considered a kind of self, then if Adam sees smells touches feels nothing, just tastes the apple, there is "continuity" and he's still Adam. But as long as he does have other senses, and from moment to moment, the taste of apple doesn't make him who is he.

I've never read any commentary which claims this is what the empirical self is in Buddhism, so would be really surprised that this counts as a categorical denial of substantial self.
You should not believe or hold a view that:

there is a self
there is no self

But whatever you consider as self is not worthy of being called self as:

you cannot control it to your will
it is impermanent

Since if you take a being as parts in terms of the 5 aggregates or 6 faculties each part which constitutes a being is not self. E.g. the eye is not self, what you see is not self, what you feel is not self, your corporeal body is not self, etc.