Looking at the numerous meanings and usages of the preposition
"zu" we can only give an incomplete answer here to cover some
special cases where we have a loose rule indeed. Whenever we use
"zu" we also indicate some spacial correlation, a direction, or
other defined relations. Der Kellner beugte sich zu dem Gast
herab. Alle Besucher sind zu dem Fest gekommen. When we consider
this then a meaning of a sentence may change: Er hat mir ein
Muster gebracht. - He gave me a sample. Er hat ein Muster zu mir
gebracht. - He brought the sample to me. Er gehört zu mir. He
became a part of me. Er gehört mir. "He is mine." Now let's look
at both of your examples: Er hat etwas zu mir gesagt und nicht
zu jemand anderem. Er hat [zu] mir etwas gesagt, was andere
nicht hören sollten. Both have nearly the same meaning, but in
the first case it is important that the speech was directed to
you. Hence you can not leave out the "zu". This is different in
the second case. The direction is not important, you leave out
the "zu". When