Question: How do you typically handle data ownership in these systems? 

Right now I just have a map that maps type(std::string or hashed int)->void\* where outside sources can add attributes via a simple AddAttribute method. 

Right now I'm having the problem of where and how to delete some of that data, since some of it is added via outside sources (e.g. reference to another entity) and some of it is legitimate data of the entity that contains it (e.g. health attribute). 

The most obvious solution would be to simply use shared pointers, but somehow I'm worried that their overhead will become noticeable when I have a lot of entities (see <https://stackoverflow.com/questions/3628081/shared-ptr-horrible-speed>). 

Another solution that I could come up with was to first use two distinct methods: "AddAttributeExternal" and "AddAttributeInternal" (or something similar, naming is irrelevant) where the attribute would only delete data that was added via the second. But I would like to avoid this since it introduces duplicate code and can probably be error prone when you introduce a lot of changing data.

Are there better/more optimized solutions or do I need to change the way I think about entity data altogether?

Answer: 
To make it more general than in the comments. There is nothing special in an ECS regarding memory management and object ownership. The same rules apply as to any other application.

Personally I'm an advocate of the most restricted version, clear object ownership and an explicit lifetime management for objects and memory in C++. Shared resource management *only* where it really has to be for instance for Audio-resources, Render-resources etc, but that is the exception. There is also the legitimate opinion that everything has to be smart-pointer-ed and we don't care about life time management. Each of both paradigms comes with its own set of problems, I will focus on the first one.

It basically comes down to a [Fail-fast system](http://en.wikipedia.org/wiki/Fail-fast) vs. a [Fail-safe system](http://en.wikipedia.org/wiki/Fail-safe). Despite the names, in my experience the first one produces more robust code due to facing errors early and thus solving them is easier and less time consuming when it comes to debugging. Bugs will come in both version, the difference is the visibility, a good read on it is [here](http://www.martinfowler.com/ieeeSoftware/failFast.pdf).

So I would go with the easiest version, the responsibility over the lifetime of its attributes has the Entity. This goes for a health attribute as well as for a mesh or sprite or sound. They will make sure all resources that they need are available as long as they need it. If some of those resources are shared like Textures, then it makes sense to use smart-pointers for those special cases. Btw. I don't consider performance implications of smart pointers a valid argument in the discussion whether to use them or not. If you need the functionality they offer, then go for it. Just pick wisely *when* you need the functionality.
As Maik's answer says, the difference between reference data and value data (and how to manage the lifetime of references) presents the same challenges to any system that offers both types, and the use of components here doesn't really change anything. The main problem you have is that you're trying not to hard-code any types into an entity, which means you don't have the compiler helping you with distinctions between values and references.

In this situation I'd be tempted to treat everything on an entity as a value type that only lives as long as the entity does. These values could include references or names to long-lived objects elsewhere (eg. Entity IDs, textures, etc). Ideally you don't want every attribute being a reference type, eg. a pointer to an integer in the case of health.

How to manage their lifetime can be a bit trickier, but often you don't really need to delete anything during play and you don't need to worry about references outside of play, so it might suffice to load in all resources at the start of a level and free them at the end.

It's also worth noting that there's no intrinsic reason why you need to have an `AddAttribute` system that allows arbitrary properties. That level of genericity might be more trouble that it's worth, which is why many component-based systems have their modularity at the component level - ie. once you have successfully queried for a component, the system can be assured of the attributes that it contains (and can resolve references to them at compile time, in compiled languages).