Question: I'd like to know about other patterns that could be more efficient than the use of factories.

Answer: 
You could use a memory pool, the boost one is pretty good:
<http://www.boost.org/doc/libs/1_58_0/libs/pool/doc/html/boost_pool/pool.html>

And every client could configure the maximum size of the pool.
Allocations and deallocations will be really fast,and you will drop your factory implementation
Measure first. If you haven't proved that your allocator is a performance problem then you shouldn't optimize it away. Your solution to not free memory back may in fact perform worse (e.g., if you're using a simple free-list, you might be spending a lot of time on cache misses when you traverse the list).

If memory allocation is a bottleneck, look into better allocators. TCMalloc is pretty good, and there are commercial and cross-platform solutions available if you want to pay lots of money.

But use a profiler and measure before you do anything drastic.