need to maintain a symmetric library for the origin of course the edge server might well have the chunks in cache but if it does not it follows the usual cdn like procedure to retrieve them e g through a cache hierarchy or the like ultimately retrieving them from the dedupe cache parent if necessary the ghost process has the capability of determining whether a request is to be handled by the deduplication process one technique for making this determination uses tenant specific metadata and the technique described in u s pat no 7 240 100 the dedupe module may run as a buddy process or an in process library with respect to ghost the communication mechanism between ghost and the module may be over shared memory localhost tcp uds or the like in an alternative embodiment the client side dedupe module itself may be placed directly on a client device such as an end user client euc network machine a mobile device handset or the like preferably whether dedupe is turned on may be controlled by metadata configurations preferably on a per tenant basis as noted above preferably the dedupe mechanism is not invoked for files that are too small small object aversion support thus provides a way to intelligently avoid performing otherwise risky deduplication operations that might incur an extra rtt on a cache miss in one approach this may be accomplished by having ghost bypass the dedupe operation for posts and responses that include a u201c content length u201d header under a certain threshold most dynamic content however uses chunked transfer encoding which means that the size of the object is not known in advance thus absent.