Actually, by default (and as per Abadi et al. 1) we clip the L2 norm of the entire gradient vector for a given sample, i.e. vector consisting of gradients for all trainable parameters in the model stacked together.
We’ve tried experimenting with different approaches to clipping, e.g. dynamic per-layer thresholds (see implementations in clipping.py 3), but it wasn’t too fruitful