Stack Exchange
How does caffe compute gradient when there are multiple branches?
I'm now reading Caffe source code, and the question occurred to me.
Take caffe/relu_layer.cpp for example. When computing gradient, from
void ReLULayer<Dtype>::Backward_cpu(const vector<Blob<Dtype>*>& top,
    const vector<bool>& propagate_down,
    const vector<Blob<Dtype>*>& bottom) {
  if (propagate_down[0]) {
    const Dtype* bottom_data = bottom[0]->cpu_data();
    const Dtype* top_diff = top[0]->cpu_diff();
    Dtype* bottom_diff = bottom[0]->mutable_cpu_diff();
    const int count = bottom[0]->count();
    Dtype negative_slope = this->layer_param_.relu_param().negative_slope();
    for (int i = 0; i < count; ++i) {
      bottom_diff[i] = top_diff[i] * ((bottom_data[i] > 0)
          + negative_slope * (bottom_data[i] <= 0));
    }
  }
}

we can see a value is finally assigned to bottom_diff, indicating that value is the gradient of the corresponding bottom blob.
However, when multiple layers take one blob as inputs, e.g., stacking multiple ReLU layers on one blob, how does Caffe handle the gradient computation? The first ReLU layer modifies bottom_diff, and it seems that the second ReLU layer just overrides it, instead of adding two gradients.
I didn't see anywhere performing gradient summation, and I am confuses. Please inform me if I have missed something important, and thanks a lot.

Caffe automatically inserts Split layer when a top blob is used in multiple bottoms.  This is done inside Net<Dtype>::Init(...) by a call to InsertSplits(...) from caffe/utils/insert_splits.cpp.
Example:
Original network in NetParameter protobuf object (nodes here are layers):
data ---> conv1 -> conv2 -> ...
      \-> somelayer -> ...

Net Layers in memory after Net::Init():
data -> split ---> conv1 -> conv2 -> ...
               \-> somelayer -> ...

(An interesting detail, by the way: .diff in activation Blobs is assigned to by Backward(), while .diff in layer learnable parameters is added to by Backward().)
