Yes, that is what I had thought too. But from this guide 2:

If labels is multi-dimensional, then average the per_example_loss across the number of elements in each sample. For example, if the shape of predictions is (batch_size, H, W, n_classes) and labels is (batch_size, H, W) , you will need to update per_example_loss like: per_example_loss /= tf.cast(tf.reduce_prod(tf.shape(labels)[1:]), tf.float32).

In my case, the perceptual loss and the L1 loss both satisfy this criterion. So, this is what I did and things seem to be working fine:
perceptual_loss = ...
perceptual_loss /= tf.cast(tf.reduce_prod(tf.shape(activations)[1:]), tf.float32)
perceptual_loss = tf.nn.compute_average_loss(perceptual_loss, global_batch_size=BATCH_SIZE)
perceptual_loss = alpha * perceptual_loss

reconstruction_loss = tf.keras.losses.mean_absolute_error(color_images, reconstructed_images)
reconstruction_loss /= tf.cast(tf.reduce_prod(tf.shape(color_images)[1:]), tf.float32)
reconstruction_loss = tf.nn.compute_average_loss(reconstruction_loss, global_batch_size=BATCH_SIZE)
reconstruction_loss = beta * reconstruction_loss

total_loss = perceptual_loss * reconstruction_loss