This work investigates the performance of a composite neural network, which is composed of pre-trained neural network models and non-instantiated neural network models, connected to form a rooted directed graph. A pre-trained neural network model is generally a well trained neural network model targeted for a specific function. The advantages of adopting such a pre-trained model in a composite neural network are two folds. One is to benefit from other's intelligence and diligence, and the other is saving the efforts in data preparation and resources and time in training. However, the overall performance of composite neural network is still not clear. In this work, we prove that a composite neural network, with high probability, performs better than any of its pre-trained components under certain assumptions. In addition, if an extra pre-trained component is added to a composite network, with high probability the overall performance will be improved. In the empirical evaluations, distinctively different applications support the above findings.
The paper aims at justifying the performance gain that is acquired by the use of "composite" neural networks (e.g., composed of a pre-trained neural network and additional layers that will be trained for the new task).
I found the paper lacking in terms of writing and in terms of clarity in expressing scientific/mathematical ideas especially for a theory paper.
Example from the Abstract:
"The advantages of adopting such a pre-trained model in a composite neural network are two folds. One is to benefit from other’s intelligence and diligence, and the other is saving the efforts in data preparation and resources
and time in training"
The main results of the paper (Theorem 1,2,3) are of the following nature: if you use more features (i.e., "components") in the input of a network then you have "more information", and this cannot be bad. Here are the corresponding claims in the Abstract:
"we prove that a composite neural network, with high probability, performs better than any of its pre-trained components under certain assumptions."
"if an extra pre-trained component is added to a composite network, with high probability the overall performance will be improved."
However, this argument seems to be just about expressiveness; adding more features can be statistically problematic. 
Furthermore, why is it specific to pre-trained components? Essentially the theorems are about adding any features.
Finally, the assumption that the pre-trained components are linearly independent is invalid and the makes the whole analysis somewhat simplistic.
The motivating Example 1 just shows that the convex hull of a class of hypotheses can include more hypotheses than the class itself. I don't see any connection between this and the use of pre-training.
Other examples unclear statements from the intro:
"One of distinctive features of the complicated applications is their applicable data sources are boundless. Consequently, their solutions need frequent revisions."
"Although neural networks can approximate arbitrary functions as close as possible (Hornik, 1991), the major reason for not existing such competent neural networks for those complicated applications is their problems are hardly fully understood and their applicable data sources cannot be identified all at once."
There are many typos in the paper including this one about X for the XOR function:
"Assume there is a set of locations indexed as X = {(0; 0); (0; 1); (1; 0); (1; 0)} with the corresponding values Y = (0; 1; 1; 0). Obviously, the observed function is the XOR"