Wide Neural Networks of Any Architecture Are Gaussian Processes
Good results often seem &quot;obvious&quot; in ex post. So take this as a compliment.<p>Intuitively (I have never read a paper in this field), since you are talking about wide networks, I also expected that a CLT would be used. For &quot;dense&quot; layers it is pretty obvious that one should be able to characterize each layer aggregation based on a CLT, and so forth. Some sort of mild independence assumption, mixing or martingales, on the sampling should be sufficient.
I think therefore the goal of a paper for any architecture would be to figure out a way to generalize this for different layers.<p>However, one thing I notice is that you seem to assume that weights (or whatever is initialized) are initialized with a Gaussian distribution?<p>That seems a bit restricted. The appeal of this approach with wide networks, I think, is that any independent initialization of weights would lead to transformations of GP.<p>Perhaps I am misunderstanding also the implications. Could you generally trace a dependence onto the distribution of the last layer weights, even if they are not normal? Or do you need the GP for your conditioning?<p>On the one hand, Gaussian initialization is I think not really encompassing &quot;all architectures&quot; as practically used, but more specifically, it seems that the end goal of this research program would be to generalize beyond this (much like in regression, one uses CLTs exactly to get away from parametric assumptions). Or is that where you plan to go?