summaries: This paper borrows the idea of word-to-vector from NLP and applies it in reinforcement learning based Neural Architecture Search (NAS). It suggests a pretrained encoder to transform the search space to a dense and continuous architecture-embedding space. First it trains the architecture-embedding encoder and decoder with self-supervision learning like Auto-Encoder.  Then it performs reinforcement learning based Neural Architecture Search(NAS) in the architecture-embedding space.

summaries: The paper proposes an interesting idea to perform Neural Architecture Search: first, an auto-encoder is pre-trained to encode/decode an neural architecture to/from a continuous low-dimensional embedding space; then the decoder is fixed but the encoder is copied as an agent controller for reinforcement learning. The controller is optimized by taking actions in the embedding space. The reward is also different from previous works which usually only considered validation accuracy but this work also considers the generalization gap.

abstractText: The neural architecture search (NAS) algorithm with reinforcement learning can be a powerful and novel framework for the automatic discovering process of neural architectures. However, its application is restricted by noncontinuous and highdimensional search spaces, which result in difficulty in optimization. To resolve these problems, we proposed NAS in embedding space (NASES), which is a novel framework. Unlike other NAS with reinforcement learning approaches that search over a discrete and high-dimensional architecture space, this approach enables reinforcement learning to search in an embedding space by using architecture encoders and decoders. The current experiment demonstrated that the performance of the final architecture network using the NASES procedure is comparable with that of other popular NAS approaches for the image classification task on CIFAR10. The beneficial-performance and effectiveness of NASES was impressive even when only the architecture-embedding searching and pre-training controller were applied without other NAS tricks such as parameter sharing. Specifically, considerable reduction in searches was achieved by reducing the average number of searching to<100 architectures to achieve a final architecture for the NASES procedure.

authors:
Bowen Baker
Otkrist Gupta
Nikhil Naik
Ramesh Raskar

title: Designing neural network architectures using reinforcement learning

authors:
Jonathon Shlens Barret Zoph
Vijay Vasudevan
Quoc V. Le

title: Learning transferable architectures for scalable image recognition

authors:
Andrew Brock
Theodore Lim
J.M. Ritchie
Nick Weston

title: Smash: One-shot model architecture search through hypernetworks

authors:
Han Cai
Tianyao Chen
Weinan Zhang
Yong Yu
Jun Wang

title: Reinforcement learning for architecture search by network transformation

authors:
Gabriel Dulac-Arnold
Ludovic Denoyer
Philippe Preux
Patrick Gallinari

title: Fast reinforcement learning with large action sets using error-correcting output codes for mdp factorization

authors:
Gabriel Dulac-Arnold
Richard Evans
Hado van Hasselt
Peter Sunehag
Timothy Lillicrap
Jonathan Hunt
Timothy Mann
Theophane Weber
Thomas Degris
Ben Coppin

title: Deep reinforcement learning in large discrete action spaces

authors:
Thomas Elsken
Jan Hendrik Metzen
Frank Hutter

title: Neural architecture search: A survey

authors:
Hado Van Hasselt
Marco A. Wiering

title: Using continuous action spaces to solve discrete problems

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

authors:
G.E. Hinton
R.R. Salakhutdinov

title: Reducing the dimensionality of data with neural networks

authors:
Andrew Hundt
Varun Jain
Gregory D. Hager

title: sharpdarts: Faster and more accurate differentiable architecture

authors:
Sergey Ioffe
Christian Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

authors:
Diederik P. Kingma
Jimmy Lei Ba

title: Adam: A method for stochastic optimization

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny images

authors:
Hanxiao Liu
Karen Simonyan
Yiming Yang

title: Darts: Differentiable architecture

authors:
Ilya Loshchilov
Frank Hutter

title: Sgdr: Stochastic gradient descent with warm restarts

title: A method of solving a convex programming problem with convergence rate o(1/k2)

authors:
Jason Pazis
Ronald Parr

title: Generalized value functions for large action sets

authors:
Hieu Pham
Melody Y. Guan
Barret Zoph
Quoc V. Le
Jeff Dean

title: Efficient neural architecture search via parameter sharing

authors:
Samuel L. Smith
Pieter-Jan Kindermans
Chris Ying
Quoc V. Le

title: Don’t decay the learning rate, increase the batch size

title: Learning from delayed rewards

title: Simple statistical gradient-following algorithms for connectionist rein- forcement learning

authors:
Zhao Zhong
Junjie Yan
Wei Wu
Jing Shao
Cheng-Lin Liu

title: Practical block-wise neural network architecture generatio

authors:
Barret Zoph
Quoc V Le

title: Neural architecture search with reinforcement learning

heading: 1 INTRODUCTION

text:
Deep neural networks have enabled advances in image recognition, sequential pattern recognition, recommendation systems, and various tasks in the past decades. However, selecting a suitable neural architecture is frequently arduous because of the classical and new neural architectures emerging daily. In general, manual design of network architectures according to the cases is achievable. However, hyperparameter tuning and architecture engineering through manual selection requires considerable time. Furthermore, manually designing a neural network architecture requires substantial experience in deep learning.
Given the aforementioned reasons, neural architecture search (NAS), an automated architecture engineering, has been successful in the past years. The NAS algorithm is divided into three dimensions, namely search space, search strategy, and performance estimation strategy (Elsken et al., 2019). Outstanding results have been achieved using NAS with the reinforcement learning search strategy (Zoph & Le, 2017). Here, a recurrent network was used to generate a string to form a child network. However, such a type of network exhibits two problems: noncontinuous and highdimensional search space. The frequent large strings of action from the recurrent network and the discrete space result in difficulty in optimization. The critical contribution of this study is the improvement of the dimension and quality of the search space that could provide a more efficient framework to solve the two problems of searching architectures.
If a vector that can represent network architecture without discrete values is determined, then the noncontinuous aforementioned disadvantages can be addressed. We proposed the NAS in embedding space (NASES) method, which involves mapping origin architecture to architecture-embedding by using an architecture encoder. The advantage of embedding space includes the lower-dimensional and continues space, it considerably alleviates the difficulty in the optimization problem of the NAS procedure with reinforcement learning. To learn and search on the embedding space, we developed a mechanism to generate architecture encoder and decoder to promote origin architecture communication with the embedding space, and the autoencoder network was used in the mechanism (Hinton
& Salakhutdinov, 2006). The architecture simulator simulates the origin architecture space, which assists the real architecture encoder learning. The decoder realizes the relationship between origin architecture and architecture-embedding, which maps the architecture-embedding to the origin architecture.
The NASES procedure was implemented in two stages. We obtained a pretraining architecture decoder and a pretraining architecture simulator in the first stage and provided the compression rate between the embedding size and testing loss. In the second stage, we used the NASES procedure for image classification on CIFAR-10 by using the network pretrained in the first stage. The results of the experiment were efficient and indicated that NASES was highly efficient and considerably reduced the number of searching architectures to <100 in <12 GPU hours. Thus, the results were comparable with that of other popular NAS methods.

heading: 2 RELATED WORK

heading: 2.1 REINFORCEMENT LEARNING WITH ACTION EMBEDDING

text: Reinforcement learning is a general approach that can be applied broadly to various areas. However, the large and discrete action space causes problems in function approximation. The majority of studies have focused on two approaches, one approach factorizes the action space into binary subspaces (Pazis & Parr, 2011; Dulac-Arnold et al., 2011). The other approach involves embedding discrete actions into a continuous action, determining optimal actions in the continuous space, and selecting the nearest discrete action to reduce the scaling of action sizes (Dulac-Arnold et al., 2015; Hasselt & Wiering, 2009).

heading: 2.2 SEARCH STRATEGY WITH REINFORCEMENT LEARNING

text: Our search strategy is based on reinforcement learning. Zoph & Le (2017) provided a novel NAS framework, which incorporated reinforcement learning and applied it to two agents of the child network and the controller of the recurrent network. The child network generated neural architecture that can be considered the action of the controller network. Unlike the use of policy gradient by Zoph & Le (2017), Williamsu (1992), and Baker et al. (2016) used q-learning to update the weight of the network.

heading: 2.3 NAS WITH CONTINUES VECTOR

text: Most NAS procedures use a discrete search space. Unlike other approaches, such as the learning over discrete and nondifferentiable search space, Liu et al. (2018) proposed an approach of differentiable architecture search (DARTS), which was based on the continuous relaxation of the architecture representation. On the basis of DARTS, Hundt et al. (2019) proposed sharp DARTS, which is a more general, balanced, and consistent design. The closest concept to NASES is the approach proposed by Luo et al. (2018) in which an encoder and a decoder was to map neural architectures in a continuous space on gradient-based optimization and a predictor was used to achieve embedding accuracy.

heading: 3 METHODS

text: In this section, to elucidate the NASES procedure, we followed the aforementioned three dimensions: search strategy, search space, and performance estimation strategy.

heading: 3.1 SEARCH STRATEGY

text:
The search strategy is a search method for fast and accurate exploration of the space of neural architectures and involves techniques such as reinforcement learning, evolutionary algorithm, and gradient-based method. These are popular strategies in NAS.
In the NAS with reinforcement learning, which is generally designed with two components of the controller network and child network (Figure 1), the controller network is usually used to control the
child network architecture and generates string and the child network construct neural architecture by using the output of the controller in each NAS iteration. The controller network calculates the policy gradient to update the network by validating the performance of child network. Thus, severe penalty is imposed when performance is low. The controller that is constructed using a multilayer perceptron rather than the recurrent neural layer, generates such continuous value, rather than a string, in the NASES. This is discussed in the next subsection.

heading: 3.2 SEARCH SPACE

text: The main contributions of NASES is in the search space domain, which resolves the two aforementioned problems of noncontinuous and high-dimensional space in reinforcement learning; these problems lead to difficult optimization. NASES is similar to the general NAS procedure, which also includes the child network and controller network. The optimization of maximize accuracy is also used as a policy gradient method. However, our method differed from the general NAS procedure; first, we developed an architecture encoder as the controller network to control the architecture of the child network and projected origin architecture into architecture-embedding. Second, we devised the architecture decoder network, which decodes architecture-embedding from the architecture controller network to the origin architecture to ensure the child network can understand and generate network by using architecture-embedding (see Figure 2). That is, to alleviate these problems, the architecture decoder functions as a translator to translate low-dimension embedding into high-dimension vector for smooth childcontroller network communication. Furthermore, the search space was bounded using micro search in this study. Thus, we did not apply the cell-based trick of hierarchical representation because we attempted to search neural network on the complete architecture and not only on the cell.

heading: 3.2.1 THREE PRINCIPAL FUNCTIONS OF NASES.

text:
The NASES, has three principal functions. This section describes the functions of the architecture decoder, architecture simulator, and controller network.
Architecture Decoder To obtain architecture-embedding decoder, first we created an approximate of virtual distribution transformation, which projected the low-dimension space into high-dimension space. That is, we transformed architecture-embedding into origin architecture.
fθ : Rn → Rm
a = fθ(â)
where fθ is a decode function parameterized by θ,Rn is the architecture-embedding space,Rm is the origin architecture space, a is the set of origin architecture, â is the set of architecture-embedding. This function released the controller network architecture and could be projected on another space not bounded on the origin architecture space. This function is efficient and provides distribution transformers. To develop this approximator, an architecture simulator is required, which is discussed in the next subsection.
Architecture Simulator An architecture simulator is an approach of the approximator of the virtual distribution transformation; its purpose creation of a function that simulates distribution to achieve architecture-embedding. Furthermore, the distribution of simulation is not limited to discrete or continuous space.
sv : U → Rn
â = sv(U)
where sv is a function parametrized by v and U is a uniform distribution.
Obtaining the architecture encoder and architecture simulator To obtain the architecture encoder and simulator, we used the autoencoder network (Hinton & Salakhutdinov, 2006). The autoencoder network is a unsupervised algorithm for distribution transformation and dimension reduction. The autoencoder generates a representation by using the reduced encoding closest to its original input. Therefore, the architecture encoder and simulator were assembled in the autoencoder network. We pretrained an autoencoder network before training the controller, in which input space and target space had the same distribution. We used uniform distribution. For convenient policy gradient learning, the activation function of the middle layer used was sigmoid; which leads the output with Bernoulli distribution.
Controller A controller controls the child neural architecture. The neural architecture and hyperparameters of the controller are copied from the architecture simulator. The input is the child state of network, and the output is architecture-embedding with Bernoulli distribution.
gk : Rm → Rn
where gk is an encode function parameterized by k, Rn is the architecture-embedding space, Rm is the origin architecture space, A is the set of origin architecture, and â is the set of architectureembedding. We did not retrain the controller on the NASES procedures again and only fine-tuned weights. The controller network can explore the architecture-embedding if the weights are initialized using the pretraining simulator of the autoencoder network because the pretraining simulator
can project uniform distribution into architecture-embedding. Another advantage is fast searching because the controller network is not required to learn projecting uniform distribution into architecture-embedding again and focuses on searching architectures. Moreover, we devised a new reward function by using the accuracy of the child network in training and validation. The reward is obtained only from the accuracy of the validation set of the past NAS with reinforcement learning. In this study, the reward function is different from the general function, and it not only considers accuracy on the validation set but also estimates generalization errors (Eq. 1).
Reward =(ACCval − (ACCtr −ACCval))3
=(2ACCval −ACCtr)3 (1)

heading: 3.2.2 CHILD MODEL

text: The child model receives continuous vectors from the controller and generates neural architecture by using the pretrained decoder model. Here, we described the NASES mechanism for creating a network architecture. We required four hyperparameters, namely number of filters, filter size, kernel type, and connection coefficient, in a layer. More details are in Appendix A.

heading: 3.3 PERFORMANCE ESTIMATION STRATEGY

text:
To reduce the computational burden, we used two approaches of obtaining lower fidelities to estimate performance. First, the learning scheduler follows cosine annealing with lmax = 0.05, lmin = 0.001 and T0 = epochs (Loshchilov & Hutter, 2017). Second, each architecture search was run for 10 epochs on the search phrase, and final architecture is run for 700 epochs. The detailed is presented in Alg 1.
Algorithm 1 Neural Architecture Search in Embedding Space Require: uniform distribution U ; architecture simulator sv; architecture decoder fθ; controller g;
child network c number of optimization iterations L; number of epochs in training phase e1; number of epochs in final phase e2;
1: sv and fθ were assembled in an autoencoder network 2: Train the autoencoder network and evaluate its performance by uniform distribution U . 3: The controller g weights initialized with architecture simulator pretrained weights v. 4: for l=1, ..., L do 5: Generate child network c by architecture a. 6: Train and evaluate data child network c in epochs e1 by training and validation set. 7: Compute reward r from train and validation performance. 8: Train controller network gv using reward r. 9: Project architecture a into architecture-embedding â by using the controller network gv .
10: Decode architecture-embedding â into architecture a by using the architecture decoder fθ. 11: end for 12: Train and evaluate on the child network with the highest reward in epoch e2 by training and
testing set.

heading: 4 EXPERIMENTS

text: We describe two stages of the experiment in this section. The first stage involves training an architecture decoder and architecture simulator network and selecting an appropriate compression ratio of architecture-embedding. The second stage involves applying the result of first stage to discover novel neural architectures for image classification on CIFAR-10 (Krizhevsky & Hinton, 2009) by using NASES.

heading: 4.1 FIRST STAGE: PRETRAINING ARCHITECTURE DECODER AND SIMULATOR NETWORK

heading: 4.1.1 DATASET

text: In the first stage of NASES experiment, our goal was to map origin architecture to architectureembedding. We assembled the architecture simulator and decoder in an autoencoder network and trained this autoencoder network instead of training the simulator and decoder. To mimic the origin architecture space, we sampled 300000 as a training set data from uniform distribution and sampled 100000 as the testing set data. In this case, we sampled uniform distribution at the interval [0, 30].

heading: 4.1.2 TRAINING DETAILS

text: The optimization of the autoencoder network was achieved by using an Adam (Kingma & Ba, 2015) optimizer with a learning rate of 0.00001. During the training of the autoencoder network, the learning schedule is increased the batch size instead of decaying the learning rate (Smith et al., 2018) and saved weights with the lowest test loss during testing. In the network architecture, the policy gradient methods update the probability distribution of actions so that the controllers actions with high expected reward exhibit a high probability for an observed state. Therefore, the activation function is a sigmoid function used in the middle layer hidden of the autoencoder network, and it leads the distribution of hidden output to Bernoulli distribution, suitable for computing the policy gradient to update the controller network. Other details regarding the experimental procedures are as follows: Three fully connected layers with the number units of 1000, 500, and 100 were used for the simulator and decoder networks; the first layer exhibited the ReLu activation function, and the second and third layers exhibited the tanh activation function. The architecture decoder network exhibited the same neural architecture and hyperparameters setting as the architecture simulator network. The loss function used a least square error.

heading: 4.1.3 RESULT

text: To understand the loss of information from the embedding space, we evaluated the compression ratios of 0.83, 0.67, 0.5, 0.33, 0.17, 0.08, and 0.02 to examine the utility of compression. We set an upper bound of testing loss as a baseline, because if the decoder is incapable of decoding on architecture-embedding, the best strategy always is the predicted average value (average is 15 in this case). The utility of compression ratio on different sizes of embedding by using the autoencoder network is presented in Table 1. The testing loss represents the loss of information after compression by the autoencoder network, and the loss of information increases with the compression ratio. Appendix B, Figure 3 illustrates the double-axis plot of the testing loss and embedding size (according to Table 1). As illustrated in Appendix B, Figure 3, a definite trade-off exists between loss and compression rate; high compression rate leads to high information loss. In this case, we suggested a range of 20-30 as the appropriate embedding size by using the double-axis plot with cross-area of testing loss curve and embedding size curve. In this range, the compression rate is more than half and information loss is considerably low. For image classification (second experiment stage) we followed the experiment results of the first stage, and the set the compression rate at 0.33. That is, the origin size of 60 was projected into the embedding size of 20.

heading: 4.2 SECOND STAGE: IMAGE CLASSIFICATION ON CIFAR-10

heading: 4.2.1 DATASET

text: The second stage of the experiment is a multiclass classification for assigning a class to the image object. The CIFAR-10 (Krizhevsky & Hinton, 2009) data set consists of 60000 color images of 32 32 RGB in 100 classes. Each class has 6000 images with 5000 training data and 1000 testing data. Additionally, to achieve standardization and normalization, we applied only three standard data augmentation techniques: (1) Subtracting the mean and then dividing the answer by the standard deviation, which ensures that all variables have mean zero and standard deviation 1. (2) Centrally padding on training set to 40 40 and randomly cropping images back to 32 32. (3) Randomly flipping images horizontally.

heading: 60 60 1 1.74 2.23

heading: 4.2.2 SETTINGS

text:
The spilled validation ratio was 0.9; we then randomly split 45000 and 5000 images for training and validation, respectively, in the neural searching procedure. Finally, we used 50000 images for training and 10000 images for testing when the NASES search procedure was complete. Each architecture search procedure was run for 10 epochs on the search phase, and the final architecture was run for 700 epochs.
The child network is described in the paragraph following the method section. The hyperparameters setting of the child network considers ENAS (Pham et al., 2018) as a reference. It was trained with Nesterov momentum, the momentum of 0.9 (Nesterov, 1983). The learning schedule followed the learning rate decay with a cosine annealing for each batch (lmax = 0.05, lmin = 0.001, T0 = epochs) (Loshchilov & Hutter, 2017), batch size of 128, weight decay of 1e-4. We initialize w with He initialization(He et al., 2015) in the child network. We designed a 15-layer convolutional architecture by using 60 hyperparameters ( a layer for four hyperparameters contains: number of filters, filer size, kernel type, and connection coefficient). The NASES mechanism provided effective mechanism, and it only required applying 20 hyperparameters on embedding vector by using the 0.33 compression ratio. By following the controller network perspectives, we took the pretrained parameters of an already trained model from the first phase experiment of the pretrained architecture simulator. The controller hyperparameters setting and neural architecture followed the first stage of the experiment.
According to the reward function engineering perspectives, Eq. 1 was used as our reward function; this function not only considered accuracy of the validation set but also estimated the generalization error. We stored the reward in a reward pool in each NASES iteration. Furthermore, we normalized and updated the rewards of the reward pool at the end of each iteration. To prevent the dependency of the final architecture on initial architectures, we ran random architectures ten times to collect rewards without updating parameters of the controller network in the beginning. We sampled three architecture examples from reward pool to update the controller network in each NASES iteration. Furthermore, the epsilon-greedy approach (Watkins, 1989) occurred randomly with the probability epsilon. Therefore, we have 10 % to generate architecture randomly to out of the reward pool too.

heading: 4.2.3 RESULT

text:
We ran the NASES procedure five times by using different random seeds on a single Nvidia V100 GPU, and NASES required approximately 12 hours to determine the final architecture for a NASES procedure. Furthermore, the average number of searching architectures to achieve final architecture for the NASES procedure was <100, the number of searching architectures was reduced considerably compared with past NAS approaches. To determine the performance of the architecture, we evaluated the child network by using the final architecture network on the CIFAR-10 test dataset.
Table 2 summarizes the performance of NASES and other NAS approaches by using the macro search algorithms. This final architecture is presented in Appendix C, Figure 4.
In Table 2, the approaches have been into three levels based on the number of parameters. The first block of Table 2 presents the performances of NAS approaches; the NASES final architecture that achieves 4.07 error rate on the testing set uses only 8 million parameters, which is comparable with other NAS approaches. For comparing more approaches and models, we added a number of filters to each layer of the final architecture. The second block of Table 2 represents the performance of NAS approaches when the number of parameters was approximately 20 million, and the NASES final architecture can be improved to 3.93 error rate by adding 100 to the number of filters of each layer. Finally, to evaluate the high parameter network, we added 150 to the number of filters of each layer. Notably, the NASES final architecture that achieved 3.71 error rate only used 28.4 million parameters, which was better than approximately 38 million parameters used by the ENAS (Pham et al., 2018) and NAS (Zoph & Le, 2017). NASES required approximately half GPU day to discover the final architecture. The beneficial-performance and effectiveness of NASES was impressive even when only the architecture-embedding searching and pre-training controller were applied without other NAS tricks such as sharing parameters (Pham et al., 2018).

heading: 5 CONCLUSION

text: NAS with reinforcement learning is a powerful and novel framework for the automatic discovering process of neural architectures. Here, we designed a novel NAS framework, and this approach alleviated the two problems of noncontinuous and high-dimensional search space of NAS with reinforcement learning. We named this NAS framework NASES, in which the controller can be searched on embedding space by using the architecture decoder and architecture simulator. We achieved favorable results for image classification on CIFAR-10; the NASES exhibited efficient performance and high effectiveness when the number of searching architecture was reduced to <100 architectures. We proposed a simple method to estimate the compression ratio of architecture-embedding.

heading: A THE CHILD MODEL DETAILS

text:
A.1 THE OPERATIONS.
The 12 operations were provided to the controller by using two hyperparameters, namely filter size and kernel type. The filter size represents the amount of neighbor information during the current layer processing, and kernel type represents the components of the neural network, including the convolution layer, depthwise-separable convolution layer, maximum pooling layer, and average pooling layer. The child network receives these continuous vectors, and we developed a rule that transforms into discrete vectors. For example, in the kernel type, the rule is that ≤ 7 assigns depthwise-separable convolution, >7 or ≤ 15 assigns convolution. For filter size, the rule is that ≤ 10 assigns to the 3 × 3 filter size. According to this principle, the operations available for the child network are convolution with filter sizes 3 × 3, 5 × 5, and 7 × 7; depthwise-separable convolution with filter sizes 3× 3, 5× 5, and 7× 7; maximum pooling with filter sizes 3× 3, 5× 5, and 7× 7; and average pooling with filter sizes 3 × 3, 5 × 5, and 7 × 7. In addition, each depthwise-separable convolution was applied twice (Barret Zoph & Le, 2018).
A.2 SKIP CONNECTION.
The skip connections are essential connections that occur from the early layers to the later layers through addition or straight up concatenation. The reasoning behind this skip connection is that they exhibit an uninterrupted gradient flow from the first layer to the last layer, which tackles the vanishing gradient problem. In this study, the hyperparameter of the connection coefficient assigned the early layer to connect to last multiple layers with closer connection coefficient; these layers were concatenated in the channel dimension at the end of the layer. Therefore, we created another rule, and the layers are a connection when the connection coefficient is close to three; for example, the layer one and two are a connection when their connection coefficient is between two and five.
A.3 THE ORDER OF THE BLOCKS IN EACH LAYER.
Performance was affected by the order of blocks in each layer. We applied the order of ReLu-convbatchnorm (Ioffe & Szegedy, 2015). Moreover, the kernel size of 1 × 1 convolution filters can be applied to change the dimensionality in the filter space. We applied the order of ReLu-convbatchnorm to 1 × 1 kernel size convolution layers before the convolution layer, except for the first layer.
A.4 GLOBAL AVERAGE POOLING.
We employed a trick into the NASES of the global average pooling (Min Lin & Yan, 2013) which is an operation that calculates the average output of each feature map in the final convolution layer for reducing the number of parameters from the full connection layer.

heading: C FINAL ARCHITECTURE

heading: B DOUBLE-AXIS PLOT OF THE TESTING LOSS AND EMBEDDING SIZE


summaries: In this paper, the authors propose the Homotopy Training Algorithm (HTA) for neural network optimization problems. They claim that HTA starts with several simplified problems and tracks the solution to the original problem via a continuous homotopy path. They give the theoretical analysis and conduct experiments on the synthetic data and the CIFAR-10 dataset.

summaries: The work proposes to learn neural networks using homotopy-based continuation method. The method divides the parameter space into two groups (extendable to multiple groups) and introduces a homotopy function which includes the original optimization problem as an extreme case.  By varying the homotopy parameter, one can construct a continuous path from a supposedly easier to solve optimization problem to the problem of interest. The authors prove convergence in the non-convex case, the existence of solution path in the convex case and demonstrate the effectiveness of the proposed method on synthetic and real datasets.

abstractText: We present a Homotopy Training Algorithm (HTA) to solve optimization problems arising from neural networks. The HTA starts with several decoupled systems with low dimensional structure and tracks the solution to the high dimensional coupled system. The decoupled systems are easy to solve due to the low dimensionality but can be connected to the original system via a continuous homotopy path guided by the HTA. We have proved the convergence of HTA for the non-convex case and existence of the homotopy solution path for the convex case. The HTA has provided a better accuracy on several examples including VGG models on CIFAR-10. Moreover, the HTA would be combined with the dropout technique to provide an alternative way to train the neural networks.

authors:
D. Bates
J. Hauenstein
A. Sommese
C. Wampler

title: Adaptive multiprecision path tracking

venue: SIAM Journal on Numerical Analysis,

authors:
D. Bates
J. Hauenstein
A. Sommese

title: Efficient path tracking methods

venue: Numerical Algorithms,

title: Large-scale machine learning with stochastic gradient descent

venue: In Proceedings of COMPSTAT’2010,

authors:
L. Bottou
F. Curtis
J. Nocedal

title: Optimization methods for large-scale machine learning

venue: Siam Review,

authors:
Z. Cang
G. Wei

title: Integration of element specific persistent homology and machine learning for protein-ligand binding affinity prediction. International journal for numerical methods in biomedical engineering, 34(2):e2914, 2018

authors:
Z. Cang
L. Mu
G. Wei

title: Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening

venue: PLoS computational biology,

authors:
S. Chow
T. Yang
H. Zhou

title: Global optimizations by intermittent diffusion. In Chaos, CNN, Memristors and Beyond: A Festschrift for Leon Chua With DVD-ROM, composed by Eleonora

venue: World Scientific,

authors:
I. Goodfellow
Y. Bengio
A. Courville

title: Deep Learning

authors:
N. Hammerla
S. Halloran
T. Ploetz

title: Deep, convolutional, and recurrent models for human activity recognition using wearables

venue: IJCAI 2016,

authors:
W. Hao
J. Hauenstein
B. Hu
Y. Liu
A. Sommese
Y.-T. Zhang

title: Bifurcation for a free boundary problem modeling the growth of a tumor with a necrotic core

venue: Nonlinear Analysis: Real World Applications,

title: Sur les fonctions convexes et les inégalités entre les valeurs moyennes

venue: Acta mathematica,

authors:
D. Kingma
J. Ba

venue: arXiv preprint arXiv:1412.6980,

authors:
K. Konda
A. Königs
H. Schulz
D. Schulz

title: Real time interaction with mobile robots using hand gestures

venue: In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction,

authors:
F. Maire
L. Mejias
A. Hodgson

title: A convolutional neural network for automatic analysis of aerial imagery

venue: In Digital lmage Computing: Techniques and Applications (DlCTA),

authors:
A. Morgan
A. Sommese

title: Computing all solutions to polynomial systems using homotopy continuation

venue: Applied Mathematics and Computation,

authors:
J. Park
I. Sandberg

title: Universal approximation using radial-basis-function networks

venue: Neural computation,

authors:
K. Simonyan
A. Zisserman

title: Very deep convolutional networks for large-scale image recognition

venue: Computer Science,

authors:
N. Srivastava
G. Hinton
A. Krizhevsky
I. Sutskever
R. Salakhutdinov

title: Dropout: a simple way to prevent neural networks from overfitting

venue: The Journal of Machine Learning Research,

authors:
C. Wampler
A. Sommese

title: The Numerical solution of systems of polynomials arising in engineering and science

heading: 1 INTRODUCTION

text:
Deep learning methods, as the rising star among all machine learning methods in recent years, have already had great success in many applications. Many advancements Cang et al. (2018); Cang & Wei (2018); Yin et al. (2018; 2016) on deep learning have been made in the last few years. However, as the size of new state-of-the-art models keep growing larger and larger, they rely more and more heavily on efficient algorithms for training and making inferences from such models. This clearly puts strong limitations on the application scenarios of nerual netowrks on roboticsKonda et al. (2012), auto-pilot automobiles Hammerla et al. (2016), and aerial systems Maire et al. (2014). At present, there are two big challenges in fundamentally understanding deep neural networks:
• How to efficiently solve the highly nonlinear and non-convex optimization problems that arise during the training of a deep learning model?
• How to design a deep neural network structure for specific problems?
In order to solve these challenges, in this paper, we will present a new training algorithm based on homotopy continuation method Bates et al. (2011; 2008); Morgan & Sommese (1987) which has been successfully used to study nonlinear problems such as nonlinear differential equations and optimizations Hao et al. (2012); Hao & Harlim. In order to tackle the nonlinear optimization problem in neural networks, the homotopy training algorithm is designed and shows efficiency and feasibility for both convolutional and fully connected neural networks with complex structures. The idea of the HTA is to start with a simplified optimization problem and construct a continuous path to the original problem. Specifically, let’s consider the following general optimization problem
min θ f(θ) where θ ∈ Rn. (1)
We then divide θ in two groups, namely, θ = θ1∪ θ2, where θ1 ∈ Rn1 and θ2 ∈ Rn2 (n1 +n2 = n). (Although that it can be easily extended to any m decompositions, we use m = 2 as an illustration of the HTA idea.) Then we first solve the following decoupled simplified optimization problems
min θ1 f(θ1, θ̃2) and min θ2 f(θ̃1, θ2) (2)
where θ̃ = θ̃1 ∪ θ̃2 is a random chosen point in Rn for instance we choose θ̃ = 0 in the dropout Srivastava et al. (2014). Second we define the following homotopy function
H(Θ, L) = f(θ) + L‖θ1 − θ̃1‖2 + f(η) + L‖η2 − θ̃2‖2 + 1
L ‖θ − η‖2, (3)
where Θ = (θ, η) and L ∈ (0,∞) is the homotopy parameter. When L → ∞, we have two decoupled optimization problems which we have already solved in the first step; when L → 0, we recover the original optimization by forcing θ = η. By gradually increasing parameter L, we are able to construct a continuous path Θ(L)between two decoupled low-dimensional optimization problems and the original optimization problem.
Remark The HTA can be reviewed in point view of nonlinear equations by considering the necessary condition of the optimization problem (1):
H(Θ, L) = (∇θf(θ) η − θ ) 1 L + L  ∇θ2f(θ) θ1 − θ̃1 ∇η1f(η) η2 − θ̃2  . When L is large, θ2 and η1 can be solved by ∇θ2f(θ) = 0 and ∇η1f(η) = 0 respectively. When L is small, we have the solution of ∇θf(θ) = 0 by the diagonal homotopy θ − η = 0. This technique has been widely used in solving systems of nonlinear equations especially for polynomial systems Bates et al. (2008); Wampler & Sommese (2005). Since the θ̃ is randomly chosen, the universal approximation theory Park & Sandberg (1991) can guarantee that the random point θ̃ in Rn is always connected to the solution of ∇θf(θ) = 0. In another word, the solution path Θ(L) always exists or the probability that Θ(L) is not connected is zero Wampler & Sommese (2005).

heading: 2 APPLICATIONS OF HTA ON NEURAL NETWORK

heading: 2.1 CONVOLUTIONAL NEURAL NETWORKS

text:
In this subsection, we will specify the variable θ and the objective function f(θ) of (3) in the convolutional neural network (CNN) setup. Definition 2.1. Suppose we have two tensors, Al×m×n = (ai,j,k)1≤i≤l,1≤j≤m,1≤k≤n, Bc×l×m′×n′ = (bh,i,j,k)1≤h≤c,1≤i≤l,1≤j≤m′,1≤k≤n′ , where c,m, n,m′, n′, l ∈ N and m′ ≤ m,n′ ≤ n. The convolution of two tensors is defined by the following:
A ∗Bh,j,k = l∑
i′=1 m′∑ j′=1 n′∑ k′=1 ai′,j′+j−1,k′+k−1 · bh,i′,j′,k′ , (4)
where B is called the convolutional kernel.
Then with the input x and the output y, the CNN is written as
y(x; θ) = σ(σ(· · ·σ(σ(x ∗W1 + b1) ∗W2 + b2) · · ·+ bn−1) ∗Wn + bn) (5) where Wi ∈ Rci,li,mi,ni and σ is the activation function. Then we have θ = {Wi, bi}ni=1 which can be easily divided into two subgroups θj = {Wij , bi}ni=1 where Wij ∈ R ci 2 ,li,mi,ni . The objective function f(θ) in (3) is defined by the cross entropy loss function Goodfellow et al. (2016) and defined as L(x, y) = −x(y) + log( ∑ j e x[j]). Due to the random algorithms in machine learning Bottou (2010); Kingma & Ba (2014), the homotopy function in (3) can be rewritten as H(Θ; ξ, L) where ξ is a random variable due to random algorithms.

heading: 2.2 FULLY CONNECTED NEURAL NETWORKS

text:
Similar to the CNN, a fully connected neural network with n hidden layers is written as
y(x, θ) = σ(Wnσ(Wn−1σ(W2σ(W1x+ b1) + b2) · · ·+ bn−1) + bn), (6) where Wi ∈ Rn×di , βi ∈ Rdi , n is the dimension of the input x and di is the width of the i-th layer. Then we have θ = {Wi, bi}ni=1 which can be easily divided into two subgroups θ1 = {Wi, bi} n/2 i=1 and θ2 = {Wi, bi}ni=n/2+1. Or we can use the dropout technique to randomly split θ as θ1 and θ2. Similarly, the homotopy function in (3) can be also rewritten as H(Θ; ξ, L) due to the random algorithms Bottou (2010); Kingma & Ba (2014).

heading: 3 THEORETICAL ANALYSIS

text:
In this section, we will provide a theoretical analysis of the HTA by using stochastic gradient scheme to solve (3) for each L. We redefine our stochastic gradient scheme for the homotopy process as
θk+1 = θk − γkG(θk; ξk, Lk), (7)
where G(θ; ξ, L) := ∇θF (θ; ξ, L), γk is the learning rate, L0 = M , Lk ↗ 0 and M is a given large number. Then we consider the expectation of the homotopy function as Eξ(H(θ; ξ, L)), for simplicity, and denote Eξ[H(θ; ξ, L)] as g(θ;L).

heading: 3.1 CONVERGENCE OF THE HTA FOR ANY GIVEN L IN THE NON-CONVEX CASE

text:
Theorem 3.1. (Nonconvex Convergence) If {θk} is contained in a bounded open set, supposing that (7) is run with a step size sequence satisfying
∞∑ k=1 γk =∞ and ∞∑ k=1 γ2k <∞, (8)
then we have
E[ 1
Ak K∑ k=1 γk‖∇g(θk;L)‖22]→ 0 as K →∞ with Ak := K∑ k=1 γk. (9)
Proof. First, we prove the Lipschitz-continuous objective gradients condition Bottou et al. (2018), which means that g(θ;L) is C1 and ∇g(θ;L) is Lipschitz continuous:
• g(θ;L) is C1. Since y(xξ; θ) ∈ C1 and L(·, y) is C1, we have that H(θ; ξ, L) ∈ C1 or that ∇θH(θ; ξ, L) is continuous. Considering
∇θg(θ;L) = ∇θEξ(H(θ; ξ, L)) = Eξ(∇θH(θ; ξ, L)), (10)
we have that∇θg(θ, L) is continuous or that g(θ;L) ∈ C1.
• ∇g(θ;L) is Lipschitz continuous. Since ∇θg(θ;L) = Eξ(∇θH(θ; ξ, L)), we only need to prove Eξ(∇θL(y(xξ; θ), yξ)) is Lipschitz continuous. In fact
∇θL(y(xξ; θ), yξ) = ∇xL(y(xξ; θ), yξ) · ∇θy(xξ; θ) (11)
we will show that ∇xL(y(xξ; θ), yξ) and ∇θy(xξ; θ) are bounded and Lipschitz continuous. Since that both σ(x) = 11+e−x and σ
′(x) = σ(x)(1−σ(x)) are bounded and Lipschitz continuous, and that {θk} is in an open set (assumption of Theorem 3.1), ∇θy(xξ; θ) is bounded and Lipschitz continuous. (xξ is bounded because the size of our dataset is finite.) By differentiating L(x, y), we have
∇xL(x, y) = (−δ1y + ex1∑ j e xj , · · · ,−δny + exn∑ j e xj ), (12)
where δ is the Kronecker delta. since
∂
∂xk exi∑ j e xj =  exi ∑ j e xj−(exi )2 ( ∑ j e xj )2 k = i −exiexk ( ∑
j e xj )2
k 6= i, (13)
which implies that ∣∣ ∂ ∂xk exi∑ j e xj ∣∣ ≤ 2, we have ∇xL(·, y) is Lipschitz continuous and bounded. Therefore ∇xL(y(xξ; θ), yξ) is Lipschitz continuous and bounded. Therefore we have that ∇g(θ;L) is Lipschitz continuous.
Second, we prove the first and second moment limits condition Bottou et al. (2018):
a. According to our theorem assumption, {θk} is contained in an open set which is bounded. Since that g is continuous, g is bounded;
b. Since that G(θk; ξk, L) = ∇θF (θk; ξk, L) is continuous, we have
Eξk [G(θk; ξk, L)] = ∇θEξk [H(θk; ξk, L)] = ∇θg(θk;L). (14)
Therefore
∇g(θk;L)TEξk [G(θk; ξk, L)] = ∇g(θk;L)T ·∇g(θk;L) = ‖∇g(θk;L)‖22 ≥ u‖∇g(θk;L)‖22 (15) for 0 < u ≤ 1. On the other hand, we have
‖Eξk [G(θk; ξk, L)]‖2 = ‖∇g(θk;L)‖2 ≤ uG‖∇g(θk;L)‖22 (16)
for uG ≥ 1.
c. Since that∇H(θk; ξk, L) is bounded for a given L, we have Eξk [‖∇H(θk; ξk, L)‖22] is also bounded. Thus
Vξk [G(θk; ξk, L)] := Eξk [‖G(θk; ξk, L)‖22]−‖Eξk [G(θk; ξk, L)]‖22 ≤ Eξk [‖G(θk; ξk, L)‖22], (17) which implies that Vξk [G(θk; ξk, L)] is bounded.
We have checked assumptions 4.1 and 4.3 in Bottou et al. (2018). By theorem 4.10 in Bottou et al. (2018), with the diminishing stepsize, namely,
∞∑ k=1 γk =∞ and ∞∑ k=1 γ2k <∞, (18)
the following convergence is obtained
E[ 1
Ak K∑ k=1 γk‖∇g(θk;L)‖22]→ 0 as K →∞. (19)
3.2 EXISTENCE OF SOLUTION PATH θ(L) FOR THE CONVEX CASE
Second we explore the existence of solution path θ(L) when t varies fromM to 0 for the convex case theoretically. This is quite difficult for the non-convex case although we may find it numerically. However, theoretical analysis of the convex case would also give us some guidance on how the HTA obtains an optimal solution for L = 0.
Theorem 3.2. Assume that g(·, ·) is convex and differentiable and that ‖G(θ; ξ, L)‖ ≤ M0. Then for stochastic gradient scheme (7), we have
lim n→∞
E[g(θ̄n, L̄n)] = g(θ1∗, 0), (20)
where θ̄n = ∑n
k=0 γkθk∑n k=0 γk
and L̄n = ∑n
k=0 γkLk∑n k=0 γk .
Proof.
E[‖θk+1 − θ Lk+1 ∗ ‖2] = E[‖θk − γkG(θk; ξk, Lk)− θLk∗ ‖2]
−2E[〈θk − γkG(θk; ξk, Lk)− θLk∗ , θ Lk+1 ∗ − θLk∗ 〉] +E[‖θLk+1∗ − θLk∗ ‖2]. (21)
By defining
Ak = −2E[〈θk − γkG(θk; ξk, Lk)− θLk∗ , θ Lk+1 ∗ − θLk∗ 〉] + E[‖θ Lk+1 ∗ − θLk∗ ‖2], (22)
and assuming that ∑m k=0Ak ≤ A <∞, we obtain
E[‖θk+1 − θ Lk+1 ∗ ‖2] = E[‖θk − γkG(θk; ξk, Lk)− θLk∗ ‖2] +Ak
= E[‖θk − θLk∗ ‖2]− 2γkE[〈G(θk; ξk, Lk), θk − θLk∗ 〉] + γ2kE[‖G(θk; ξk, Lk)‖2] +Ak ≤ E[‖θk − θLk∗ ‖2]− 2γkE[〈G(θk; ξk, Lk), θk − θLk∗ 〉] + γ2kM2 +Ak
Since that
E[〈G(θk; ξk, Lk), θk − θLk∗ 〉] = Eξ0,··· ,ξk−1 [Eξk [〈G(θk; ξk, Lk), θk − θLk∗ 〉|ξ0, · · · , ξk−1]] = Eξ0,··· ,ξk−1 [〈∇g(θk;Lk), θk − θLk∗ 〉|ξ0, · · · , ξk−1] = E[〈∇g(θk;Lk), θk − θLk∗ 〉], (23)
we have
E[‖θk+1 − θ Lk+1 ∗ ‖2] ≤ E[‖θk − θLk∗ ‖2]− 2γkE[〈∇g(θk;Lk), θk − θLk∗ 〉] + γ2kM2 +Ak. (24)
Due to the convexity of g(·, Lk), namely, 〈∇g(θk, Lk), θk − θLk∗ 〉 ≥ g(θk;Lk)− g(θLk∗ ;Lk), (25)
we conclude that
E[‖θk+1 − θ Lk+1 ∗ ‖2] ≤ E[‖θk − θLk∗ ‖2]− 2γkE[g(θk;Lk)− g(θLk∗ , Lk)] + γ2kM2 +Ak, (26)
or
2γkE[g(θk;Lk)− g(θLk∗ ;Lk)] ≤ −E[‖θk+1 − θ Lk+1 ∗ ‖2 − ‖θk − θLk∗ ‖2] + γ2kM2 +Ak.
By summing up k from 0 to n,
2 n∑ k=0 γkE[g(θk;Lk)− g(θLk∗ ;Lk)] ≤ −E[‖θn+1 − θ Ln+1 ∗ ‖2 − ‖θ0 − θ0∗‖2] +M2 n∑ k=0 γ2k + n∑ k=0 Ak
≤ D2 +M2 n∑ k=0 γ2k + n∑ k=0 Ak, (27)
where D = ‖θ0 − θ0∗‖. Dividing 2 ∑n k=0 γk on both sides, we have 1∑n k=0 γk n∑ k=0 γkE[g(θk;Lk)− g(θLk∗ ;Lk)] ≤ D2 +M2 ∑n k=0 γ 2 k + ∑n k=0Ak 2 ∑n k=0 γk ≤ D2 +M2 ∑n k=0 γ 2 k +A 2 ∑n k=0 γk .
According to the convexity of g(·; ·) and Jensen’s inequality Jensen (1906),
1∑n k=0 γk n∑ k=0 γkE[g(θk;Lk)] ≤ E[g(θ̄n; L̄n)], (28)
where θ̄n = ∑n
k=0 γkθk∑n k=0 γk
and L̄n = ∑n
k=0 γkLk∑n k=0 γk .
Then we have E[g(θ̄n; L̄n)]− ∑n k=0 γkg(θ Lk ∗ ;Lk)∑n
k=0 γk ≤ D2 +M2
∑n k=0 γ 2 k +A
2 ∑n k=0 γk
, (29)
We choose γk such that ∑n k=0 γk = ∞ and ∑n k=0 γ 2 k < ∞, for example, γk = 1k . Taking n to infinite, we have
0 ≤ lim n→∞ E[g(θ̄n, t̄n)]− lim n→∞
∑n k=0 γkg(θ
Lk ∗ , Lk)∑n
k=0 γk ≤ 0. (30)
Because Lk ↗ 0, g(·, ·) is continuous and ∑n k=0 γk =∞, then
lim n→∞
∑n k=0 γkg(θ
Lk ∗ , Lk)∑n
k=0 γk = g(θ0∗, 0), (31)
which implies that lim n→∞ E[g(θ̄n, L̄n)] = g(θ0∗, 0). (32)

heading: 4 NUMERICAL EXPERIMENTS

heading: 4.1 A NON-CONVEX EXAMPLE

text:
We first use the following multi-dimensional highly non-convex example Chow et al. (2013) to illustrate the feasibility and efficiency of the HTA.
min f(x) = (π/n) { k sin2(πy1) +
n−1∑ i=1
(yi −A)2[1 + k sin2(πyi+1)] + (yn −A)2 } , (33)
where
x = (x1, · · · , xn) ∈ Rn, yi = 1 + (xi − 1)/4, i = 1, · · · , n and k = 10, A = 1. (34) The function has roughly 5n local minima in the region {|xi| < 10} and a unique global minimum located at xi = 1, i = 1, · · · , n. (See Figure 4.1 for an illustration of n = 2.) The idea of the homotopy setup of this problem is splitting the n dimensional optimization problem into two n/2 dimensional problems which can be optimized separately and are tracked to an optimal of solution of f(x) with respect to the homotopy parameter L. Then the objective function of the homotopy method is written as follows:
F (x, y;L) = f(x) + L‖(x1, · · · , xn2 )− (x 0 1, · · · , x0n2 )‖ 2 + g(y) + L‖(yn 2 +1 , · · · , yn)− (y0n 2 +1 , · · · , y0n)‖2
+ 1
L ‖x− y‖2,
(35) where x = (x1, · · · , xn) ∈ Rn, y = (y1, · · · , yn) ∈ Rn, (36) (x01, · · · , x0n 2 ) ∈ Rn2 , (y0n 2 +1 , · · · , y0n) ∈ R n 2 are fixed and chosen based on the initial guess. When L is large, F is equivalent to optimize
min xn
2 +1,··· ,xn g(x01, · · · , x0n2 , xn2 +1, · · · , xn) + miny1,··· ,yn 2 g(y1, · · · , yn2 , y 0 n 2 +1 , · · · , y0n), (37)
which can be solved separately. When L becomes small, the last term of (35) forces x and y to be equal. Then F (x, y;L) is equivalent to optimizing f(x) which is the original n-dimensional optimization problem. Therefore F (x, y;L) connects two optimization problems by gradually decreasing penalty factor L and builds a continuous path between the decoupled systems and the original optimization function f(x).
We compared the traditional optimization method (the quasi-Newton method) and the quasi-Newton with the homotopy setup. For n = 4, we chose the same initial value [5, 4, 5, 5] and set L from 0.9 to 0.1 for the quasi-newton with the homotopy setup. It takes 8 steps and 14 steps for the quasi-newton method and the quasi-newton with the homotopy setup to converge, respectively. However, the quasi-newton method does not find the global minimum while the HTA does. We also compared two algorithms with different initial values and count the times when converging to global minimums. Table 1 shows the results of n = 2, n = 4, n = 6 and n = 8.

heading: 4.2 APPLICATIONS ON VGG

text: Secondly, we applied the HTA to both CNN and fully connected neural networks on the CIFAR10 dataset (https://www.cs.toronto.edu/˜kriz/cifar.html) with the training and validation datasets from the CIFAR-10 website. In terms of the pre-processing step: for training dataset, we used random horizontal flip, random crop and normalization; for the validation dataset, we just used the same normalization. We implemented the HTA with stochastic gradient descent (SGD) method on VGG11, VGG13, VGG16 and VGG19 as our base models with batch normalization Simonyan & Zisserman (2014). In order to compare the SGD with the HTA method, we set all the regular parameters (e.g. learning rate, batch size, momentum, epochs, weight decay and etc) to be exactly the same, varied the homotopy parameter L from L = 0.01 to L = 0.005 with a stepsize ∆L = 0.001. For each L, we run 4 epochs with the SGD method. The code for this method can be found at https://github.com/Bill-research/homotopy which has been run on the DGX-1 (p100 version) with 8 p100 GPUs and Tesla structures. Figure 2 shows the comparison of validation loss between the HTA with fully connected neural networks and traditional method on the VGG13 model. The HTA has a less error rate (5.14%) than the traditional method (6.35%) by improving 11.68%. All the results for different models are shown in Table 2. It is clearly seen that accuracies of HTA for different models are better than that of the traditional method. For example, vgg11 with HTA gets 7.02% error rate while traditional method only gets 7.83%; The vgg16 with HTA has an error rate of 5.88% while traditional method only has 6.35%. We run the numerical experiments for 5 times for each model and the results are shown in Table 3. We also applied HTA to the only CNN part of two VGG models: vgg11 and vgg13. Figure 3 shows the validation accuracies of two models which has been improved by the HTA (91.61% (vgg11) and 93.09% (vgg13)) comparing with the traditional method (91.01% (vgg11) and 92.29% (vgg13)).

heading: 5 CONCLUSION

text: In this paper, we developed a homotopy training algorithm for solving the optimization problem arising from neural networks. This algorithm starts from decoupled low dimensional systems and gradually transforms to the coupled original system with complex structure. Then the complex neural networks can be trained by the HTA with a better accuracy. The convergence of the HTA for any given L is proved for the non-convex optimization. Then existence of solution path θ(L) is demonstrated theoretically for the convex case although that it exists numerically in the non-convex case. Several numerical examples have used to demonstrate the efficiency and feasibility of HTA. The application of HTA to VGG models on CIFAR-10 provides a better accuracy than the traditional method. Moreover, the HTA method would provide a new way to couple with the dropout techniques to improve the training accuracy which we will explore in the future.