In computer architecture, near-data processing (NDP) refers to augmenting the memory or the storage with processing power so that it can process the data stored therein. By offloading the computational burden of CPU and saving the need for transferring raw data in its entirety, NDP exhibits a great potential for acceleration and power reduction. Despite this potential, specific research activities on NDP have witnessed only limited success until recently, often owing to performance mismatches between logic and memory process technologies that put a limit on the processing capability of memory. Recently, there have been two major changes in the game, igniting the resurgence of NDP with renewed interest. The first is the success of machine learning (ML), which often demands a great deal of computation for training, requiring frequent transfers of big data. The second is the advent of NAND flash-based solid-state drives (SSDs) containing multicore processors that can accommodate extra computation for data processing. Sparked by these application needs and technological support, we evaluate the potential of NDP for ML using a new SSD platform that allows us to simulate in-storage processing (ISP) of ML workloads. Our platform (named ISP-ML) is a full-fledged simulator of a realistic multi-channel SSD that can execute various ML algorithms using the data stored in the SSD. For thorough performance analysis and in-depth comparison with alternatives, we focus on a specific algorithm: stochastic gradient decent (SGD), which is the de facto standard for training differentiable learning machines including deep neural networks. We implement and compare three variants of SGD (synchronous, Downpour, and elastic averaging) using ISP-ML, exploiting the multiple NAND channels for parallelizing SGD. In addition, we compare the performance of ISP and that of conventional in-host processing, revealing the advantages of ISP. Based on the advantages and limitations identified through our experiments, we further discuss directions for future research on ISP for accelerating ML.
In computer architecture, near-data processing (NDP) refers to augmenting the memory or the storage with processing power so that it can process the data stored therein. By offloading the computational burden of CPU and saving the need for transferring raw data in its entirety, NDP exhibits a great potential for acceleration and power reduction. Despite this potential, specific research activities on NDP have witnessed only limited success until recently, often owing to performance mismatches between logic and memory process technologies that put a limit on the processing capability of memory. Recently, there have been two major changes in the game, igniting the resurgence of NDP with renewed interest. The first is the success of machine learning (ML), which often demands a great deal of computation for training, requiring frequent transfers of big data. The second is the advent of NAND flash-based solid-state drives (SSDs) containing multicore processors that can accommodate extra computation for data processing. Sparked by these application needs and technological support, we evaluate the potential of NDP for ML using a new SSD platform that allows us to simulate in-storage processing (ISP) of ML workloads. Our platform (named ISP-ML) is a full-fledged simulator of a realistic multi-channel SSD that can execute various ML algorithms using the data stored in the SSD. For thorough performance analysis and in-depth comparison with alternatives, we focus on a specific algorithm: stochastic gradient decent (SGD), which is the de facto standard for training differentiable learning machines including deep neural networks. We implement and compare three variants of SGD (synchronous, Downpour, and elastic averaging) using ISP-ML, exploiting the multiple NAND channels for parallelizing SGD. In addition, we compare the performance of ISP and that of conventional in-host processing, revealing the advantages of ISP. Based on the advantages and limitations identified through our experiments, we further discuss directions for future research on ISP for accelerating ML.
1 INTRODUCTION
Recent successes in deep learning can be accredited to the availability of big data that has made the training of large deep neural networks possible. In the conventional memory hierarchy, the training data stored at the low level (e.g., hard disks) need to be moved upward all the way to the CPU registers. As larger and larger data are being used for training large-scale models such as deep networks (LeCun et al., 2015), the overhead incurred by the data movement in the hierarchy becomes more salient, critically affecting the overall computational efficiency and power consumption.
∗To whom correspondence should be addressed.
The idea of near-data processing (NDP) (Balasubramonian et al., 2014) is to equip the memory or storage with intelligence (i.e., processors) and let it process the data stored therein firsthand. A successful NDP implementation would reduce the data transfers and power consumption, not to mention offloading the computational burden of CPUs. The types of NDP realizations include processing in memory (PIM) (Gokhale et al., 1995) and in-storage processing (ISP) (Acharya et al., 1998; Kim et al., 2016c; Lee et al., 2016; Choi & Kee, 2015). Despite the potential of NDP, it has not been considered significantly for commercial systems. For PIM, there has been a wide performance gap between the separate processes to manufacture logic and memory chips. For ISP, commercial hard disk drives (HDDs), the mainstream storage devices for a long time, normally have limited processing capabilities due to tight selling prices.
Recently, we have seen a resurrection of NDP with renewed interest, which has been triggered by two major factors, one in the application side and the other in the technology side: First, computingand data-intensive deep learning is rapidly becoming the method of choice for various machine learning tasks. To train deep neural networks, a large volume of data is typically needed to ensure performance. Although GPUs and multicore CPUs often provide an effective means for massive computation required by deep learning, it remains inevitable to store big training data in the storage and then transfer them to the CPU/GPU level for computation. Second, NAND flash-based solidstate drives (SSDs) are becoming popular, gradually replacing HDDs in various computing sectors. To interface SSDs with the host seamlessly replacing HDDs, SSDs require various software running inside, e.g., for address translation and garbage collection (Kim et al., 2002; Gupta et al., 2009). To suit such needs, SSDs are often equipped with multicore processors, which provide far more processing capabilities than those in HDDs. Usually, there exists a plenty of idle time in the processors in SSDs that can be exploited for other purposes than SSD housekeeping (Kim et al., 2010; 2016b).
Motivated by these changes and opportunities, we propose a new SSD platform that allows us to simulate in-storage processing (ISP) of machine learning workloads and evaluate the potential of NDP for machine learning in ISP. Our platform named ISP-ML is a full-fledged system-level simulator of a realistic multi-channel SSD that can execute various machine learning algorithms using the data stored in the SSD. For thorough performance analysis and in-depth comparison with alternatives, we focus on describing our implementation of a specific algorithm in this paper: the stochastic gradient decent (SGD) algorithm, which is the de facto standard for training differentiable learning machines including deep neural networks. Specifically, we implement three types of parallel SGD: synchronous SGD (Zinkevich et al., 2010), Downpour SGD (Dean et al., 2012), and elastic averaging SGD (EASGD) (Zhang et al., 2015). We compare the performance of these implementations of parallel SGD using a 10 times amplified version of MNIST (LeCun et al., 1998). Furthermore, to evaluate the effectiveness of ISP-based optimization by SGD, we compare the performance of ISP-based and the conventional in-host processing (IHP)-based optimization.
To the best of the authors’ knowledge, this work is one of the first attempts to apply NDP to a multichannel SSD for accelerating SGD-based optimization for training differentiable learning machines. Our specific contributions can be stated as follows:
• We created a full-fledged ISP-supporting SSD platform called ISP-ML, which required multiyear team efforts. ISP-ML is versatile and can simulate not only storage-related functionalities of a multi-channel SSD but also NDP-related functionalities in realistic manner. ISP-ML can execute various machine learning algorithms using the data stored in the SSD while supporting the simulation of multi-channel NAND flash SSDs to exploit data-level parallelism.
• We thoroughly tested the effectiveness of our platform by implementing and comparing multiple versions of parallel SGD, which is widely used for training various machine learning algorithms including deep learning. We also devised a methodology that can carefully and fairly compare the performance of IHP-based and ISP-based optimization.
• We identified intriguing future research opportunities in terms of exploiting the parallelism provided by the multiple NAND channels inside SSDs. As in high-performance computing, there exist multiple “nodes” (i.e., NAND channel controllers) for sharing workloads, but the communication cost is negligible (due to negligible-latency on-chip communication) unlike the conventional parallel computing. Using our platform, we envision new designs of parallel optimization and training algorithms that can exploit this characteristic, producing enhanced results.
2 BACKGROUND AND RELATED WORK
2.1 MACHINE LEARNING AS AN OPTIMIZATION PROBLEM
Various types of machine learning algorithms exist (Murphy, 2012; Goodfellow et al., 2016), and their core concept can often be explained using the following equations:
F (D, θ) = L(D, θ) + r(θ) (1) θt+1 = θt + ∆θ(D) (2)
∆θ(D) = −η∇F (D, θ) (3)
whereD and θ denote the input data and model parameters, respectively, and a loss function L(D, θ) reflects the difference between the optimal and current hypotheses. A regularizer to handle overfitting is denoted by r(θ), and the objective function F (D, θ) is the sum of the loss and regularizer terms. The main purpose of supervised machine learning can then be formulated as finding optimal θ that minimizes F (D, θ). Gradient descent is a first-order iterative optimization algorithm to find the minimum value of F (D, θ) by updating θ on every iteration t to the direction of negative gradient of F (D, θ), where η is the learning rate. SGD computes the gradient of the parameters and updates them using a single training sample per iteration. Minibatch (stochastic) gradient decent uses multiple (but far less than the whole) samples per iteration. As will be explained shortly, we employ minibatch SGD in our framework, setting the size of a minibatch to the number of training samples in a NAND flash page, which is named ‘page-minibatch’ (see Figure 2).
2.2 PARALLEL AND DISTRIBUTED SGD
Zinkevich et al. (2010) proposed an algorithm that implements parallel SGD in a distributed computing setup. This algorithm often suffers from excessive latency caused by the need for synchronization of all slave nodes. To overcome this weakness, Recht et al. (2011) proposed the lock-free Hogwild! algorithm that can update parameters asynchronously. Hogwild! is normally implemented in a single machine with a multicore processor. Dean et al. (2012) proposed the Downpour SGD for a distributed computing systems by extending the Hodwild! algorithm. While they successfully implemented asynchronous SGD in a distributed computing system, it often fails to overcome communication bottlenecks and shows inefficient bandwidth usage, caused by substantial data movements between computing nodes. Recently proposed EASGD (Zhang et al., 2015) attempted to minimize communication overhead by reducing the frequency of parameter updates. Many EASGD-based approaches reported its effectiveness in distributed environments.
2.3 FUNDAMENTALS OF SOLID-STATE DRIVES (SSDS)
SSDs have emerged as a type of next-generation storage device using NAND flash memory (Kim et al., 2010). As shown in the right image in Figure 1(a), a typical SSD consists of an SSD controller, a DRAM buffer, and a NAND flash array. The SSD controller is typically composed of an embedded processor, a cache controller, and channel controllers. The DRAM component, controlled by the cache controller, plays the role of a cache buffer when the NAND flash array is read or written. The NAND flash array contains multiple NAND chips that can be accessed simultaneously thanks to multi-channel configurations and per-channel controllers. Every channel controller is managed by the software called flash translation layer (FTL), which executes wear-leveling and garbage collection to improve the performance and durability of the NAND flash array.
2.4 PREVIOUS WORK ON NEAR-DATA PROCESSING
Most of the previous work on ISP focused on popular but inherently simple algorithms, such as scan, join, and query operations (Kim et al., 2016c). Lee et al. (2016) proposed to run the merge operation (frequently used by external sort operation in Hadoop) inside an SSD to reduce IO transfers and read/write operations, also extending the lifetime of the NAND flash inside the SSD. Choi & Kee (2015) implemented algorithms for linear regression, k-means, and string match in the flash memory controller (FMC) via reconfigurable stream processors. In addition, they implemented a MapReduce application inside the embedded processor and FMC of the SSD by using partitioning and pipelining methods that could improve performance and reduce power consumption. BlueDBM (Jun
et al., 2015) is an ISP system architecture for distributed computing systems with a flash memorybased embedded field programmable gate array (FPGA). The authors implemented nearest-neighbor search, graph traversal, and string search algorithms. No prior work ever implemented and evaluated SSD-based optimization of machine learning algorithms using SGD.
3 PROPOSED METHODOLOGY
Figure 1(a) shows the block diagram of a typical computing system, which is assumed to have an SSD as its storage device. Also shown in the figure is a magnified view of the SSD block diagram that shows the major components of an SSD and their interconnections. Starting from the baseline SSD depicted above, we can implement ISP functionalities by modifying the components marked with black boxes (i.e., ISP HW and ISP SW in the figure). Figure 1(b) shows the detailed schematic of our proposed ISP-ML platform that corresponds to the SSD block (with ISP components) shown in Figure 1(a).
In this section, we provide more details of our ISP-ML framework. In addition, we propose a performance comparison methodology that can compare the performance of ISP and the conventional IHP in a fair manner. As a specific example of the ML algorithms that can be implemented in ISP-ML, we utilize parallel SGD.
3.1 ISP-ML: ISP PLATFORM FOR MACHINE LEARNING ON SSDS
Our ISP-ML is a system-level simulator implemented in SystemC on the Synopsys Platform Architect environment (http://www.synopsys.com). ISP-ML can simulate hardware and software ISP components marked in Figure 1(b) simultaneously. This integrative functionality is crucial for design space exploration in SSD developments. Moreover, ISP-ML allows us to execute various machine learning algorithms described in high-level languages (C or C++) directly on ISP-ML only with minor modifications.
At the conception of this research, we could not find any publicly available SSD simulator that could be modified for implementing ISP functionalities. This motivated us to implement a new simulator. There exist multiple ways of realizing the idea of ISP in an SSD. The first option would be to use the embedded core inside the SSD controller (Figure 1(a)). This option does not require designing a new hardware logic and is also flexible, since the ISP capability is implemented by software. However, this option is not ideal for exploiting hardware acceleration and parallelization. The second option would be to design dedicated hardware logics (such as those boxes with black marks in Figure 1(a) and the entire Figure 1(b)) and integrate them into the SSD controller. Although significantly more efforts are needed for this option compared the first, we chose this second option due to its long-term advantages provided by hardware acceleration and power reduction.
Specifically, we implemented two types of ISP hardware components, in addition to the software components. First, we let each channel controller not only manage read/write operations to/from its NAND flash channel (as in the usual SSDs) but also perform primitive operations on the data stored in its NAND channel. The type of primitive operation performed depends on the machine learning algorithm used (the next subsection explains more details of such operations for SGD). Additionally, each channel controller in ISP-ML (slave) communicates with the cache controller (master) in a master-slave architecture. Second, we designed the cache controller so that it can collect the outcomes from each of the channel controllers, in addition to its inherent functionality as a cache (DRAM) manager inside the SSD controller. This master-slave architecture can be interpreted as a tiny-scale version of the master-slave architecture commonly used in distributed systems. Just as the channel controllers, the exact functionality of the cache controller can be optimized depending on the specific algorithm used. Both the channel controllers and the cache controller have internal memory, but the memory size in the latter is far greater than that in the former.
Specific parameters and considerations used in our implementation can be found in Section 4.1. There are a few points worth mentioning. Unlike existing conventional SSD simulators, the baseline SSD implemented in ISP-ML can store data in the NAND flash memory inside. In order to support reasonable simulation speed, we modeled ISP-ML at cycle-accurate transaction level while minimizing negative impact on accuracy. We omit to describe other minor details of hardware logic implementations, as are beyond the scope of the conference.
3.2 PARALLEL SGD IMPLEMENTATION ON ISP-ML
Using our ISP-ML platform, we implemented the three types of parallel SGD algorithms outlined in Figure 2: synchronous SGD (Zinkevich et al., 2010), Downpour SGD (Dean et al., 2012), and EASGD (Zhang et al., 2015). For brevity, we focus on describing the implementation details of these algorithms in ISP-ML and omit the purely algorithmic details of each algorithm; we refer the interested to the corresponding references. Note that the size of a minibatch for the minibatch SGD in our framework is set to the number of training samples in a NAND flash page (referred to as ‘page-minibatch’ in Figure 2).
For implementing synchronous SGD, we let each of the n channel controllers synchronously compute the gradient. Firstly, each channel controller reads page-sized data from the NAND flash memory and then stores the data in the channel controller’s buffer. Secondly, the channel controller pulls the cache controller’s parameters (θcache) and stores them in the buffer. Using the data and parameters stored in the buffer, each channel controller calculates the gradient in parallel. After transferring the gradient to the cache controller, the channel controllers wait for a signal from the cache controller. The cache controller aggregates and updates the parameters and then sends the channel controller signals to pull and replicate the parameters.
We implemented Downpour SGD in a similar way to implementing synchronous SGD; the major difference is that each channel controller immediately begins the next iteration after transferring the
gradient to cache controller. The cache controller updates the parameters with the gradient from the channel controllers sequentially.
For EASGD, we let each of the channel controllers have its own SGD parameters unlike synchronous SGD and Downpour SGD. Each channel controller pulls the parameters from the cache controller after computing the gradient and updating its own parameters. Each channel controller calculates the differences between its own parameters and the cache controller’s parameters and then pushes the differences to the cache controller.
Of note is that, besides its widespread use, SGD has some appealing characteristics that facilitate hardware implementations. We can implement parallel SGD on top of the master-slave architecture realized by the cache controller and the channel controllers. We can also take advantage of effective techniques developed in the distributed and parallel computation domain. Importantly, each SGD iteration is so simple that it can be implemented without incurring excessive hardware overhead.
3.3 METHODOLOGY FOR IHP-ISP PERFORMANCE COMPARISON
To evaluate the effectiveness of ISP, it is crucial to accurately and fairly compare the performances of ISP and the conventional IHP. However, performing this type of comparison is not trivial (see Section 4.3 for additional discussion). Furthermore, the accurate modeling of commercial SSDs equipped with ISP-ML is impossible due to lack of information about commercial SSDs (e.g., there is no public information on the FTL and internal architectures of any commercial SSD). Therefore, we propose a practical methodology for accurate comparison of IHP and ISP performances, as depicted in Figure 3. Note that this comparison methodology is applicable not only to the parallel SGD implementations explained above but also to other ML algorithms that can be executed in ISP-ML.
In the proposed comparison methodology, we focus on the data IO latency time of the storage (denoted as TIO), since it is the most critical factor among those that affect the execution time of IHP. The total processing time of IHP (IHPtime or Ttotal) can then be divided into the data IO time and the non-data IO time (TnonIO) as follows:
IHPtime = Ttotal = TnonIO + TIO. (4)
To calculate the expected IHP simulation time adjusted to ISP-ML, the data IO time of IHP is replaced by the data IO time of the baseline SSD in ISP-ML (TIOsim). By using Eq. (4), the expected IHP simulation time can then be represented by
Expected IHP simulation time = TnonIO + TIOsim = Ttotal − TIO + TIOsim. (5)
The overall flow of the proposed comparison methodology is depicted in Figure 3(b). First, the total processing time (Ttotal) and the data IO time of storage (TIO) are measured in IHP, extracting the IO trace of storage during an application execution. The simulation IO time (TIOsim) is then measured using the IO trace (extracted from IHP) on the baseline SSD of ISP-ML. Finally, the expected IHP simulation time is calculated by plugging the total processing time (Ttotal), the data IO time of storage (TIO) and the simulation IO time (TIOsim) into Eq. (5). With the proposed method and ISPML, which is applicable to a variety of IHP environments regardless of the type of storage used, it is possible to quickly and easily compare performances of various ISP implementations and IHP in a simulation environment.
4 EXPERIMENTAL RESULTS
4.1 SETUP
All the experiments presented in this section were run on a machine equipped with an 8-core Intel(R) Core i7-3770K CPU (3.50GHz) with DDR3 32GB RAM, Samsung SSD 840 Pro, and Ubuntu 14.04 LTS (kernel version: 3.19.0-26-generic). We used ARM 926EJ-S (400MHz) as the embedded processor inside ISP-ML and DFTL (Gupta et al., 2009) as the FTL of ISP-ML. The simulation model we used was derived from a commercial product (Micron NAND MT29F8G08ABACA) and had the following specifications: page size = 8KB, tprog = 300µs, tread =75µs, and tblock erase = 5ms.1 Each channel controller had 24KB of memory [8KB (page size) for data and 16KB for ISP] and a floating point unit (FPU) having 0.5 instruction/cycle performance (with pipelining). The cache controller had memory of (n+ 1)× 8KB (page size), where n is the number of channels (n = 4, 8, 16). Depending on the algorithm running in ISP-ML, we can adjust these parameters.
Note that the main purpose of our experiments in this paper was to verify the functionality of our ISP-ML framework and to evaluate the effectiveness of ISP over the conventional IHP using SGD, even though our framework is certainly not limited only to SGD. To this end, we selected logistic regression, a fundamental ML algorithm that can directly show the advantage of ISP-based optimizations over IHP-based optimizations without unnecessary complications. We thus implemented the logistic regression algorithm as a single-layer perceptron (with cross entropy loss) in SystemC and uploaded it to ISP-ML. As stated in Section 5.3, our future work includes the implementation and testing of more complicated models (such as deep neural networks) by reflecting the improvement opportunities revealed from the experiments presented in this paper.
As test data, we utilized the samples from the MNIST database (LeCun et al., 1998). To amplify the number of training samples (for showing the scalability of our approach), we used elastic distortion (Simard et al., 2003), producing 10 times more data than the original MNIST (approximately 600,000 training and 10,000 test samples were used in total). To focus on the performance evaluation of running ISP operations, we preloaded our NAND flash simulation model with the simulation
1These are conservative settings, compared with those of the original commercial product; using the specifications of a commercial product will thus improve the performance of ISP-ML.
data (the same condition was used for the alternatives for fairness). Based on the size of a training sample this dataset and the size of a NAND page (8KB), we set the size of each minibatch to 10.
4.2 PERFORMANCE COMPARISON: ISP-BASED OPTIMIZATION
As previously explained, to identify which SGD algorithm would be best suited for use in ISP, we implemented and analyzed three types of SGD algorithms: synchronous SGD, Downpour SGD, and EASGD. For EASGD, we set the moving rate (α) and the communication period (τ ) to 0.001 and 1, respectively. For a fair comparison, we chose different learning rates for different algorithms that gave the best performance for each algorithm. Figure 4 shows the test accuracy of three algorithms with varying numbers of channels (4, 8, and 16) with respect to wall-clock time.
As shown in Figure 4, using EASGD gave the best convergence speed in all of the cases tested. EASGD outperformed synchronous and Downpour SGD by factors of 5.24 and 1.96 on average, respectively. Synchronous SGD showed a slower convergence speed when compared to Downpour SGD because it could not start learning on the next set of minibatch until the results of all the channel controllers reported to the cache controller. Moreover, one delayed worker could halt the entire process. This result suggests that EASGD is adequate for all the channel configurations tested in that ISP can benefit from ultra-fast on-chip level communication and employ application-specific hardware that can eliminate any interruptions from other processors.
4.3 PERFORMANCE COMPARISON: IHP VERSUS ISP
In large-scale machine learning, the computing systems used may suffer from memory shortage, which incurs significant data swapping overhead. In this regard, ISP can provide an effective solution that can potentially reduce data transfer penalty by processing core operations at the storage level.
In this context, we carried out additional experiments to compare the performance of IHP-based and ISP-based EASGD. We tested the effectiveness of ISP in a memory shortage situation with 5 different configurations of IHP memory: 2GB, 4GB, 8GB, 16GB, and 32GB. We assumed that the host already loaded all of the data to the main memory for IHP. This assumption is realistic because
state-of-the-art machine learning techniques often employ a prefetch strategy to hide the initial data transfer latency.
As depicted in Figure 5, ISP-based EASGD with 16 channels gave the best performance in our experiments. The convergence speed of the IHP-based optimization slowed down, in accordance with the reduced memory size. The results with 16GB and 32GB of memory gave similar results because using 16GB of memory was enough to load and allocate most of the resource required by the process. As a result, ISP was more efficient when memory was insufficient, as would be often the case with large-scale datasets in practice.
4.4 CHANNEL PARALLELISM
To closely examine the effect of exploiting data-level parallelism on performance, we compared the accuracy of the three SGD algorithms, varying the number of channels (4, 8, and 16), as shown in Figure 6. All the three algorithms resulted in convergence speed-up by using more channels; synchronous SGD achieved 1.48× speed-up when the number of channels increased from 8 to 16. From Figure 6(d), we can also note that the convergence speed-up tends to be proportional to number of channels. These results suggest that the communication overhead in ISP is negligible, and that ISP does not suffer from the communication bottleneck that commonly occurs in distributed computing systems.
4.5 EFFECTS OF COMMUNICATION PERIOD IN ASYNCHRONOUS SGD
Finally, we investigated how changes in the communication period (i.e., how often data exchange occurs during distributed optimization) affect SGD performance in the ISP environment. Figure 7 shows the test accuracy of the Downpour SGD and EASGD algorithms versus wall-clock time when we varied their communication periods. As described in Zhang et al. (2015), Downpour SGD normally achieved a high performance for a low communication period [τ = 1, 4] and became unstable for a high communication period [τ = 16, 64] in ISP. Interestingly, in contrast to the conventional
distributed computing system setting, the performance of EASGD decreased as the communication period increased in the ISP setting. This is because the on-chip communication overhead in ISP is significantly lower than that in the distributed computing system. As a result, there would be no need for extending the communication period to reduce communication overhead in the ISP environment.
5 DISCUSSION
5.1 PARALLELISM IN ISP
Given the advances in underlying hardware and semiconductor technology, ISP can provide various advantages for data processing involved in machine learning. For example, our ISP-ML could minimize (practically eliminate) the communication overheads between parallel nodes leveraged by ultra-fast on-chip communication inside an SSD. Minimizing communication overheads can improve various key aspects of data-processing systems, such as energy efficiency, data management, security, and reliability. By exploiting this advantage of fast on-chip communications in ISP, we envision that we will be able to devise a new kind of parallel algorithms for optimization and machine learning running on ISP-based SSDs.
Our experiment results also revealed that a high degree of parallelism could be achieved by increasing the number of channels inside an SSD. Some of the currently available commercial SSDs have as many as 16 channels. Given that the commercial ISP-supporting SSDs would (at least initially) be targeted at high-end SSD markets with many NAND flash channels, our approach is expected to add a valuable functionality to such SSDs. Unless carefully optimized, a conventional distributed system will see diminishing returns as the number of nodes increases, due to the increased communication overhead and other factors. Exploiting a hierarchy of parallelism (i.e., parallel computing nodes, each of which has ISP-based SSDs with parallelism inside) may provide an effective acceleration scheme, although a fair amount of additional research is needed before we can realize this idea.
5.2 ISP-IHP COMPARISON METHODOLOGY
To fairly compare the performances of ISP and IHP, it would be ideal to implement ISP-ML in a real semiconductor chip, or to simulate IHP in the ISP-ML framework. Selecting either option, however, is possible but not plausible (at least in academia), because of high cost of manufacturing a chip, and the prohibitively high simulation time for simulating IHP in the Synopsys Platform Architect environment (we would have to implement many components of a modern computer system in order to simulate IHP). Another option would be to implement both ISP and IHP using FPGAs, but it will take another round of significant efforts for developments.
To overcome these challenges (still assuring a fair comparison between ISP and IHP), we have proposed the comparison methodology described in Section 3.3. In terms of measuring the absolute running time, our methodology may not be ideal. However, in terms of highlighting relative performance between alternatives, our method should provide a satisfactory solution.
Our comparison methodology extracts IO trace from the storage while executing an application in the host, which is used for measuring simulation IO time in the baseline SSD in ISP-ML. In this procedure, we assume that the non-IO time of IHP is consistent regardless of the kind of storage the host has. The validity of this assumption is warranted by the fact that the amount of non-IO time changed by the storage is usually negligible compared with the total execution time or IO time.
5.3 OPPORTUNITIES FOR FUTURE RESEARCH
In this paper we focused on the implementation and testing of ISP-based SGD as a proof of concept. The simplicity and popularity of (parallel) SGD underlie our choice. By design, it is possible to run other algorithms in our ISP-ML framework immediately; recall that our framework includes a general-purpose ARM processor that can run executables compiled from C/C++ code. However, it would be meaningless just to have an ISP-based implementation, if its performance is unsatisfactory. To unleash the full power of ISP, we need additional ISP-specific optimization efforts, as is typically the case with hardware design.
With this in mind, we have started implementing deep neural networks (with realistic numbers of layers and hyperparameters) using our ISP-ML framework. Especially, we are carefully devising a way of balancing the memory usage in the DRAM buffer, the cache controller, and the channel controllers inside ISP-ML. It would be reasonable to see an SSD with a DRAM cache with a few gigabytes of memory, whereas it is unrealistic to design a channel controller with that much memory. Given that a large amount of memory is needed only to store the parameters of such deep models, and that IHP and ISP have different advantage and disadvantages, it would be intriguing to investigate how to make IHP and ISP can cooperate to enhance the overall performance. For instance, we can let ISP-based SSDs perform low-level data-dependent tasks while assigning high-level tasks to the host, expanding the current roles of the cache controller and the channel controllers inside ISP-ML to the whole system level.
Our future work also includes the following: First, we will be able to implement adaptive optimization algorithms such as Adagrad (Duchi et al., 2011) and Adadelta (Zeiler, 2012). Second, precomputing meta-data during data writes (instead of data reads) could provide another direction of research that can bring even more speedup. Third, we will be able to implement data shuffle functionality in order to maximize the effect of data-level parallelism. Currently, ISP-ML arbitrarily splits the input data into its multi-channel NAND flash array. Fourth, we may investigate the effect of NAND flash design on performance, such as the NAND flash page size. Typically, the size of a NAND flash page significantly affects the performance of SSDs, given that the page size (e.g., 8KB) is the basic unit of NAND operation (read and write). In case where the size of a single example often exceeds the page size, frequent data fragmentation is inevitable, eventually affecting the overall performance. The effectiveness of using multiple page sizes was already reported for conventional SSDs (Kim et al., 2016a), and we may borrow this idea to further optimize ISP-ML.
ACKNOWLEDGMENTS
The authors would like to thank Byunghan Lee at Data Science Laboratory, Seoul National University for proofreading the manuscript. This work was supported in part by BK21 Plus (Electrical and Computer Engineering, Seoul National University) in 2016, in part by a grant from SK Hynix, and in part by a grant from Samsung Electronics.

For more than a decade, near data processing has been a key requirement for large scale linear learning platforms, as the time to load the data exceeds the learning time, and this has justified the introduction of approaches such as Spark
Deep learning usually deals with the data that can be contained in a single machine and the bottleneck is often the CPU-GPU bus or the GPU-GPU-bus, so a method that overcomes this bottleneck could be relevant.
Unfortunately, this work is still very preliminary and limited to linear training algorithms, so of little interest yet to ICLR readership. I would recommend publication to a conference where it can reach the large-scale linear ML audience first, such as ICML. This paper is clear and well written in the present form and would probably mostly need a proper benchmark on a large scale linear task. Obviously, when the authors have convincing DNN learning simulations, they are welcome to target ICLR, but can the flash memory FPGA handle it?
For experiments, the choice of MNIST is somewhat bizarre: this task is small and performance is notoriously terrible when using linear approaches (the authors do not even report it)


SOLVING INVERSE PROBLEMS
Consider an ill-posed inverse problem of estimating causal factors from observations, one of which is known to lie near some (unknown) low-dimensional, nonlinear manifold expressed by a predefined Mercer-kernel. Solving this problem requires simultaneous estimation of these factors and learning the low-dimensional representation for them. In this work, we introduce a novel non-linear dimensionality regularization technique for solving such problems without pre-training. We re-formulate Kernel-PCA as an energy minimization problem in which low dimensionality constraints are introduced as regularization terms in the energy. To the best of our knowledge, ours is the first attempt to create a dimensionality regularizer in the KPCA framework. Our approach relies on robustly penalizing the rank of the recovered factors directly in the implicit feature space to create their low-dimensional approximations in closed form. Our approach performs robust KPCA in the presence of missing data and noise. We demonstrate state-of-the-art results on predicting missing entries in the standard oil flow dataset. Additionally, we evaluate our method on the challenging problem of Non-Rigid Structure from Motion and our approach delivers promising results on CMU mocap dataset despite the presence of significant occlusions and noise.
1 INTRODUCTION
Dimensionality reduction techniques are widely used in data modeling, visualization and unsupervised learning. Principal component analysis (PCAJolliffe (2002)), Kernel PCA (KPCASchölkopf et al. (1998)) and Latent Variable Models (LVMsLawrence (2005)) are some of the well known techniques used to create low dimensional representations of the given data while preserving its significant information.
One key deployment of low-dimensional modeling occurs in solving ill-posed inference problems. Assuming the valid solutions to the problem lie near a low-dimensional manifold (i.e. can be parametrized with a reduced set of variables) allows for a tractable inference for otherwise underconstrained problems. After the seminal work of Candès & Recht (2009); Recht et al. (2010) on guaranteed rank minimization of the matrix via trace norm heuristics Fazel (2002), many ill-posed computer vision problems have been tackled by using the trace norm — a convex surrogate of the rank function — as a regularization term in an energy minimization frameworkCandès & Recht (2009); Zhou et al. (2014). The flexible and easy integration of low-rank priors is one of key factors for versatility and success of many algorithms. For example, pre-trained active appearance models Cootes et al. (2001) or 3D morphable models Blanz & Vetter (1999) are converted to robust feature tracking Poling et al. (2014), dense registration Garg et al. (2013b) and vivid reconstructions of natural videos Garg et al. (2013a) with no a priori knowledge of the scene. Various bilinear factorization problems like background modeling, structure from motion or photometric stereo are also addressed with a variational formulation of the trace norm regularization Cabral et al. (2013).
On the other hand, although many non-linear dimensionality reduction techniques — in particular KPCA — have been shown to outperform their linear counterparts for many data modeling tasks, they are seldom used to solve inverse problems without using a training phase. A general (discriminative) framework for using non-linear dimensionality reduction is: (i) learn a low-dimensional representation for the data using training examples via the kernel trick (ii) project the test examples on the learned manifold and finally (iii) find a data point (pre-image) corresponding to each projection in the input space.
This setup has two major disadvantages. Firstly, many problems of interest come with corrupted observations — noise, missing data and outliers — which violate the low-dimensional modeling assumption.Secondly, computing the pre-image of any point in the low dimensional feature subspace is non-trivial: the pre-image for many points in the low dimensional space might not even exist because the non linear feature mapping function used for mapping the data from input space to the feature space is non-surjective.
Previously, extensions to KPCA like Robust KPCA (RKPCANguyen & De la Torre (2009)) and probabilistic KPCA (PKPCASanguinetti & Lawrence (2006)) with missing data have been proposed to address the first concern, while various additional regularizers have been used to estimate the pre-image robustly Bakir et al. (2004); Mika et al. (1998); Kwok & Tsang (2004); Abrahamsen & Hansen (2009).
Generative models like LVMs Lawrence (2005) are often used for inference by searching the lowdimensional latent space for a location which maximizes the likelihood of the observations. Problems like segmentation, tracking and semantic 3D reconstruction Prisacariu & Reid (2011); Dame et al. (2013) greatly benefit from using LVM. However, the latent space is learned a priori with clean training data in all these approaches.
Almost all non-linear dimensionality reduction techniques are non-trivial to generalize for solving ill-posed problems (See section 4.2) without a pre-training stage. Badly under-constrained problems require the low-dimensional constraints even for finding an initial solution, eliminating applicability of the standard “projection + pre-image estimation” paradigm. This hinders the utility of nonlinear dimensionality reduction and a suitable regularization technique to penalize the non-linear dimensionality is desirable.
Sum and Substance: A closer look at most non-linear dimensionality reduction techniques reveals that they rely upon a non-linear mapping function which maps the data from input space to a (usually) higher dimensional feature space. In this feature space the data is assumed to lie on a low-dimensional hyperplane — thus, linear low-rank prior is apt in the feature space. Armed with this simple observation, our aim is to focus on incorporating the advances made in linear dimensionality reduction techniques to their non-linear counterparts, while addressing the problems described above. Figure 1 explains this central idea and proposed dimensionality regularizer in a nutshell with Non Rigid Structure from Motion (NRSfM) as the example application.
Our Contribution: In this work we propose a unified for simultaneous robust KPCA and preimage estimation while solving an ill-posed inference problem without a pre-training stage.
In particular we propose a novel robust energy minimization algorithm which handles the implicitness of the feature space to directly penalize its rank by iteratively: (i) creating robust low-dimensional representation for the
data given the kernel matrix in closed form and (ii) reconstructing the noise-free version of the data (pre-image of the features space projections) using the estimated low-dimensional representations in a unified framework.
The proposed algorithm: (i) provides a novel closed form solution to robust KPCA; (ii) yields stateof-the-art results on missing data prediction for the well-known oil flow dataset; (iii) outperforms state-of-the-art linear dimensionality (rank) regularizers to solve NRSfM; and (iv) can be trivially generalized to incorporate other cost functions in an energy minimization framework to solve various ill-posed inference problems.
2 PROBLEM FORMULATION
This paper focuses on solving a generic inverse problem of recovering causal factor S = [s1, s2, · · · sN ] ∈ X × N from N observations W = [w1, w2, · · · wN ] ∈ Y × N such that f(W,S) = 0. Here function f(observation,variable), is a generic loss function which aligns the observations W with the variable S (possibly via other causal factors. e.g. R or Z in Section 4.1 and 4.2).
If, f(W,S) = 0 is ill-conditioned (for example when Y X ), we want to recover matrix S under the assumption that the columns of it lie near a low-dimensional non-linear manifold. This can be done by solving a constrained optimization problem of the following form:
min S
rank(Φ(S))
s.t. f(W,S) ≤ (1)
where Φ(S) = [φ(s1), φ(s2), · · · , φ(sN )] ∈ H ×N is the non-linear mapping of matrix S from the input space X to the feature space H (also commonly referred as Reproducing Kernel Hilbert Space), via a non-linear mapping function φ : X → H associated with a Mercer kernel K such that K(S)i,j = φ(si) Tφ(sj).
In this paper we present a novel energy minimization framework to solve problems of the general form (1).
As our first contribution, we relax the problem (1) by using the trace norm of Φ(S) — the convex surrogate of rank function — as a penalization function. The trace norm ‖M‖∗ =: ∑ i λi(M) of a matrix M is the sum of its eigenvalues λi(M) and was proposed as a tight convex relaxation1 of the rank(M) and is used in many vision problems as a rank regularizer Fazel (2002). Although the rank minimization via trace norm relaxation does not lead to a convex problem in presence of a non-linear kernel function, we show in 3.2 that it leads to a closed-form solution to denoising a kernel matrix via penalizing the rank of recovered data (S) directly in the feature space.
With these changes we can rewrite (1) as:
min S
f(W,S) + τ‖Φ(S)‖∗ (2)
where τ is a regularization strength.2
It is important to notice that although the rank of the kernel matrix K(S) is equal to the rank of Φ(S), ‖K(S)‖∗ is merely ‖Φ(S)‖2F . Thus, directly penalizing the sum of the singular values of K(S) will not encourage low-rank in the feature space.3
Although we have relaxed the non-convex rank function, (2) is in general difficult to minimize due to the implicitness of the feature space. Most widely used kernel functions like RBF do not have a explicit definition of the function φ. Moreover, the feature space for many kernels is high(possibly infinite-) dimensional, leading to intractability. These issues are identified as the main
1More precisely, ‖M‖∗ was shown to be the tight convex envelope of rank(M)/‖M‖s, where ‖M‖s represent spectral norm of M .
21/τ can also be viewed as Lagrange multiplier to the constraints in (1). 3Although it is clear that relaxing the rank of kernel matrix to ‖K(S)‖∗ is suboptimal, works like Huang et al. (2012); Cabral et al. (2013) with a variational definition of nuclear norm, allude to the possibility of kernelization. Further investigation is required to compare this counterpart to our tighter relaxation.
barriers to robust KPCA and pre-image estimation Nguyen & De la Torre (2009). Thus, we have to reformulate (2) by applying kernel trick where the cost function (2) can be expressed in terms of the kernel function alone.
The key insight here is that under the assumption that kernel matrix K(S) is positive semidefinite, we can factorize it as: K(S) = CTC. Although, this factorization is non-unique, it is trivial to show the following: √
λi(K(S)) = λi(C) = λi(Φ(S))
Thus: ‖C‖∗ = ‖Φ(S)‖∗ ∀ C : CTC = K(S) (3) where λi(.) is the function mapping the input matrix to its ith largest eigenvalue.
The row space of matrix C in (3) can be seen to span the eigenvectors associated with the kernel matrix K(S) — hence the principal components of the non-linear manifold we want to estimate.
Using (3), problem (2) can finally be written as:
min S,C
f(W,S) + τ‖C‖∗
s.t. K(S) = CTC (4)
The above minimization can be solved with a soft relaxation of the manifold constraint by assuming that the columns of S lie near the non-linear manifold.
min S,C
f(W,S) + ρ
2 ‖K(S)− CTC‖2F + τ‖C‖∗ (5)
As ρ → ∞, the optimum of (5) approaches the optimum of (4) . A local optimum of (4) can be achieved using the penalty method of Nocedal & Wright (2006) by optimizing (5) while iteratively increasing ρ as explained in Section 3.
Before moving on, we would like to discuss some alternative interpretations of (5) and its relationship to previous work – in particular LVMs. Intuitively, we can also interpret (5) from the probabilistic viewpoint as commonly used in latent variable model based approaches to define kernel function Lawrence (2005). For example a RBF kernel with additive Gaussian noise and inverse width γ can be defined as: K(S)i,j = e−γ‖si−sj‖ 2
+ , where ∼ N (0, σ). In other words, with a finite ρ, our model allows the data points to lie near a non-linear low-rank manifold instead of on it. Its worth noting here that like LVMs, our energy formulation also attempts to maximize the likelihood of regenerating the training data W , (by choosing f(W,S) to be a simple least squares cost) while doing dimensionality reduction.
Note that in closely related work Geiger et al. (2009), continuous rank penalization (with a logarithmic prior) has also been used for robust probabilistic non-linear dimensionality reduction and model selection in LVM framework. However, unlike Geiger et al. (2009); Lawrence (2005) where the non-linearities are modeled in latent space (of predefined dimensionality), our approach directly penalizes the non-linear dimensionality of data in a KPCA framework and is applicable to solve inverse problems without pre-training.
3 OPTIMIZATION
We approach the optimization of (5) by solving the following two sub-problems in alternation:
min S
f(W,S) + ρ
2 ‖K(S)− CTC‖2F (6)
min C
τ‖C‖∗ + ρ
2 ‖K(S)− CTC‖2F (7)
Algorithm 1 outlines the approach and we give a detailed description and interpretations of both sub-problems (7) and (6) in next two sections of the paper.
3.1 PRE-IMAGE ESTIMATION TO SOLVE INVERSE PROBLEM.
Subproblem (6) can be seen as a generalized pre-image estimation problem: we seek the factor si, which is the pre-image of the projection of φ(si) onto the principle subspace of the RKHS stored in
Algorithm 1: Inference with Proposed Regularizer. Input: Initial estimate S0 of S. Output: Low-dimensional S and kernel representation C. Parameters: Initial ρ0 and maximum ρmax penalty, with scale ρs. - S = S0, ρ = ρ0 ; while ρ ≤ ρmax do
while not converged do - Fix S and estimate C via closed-form solution of (7) using Algorithm 2; - Fix C and minimize (6) to update S using LM algorithm;
- ρ = ρρs ;
CTC, which best explains the observation wi. Here (6) is generally a non-convex problem, unless the Mercer-kernel is linear, and must therefore be solved using non-linear optimization techniques. In this work, we use the Levenberg-Marquardt algorithm for optimizing (6).
Notice that (6) only computes the pre-image for the feature space projections of the data points with which the non-linear manifold (matrix C) is learned. An extension to our formulation is desirable if one wants to use the learned non-linear manifold for denoising test data in a classic pre-image estimation framework. Although a valuable direction to pursue, it is out of scope of the present paper.
3.2 ROBUST DIMENSIONALITY REDUCTION
Algorithm 2: Robust Dimensionality Reduction. Input: Current estimate of S. Output: Low-dimensional representation C. Parameters: Current ρ and regularization strength τ .
- [U Λ UT ] = Singular Value Decomposition of K(S); // Λ is a diagonal matrix, storing N
singular values λi of K(S). for i = 1 to N do
- Find three solutions (lr : r ∈ {1, 2, 3}) of: l3 − lλi + τ
2ρ = 0 ;
- set l4 = 0; - lr = max(lr, 0) ∀r ∈ {1, 2, 3, 4} ; - r = argmin
r { ρ 2 ‖λi − l2r‖2 + τ lr } ;
- λ̄i = lr ;
- C = Λ̄UT ; // Λ̄ is diagonal matrix storing λ̄i.
One can interpret sub-problem (7) as a robust form of KPCA where the kernel matrix has been corrupted with Gaussian noise and we want to generate its low-rank approximation. Although (7) is non-convex we can solve it in closed-form via singular value decomposition. This closed-form solution is outlined in Algorithm 2 and is based on the following theorem: Theorem 1. With Sn 3 A 0 let A = UΣUT denote its singular value decomposition. Then
min L
ρ 2 ||A− LTL||2F + τ ||L||∗ (8)
= n∑ i=1 (ρ 2 (σi − γ∗2i )2 + τγ∗i ) . (9)
A minimizer L∗ of (8) is given by
L∗ = Γ∗UT (10) with Γ∗ ∈ Dn+, γ∗i ∈ {α ∈ R+ | pσi,τ/2ρ(α) = 0} ⋃ {0}, where pa,b denotes the depressed cubic pa,b(x) = x 3 − ax+ b. Dn+ is the set of n-by-n diagonal matrices with non-negative entries.
Theorem 1 shows that each eigenvalue of the minimizer C∗ of (7) can be obtained by solving a depressed cubic whose coefficients are determined by the corresponding eigenvalue of the kernel matrix and the regularization strength τ . The roots of each cubic, together with zero, comprise a set of candidates for the corresponding eigenvalue of C∗. The best one from this set is obtained by choosing the value which minimizes (9) (see Algorithm 2).
As elaborated in Section 2, problem (7) can be seen as regularizing sum of square root (L1/2 norm) of the eigenvalues of the matrix K(S). In a closely related work Zongben et al. (2012), authors advocateL1/2 norm as a better approximation for the cardinality of a vector then the more commonly used L1 norm. A closed form solution for L1/2 regularization similar to our work was outlined in Zongben et al. (2012) and was shown to outperform the L1 vector norm regularization for sparse coding. To that end, our Theorem 1 and the proposed closed form solution (Algo 2) for (7) can
be seen as generalization of Zongben et al. (2012) to include the L1/2 matrix norms for which a simplified proof is included in the Appendix A. It is important to note however, that the motivation and implication of using L1/2 regularization in the context of non-linear dimensionality reduction are significantly different to that of Zongben et al. (2012) and related work Du et al. (2013); Zhao et al. (2014) which are designed for linear modeling of the causal factors. The core insight of using L1 regularization in the feature space via the parametrization given in 3 facilitates a natural way for non-linear modeling of causal factors with low dimensionality while solving an inverse problem by making feature space tractable.
4 EXPERIMENTS
In this section we demonstrate the utility of the proposed algorithm. The aims of our experiments are twofold: (i) to compare our dimensionality reduction technique favorably with KPCA and its robust variants; and (ii) to demonstrate that the proposed non-linear dimensionality regularizer consistently outperforms its linear counterpart (a.k.a. nuclear norm) in solving inverse problems.
4.1 MATRIX COMPLETION
The nuclear norm has been introduced as a low rank prior originally for solving the matrix completion problem. Thus, it is natural to evaluate its non-linear extensions on the same task. Assuming W ∈ Rm×n to be the input matrix and Z a binary matrix specifying the availability of the observations in W , Algorithm 1 can be used for recovering a complete matrix S with the following choice of f(W,Z, S): f(W,Z, S) = ‖Z ◦ (W − S)‖2F (11) where ◦ represents Hadamard product. To demonstrate the robustness of our algorithm for matrix completion problem, we choose 100 training samples from the oil flow dataset described in section 3.2 and randomly remove the elements from the data with varying range of probabilities to test the performance of the proposed algorithm against various baselines. Following the experimental setup as specified in Sanguinetti & Lawrence (2006), we repeat the experiments with 50 different samples of Z. We report the mean and standard deviation of the root mean square reconstruction error for our method with the choice of τ = 0.1, alongside five different methods in Table 1. Our method significantly improves the performance of missing data completion compared to other robust extensions of KPCA Tipping & Bishop (1999); Sanguinetti & Lawrence (2006); Nguyen & De la Torre (2009), for every probability of missing data.
Although we restrict our experiments to least-squares cost functions, it is vital to restate here that our framework could trivially incorporate robust functions like the L1 norm instead of the Frobenius norm — as a robust data term f(W,Z, S) — to generalize algorithms like Robust PCA Wright et al. (2009) to their non-linear counterparts.
4.2 KERNEL NON-RIGID STRUCTURE FROM MOTION
Non-rigid structure from motion under orthography is an ill-posed problem where the goal is to estimate the camera locations and 3D structure of a deformable objects from a collection of 2D images which are labeled with landmark correspondences Bregler et al. (2000). Assuming si(xj) ∈ R3 to be the 3D location of point xj on the deformable object in the ith image, its orthographic projection wi(xj) ∈ R2 can be written as wi(x) = Risi(xj), where Ri ∈ R2×3 is a orthographic projection matrix Bregler et al. (2000). Notice that as the object deforms, even with given camera poses, reconstructing the sequence by least-squares reprojection error minimization is an ill-posed problem. In their seminal work, Bregler et al. (2000) proposed to solve this problem with an additional assumption that the reconstructed shapes lie on a low-dimensional linear subspace and can be parameterized as linear combinations of a relatively low number of basis shapes. NRSfM was then cast as the low-rank factorization problem of estimating these basis shapes and corresponding coefficients.
Recent work, like Dai et al. (2014); Garg et al. (2013a) have shown that the trace norm regularizer can be used as a convex envelope of the low-rank prior to robustly address ill-posed nature of the problem. A good solution to NRSfM can be achieved by optimizing:
min S,R τ‖S‖∗ + F∑ i=1 N∑ j=1 Zi(xj)‖wi(xj)−Risi(xj)‖2F (12)
where S is the shape matrix whose columns are 3N dimensional vectors storing the 3D coordinates Si(xj) of the shapes and Zi(xj) is a binary variable indicating if projection of point xj is available in the image i.
Assuming the projection matrices to be fixed, this problem is convex and can be exactly solved with standard convex optimization methods. Additionally, if the 2D projections wi(xj) are noise free, optimizing (12) with very small τ corresponds to selecting the the solution — out of the many solutions — with (almost) zero projection error, which has minimum trace norm Dai et al. (2014). Thus henceforth, optimization of (12) is referred as the trace norm heuristics (TNH). We solve this problem with a first order primal-dual variant of the algorithm given in Garg et al. (2013a), which can handle missing data. The algorithm is detailed and compared favorably with the state of the art NRSfM approaches (based on linear dimensionality regularization) Appendix C.
A simple kernel extension of the above optimization problem is:
min S,R τ‖Φ(S)‖∗ + F∑ i=1 N∑ j=1
Zi(xj)‖wi(xj)−Risi(xj)‖2F︸ ︷︷ ︸ f(W,Z,R,S)
(13)
where Φ(S) is the non-linear mapping of S to the feature space using an RBF kernel.
With fixed projection matrices R, (13) is of the general form (2), for which the local optima can be found using Algorithm 1.
4.2.1 RESULTS ON THE CMU DATASET
We use a sub-sampled version of CMU mocap dataset by selecting every 10th frame of the smoothly deforming human body consisting 41 mocap points used in Dai et al. (2014).4
In our experiments we use ground truth camera projection matrices to compare our algorithm against TNH. The advantage of this setup is that with ground-truth rotation and no noise, we can avoid the model selection (finding optimal regularization strength τ ) by setting it low enough. We run the TNH with τ = 10−7 and use this reconstruction as initialization for Algorithm 1. For the proposed method, we set τ = 10−4 and use following RBF kernel width selection approach:
• Maximum distance criterion (dmax): we set the maximum distance in the feature space to be 3σ. Thus, the kernel matrix entry corresponding to the shape pairs obtained by TNH with maximum Euclidean distance becomes e−9/2.
• Median distance criterion (dmed): the kernel matrix entry corresponding to the median euclidean distance is set to 0.5.
Following the standard protocol in Dai et al. (2014); Akhter et al. (2009), we quantify the reconstruction results with normalized mean 3D errors e3D = 1σFN ∑ i ∑ j eij , where eij is the euclidean distance of a reconstructed point j in frame i from the ground truth, σ is the mean of standard deviation for 3 coordinates for the ground truth 3D structures, and F,N are number of input images and number of points reconstructed.
Table 2 shows the results of the TNH and non-linear dimensionality regularization based methods using the experimental setup explained above, both without missing data and after randomly removing 50% of the image measurements. Our method consistently beats the TNH baseline and improves the mean reconstruction error by ∼ 40% with full data and by ∼ 25% when used with 50% missing data. Figure 2 shows qualitative comparison of the obtained 3D reconstruction using TNH and proposed non-lienar dimensionality regularization technique for some sample frames from various sequences. We refer readers to Appendix B for results with simultaneous reconstruction pose optimization.
5 CONCLUSION
In this paper we have introduced a novel non-linear dimensionality regularizer which can be incorporated into an energy minimization framework, while solving an inverse problem. The proposed algorithm for penalizing the rank of the data in the feature space has been shown to be robust to noise and missing observations. We have picked NRSfM as an application to substantiate our arguments and have shown that despite missing data and model noise (such as erroneous camera poses) our algorithm significantly outperforms state-of-the-art linear counterparts.
Although our algorithm currently uses slow solvers such as the penalty method and is not directly scalable to very large problems like dense non-rigid reconstruction, we are actively considering alternatives to overcome these limitations. An extension to estimate pre-images with a problem-
4 Since our main goal is to validate the usefulness of the proposed non-linear dimensionality regularizer, we opt for a reduced size dataset for more rapid and flexible evaluation.
specific loss function is possible, and this will be useful for online inference with pre-learned lowdimensional manifolds.
Given the success of non-linear dimensionality reduction in modeling real data and overwhelming use of the linear dimensionality regularizers in solving real world problems, we expect that proposed non-linear dimensionality regularizer will be applicable to a wide variety of unsupervised inference problems: recommender systems; 3D reconstruction; denoising; shape prior based object segmentation; and tracking are all possible applications.
A PROOF OF THEOREM 3.1
Proof. We will prove theorem 1 by first establishing a lower bound for (8) and subsequently showing that this lower bound is obtained at L∗ given by (10). The rotational invariance of the entering norms allows us to write (8) as:
min Γ∈Dn, WTW=I
ρ 2 ||Σ−WΓ2WT ||2F + τ ||Γ||∗. (14)
Expanding (14) we obtain
ρ 2 min Γ,W
tr ( Σ2 ) − 2 tr ( ΣWΓ2WT ) + tr ( Γ4 ) + 2τ
ρ n∑ i=1 γi (15)
= ρ
2 min Γ,W n∑ i=1 ( σ2i + γ 4 i + 2τ ρ γi ) − 2 n∑ i=1 n∑ j=1 w2ijγ 2 j σi (16)
≥ ρ 2 min Γ n∑ i=1 ( σ2i − 2γ2i σi + γ4i + 2τ ρ γi ) (17)
= ρ
2 n∑ i=1 min γi≥0 ( σ2i − 2γ2i σi + γ4i + 2τ ρ γi ) (18)
The inequality in (17) follows directly by applying Hölder’s inequality to (16) and using the property that the column vectors wi are unitary.
Next, with L = ΓUT in (8) we have ρ
2 ||A− LTL||2F+τ ||L||∗ =
ρ 2 ||Σ− Γ2||2F + τ ||Γ||∗
= n∑ i=1 (ρ 2 (σi − γ2i )2 + τγi ) . (19)
Finally, since the subproblems in (18) are separable in γi, its minimizer must be KKT-points of the individual subproblems. As the constraints are simple non-negativity constraints, these KKT points are either (positive) stationary points of the objective functions or 0. It is simple to verify that the stationary points are given by the roots of the cubic function pσi,τ/2ρ. Hence it follows that there exists a γ ∗ i such that
ρ
2
( σ2i − 2γ2i σi + γ4i + 2τ
ρ γi
) ≥ ρ
2 (σi − γ∗2i )2 + τγ∗i , (20)
∀γi ≥ 0, which completes the proof.
A.1 VALIDATING THE CLOSED FORM SOLUTION
Given the relaxations proposed in Section 2, our assertion that the novel trace regularization based non-linear dimensionality reduction is robust need to be substantiated. To that end, we evaluate our closed-form solution of Algorithm 2 on the standard oil flow dataset introduced in Bishop & James (1993).
This dataset comprises 1000 training and 1000 testing data samples, each of which is of 12 dimensions and categorized into one of three different classes. We add zero mean Gaussian noise with variance σ to the training data5 and recover the low-dimensional manifold for this noisy training data Sσ with KPCA and contrast this with the results from Algorithm 2. An inverse width of the Gaussian kernel γ = 0.075 is used for all the experiments on the oil flow dataset.
It is important to note that in this experiment, we only estimate the principal components (and their variances) that explain the estimated non-linear manifold, i.e. matrix C by Algorithm 2, without reconstructing the denoised version of the corrupted data samples.
Both KPCA and our solution require model selection (choice of rank and τ respectively) which is beyond the scope of this paper. Here we resort to evaluate the performance of both methods under different parameters settings. To quantify the accuracy of the recovered manifold (C) we use following criteria:
5Note that our formulation assumes Gaussian noise in K(S) where as for this evaluation we add noise to S directly.
B KERNEL NRSFM WITH CAMERA POSE ESTIMATION
Extended from section 4.2
Table 4 shows the reconstruction performance on a more realistic experimental setup, with the modification that the camera projection matrices are initialized with rigid factorization and were refined with the shapes by optimizing (2). To solve NRSfM problem with unknown projection matrices, we parameterize each Ri with quaternions and alternate between refining the 3D shapes S and projection matrices R using LM. The regularization strength τ was selected for the TNH method by golden section search and parabolic interpolation for every test case independently. This ensures the best possible performance for the baseline. For our proposed approach τ was kept to 10−4 for all sequences for both missing data and full data NRSfM. This experimental protocol somewhat disadvantages the non-linear method, since its performance can be further improved by a judicious choice of the regularization strength.
6Errors from non-noisy kernel matrix can be replaced by cross validating the entries of the kernel matrix for model selection for more realistic experiment.
However our purpose is primarily to show that the non-linear method adds value even without timeconsuming per-sequence tuning. To that end, note that despite large errors in the camera pose estimations by TNH and 50% missing measurements, the proposed method shows significant (∼ 10%) improvements in terms of reconstruction errors, proving our broader claims that non-linear representations are better suited for modeling real data, and that our robust dimensionality regularizer can improve inference for ill-posed problems.
As suggested by Dai et al. (2014), robust camera pose initialization is beneficial for the structure estimation. We have used rigid factorization for initializing camera poses here but this can be trivially changed. We hope that further improvements can be made by choosing better kernel functions, with cross validation based model selection (value of τ ) and with a more appropriate tuning of kernel width. Selecting a suitable kernel and its parameters is crucial for success of kernelized algorithms. It becomes more challenging when no training data is available. We hope to explore other kernel functions and parameter selection criteria in our future work.
We would also like to contrast our work with Gotardo & Martinez (2011a), which is the only work we are aware of where non-linear dimensionality reduction is attempted for NRSfM. While estimating the shapes lying on a two dimensional non-linear manifold, Gotardo & Martinez (2011a) additionally assumes smooth 3D trajectories (parametrized with a low frequency DCT basis) and a pre-defined hard linear rank constraint on 3D shapes. The method relies on sparse approximation of the kernel matrix as a proxy for dimensionality reduction. The reported results were hard to replicate under our experimental setup for a fair comparison due to non-smooth deformations. However, in contrast to Gotardo & Martinez (2011a), our algorithm is applicable in a more general setup, can be modified to incorporate smoothness priors and robust data terms but more importantly, is flexible to integrate with a wide range of energy minimization formulations leading to a larger applicability beyond NRSfM.
C TNH ALGORITHM FOR NRSFM
In section 4.2, we have compared the proposed non-linear dimensionality reduction prior against a variant of Garg et al. (2013a) which handles missing data by optimizing:
min S,R τ‖S‖∗ + F∑ i=1 N∑ j=1 Zi(xj)‖wi(xj)−Risi(xj)‖2 (21)
This problem is convex in S given noise free projection matrix Ri’s but non-differentiable. To optimize (21), we first rewrite it in its primal-dual form by dualizing the trace norm 7 :
max Q min S,R τ < S,Q > + F∑ i=1 N∑ j=1 Zi(xj)‖wi(xj)−Risi(xj)‖2
s.t.‖Q‖s ≤ 1 (22)
where Q ∈ RX×N stores the dual variables to S and ‖.‖s represent spectral norm (highest eigenvalue) of a matrix.
7For more details on primal dual formulation and dual norm of the trace norm see Rockafellar (1974); Recht et al. (2010); Chambolle & Pock (2011).
Algorithm 3: Trace norm Heuristics. Input: Initial estimates S0, R0 of S and R. Output: Low-dimensional S and camera poses R. Parameters: Regularization strength τ , measurements W and binary mask Z.
Dataset PTAAkhter et al. (2009) CSF2Gotardo & Martinez (2011b) BMMDai et al. (2014) TNH Drink 0.0229 0.0215 0.0238 0.0237 Pick-up 0.0992 0.0814 0.0497 0.0482 Yoga 0.0580 0.0371 0.0334 0.0333 Stretch 0.0822 0.0442 0.0456 0.0431
We choose quaternions to perametrize the 2 × 3 camera matrices Ri to satisfy orthonormality constraints as done in Garg et al. (2013a) and optimize the saddle point problem (22) using alternation. In particular, for a single iteration: (i) we optimize the camera poses Ri’s using LM, (ii) take a steepest descend step for updating S and (ii) a steepest ascend step for updating Q which is followed by projecting its spectral norm to unit ball. Given ground truth camera matrices ( without step (i)), alternation (ii-iii) can be shown to reach global minima of (22). Algorithm 3 outlines TNH algorithm.
As the main manuscript uses NRSfM only as a practical application of our non-linear dimensionality reduction prior, we have restricted our NRSfM experiments to only compare the proposed method against its linear counterpart. For the timely evaluation, the reported experiments we conducted on sub-sampled CMU mocap dataset. Here, we supplement the arguments presented in the main manuscript by favorably comparing the linear dimensionality reduction based NRSfM algorithm(TNH) to other NRSfM methods on full length CMU mocap sequences.

This paper presents an approach to non-linear kernel dimensionality reduction with a trace norm regularizer in the feature space. The authors proposed an iterative minimization approach in order to obtain a local optimum of a relaxed problem.
The paper contains errors and the experimental evaluation is not convincing. Only old techniques are compared against in very toy datasets.
The authors claim state-of-the-art, however, the oil dataset is not a real benchmark, and the comparisons are to very old approaches.
The experimental evaluation should demonstrate robustness to more complex noise and outliers, as this was one of the motivations in the introduction.
The authors do not address the out-of-sample problem. This is a problem of kernel-based methods vs LVMs, and thus should be address here.
The paper contains errors:
- The last paragraph of section 1 says that this paper proposes a closed form solution to robust KPCA. This is simply wrong, as the proposed approach consists of iteratively solving iterativey a set of closed form updates  and Levenberg-Marquard optimizationd. This is not any more closed form!
- In the same paragraph (and later in the text) the authors claim that the proposed approach can be trivially generalized to incorporate other cost functions. This is not true, as in general there will be no more inner loop closed form updates and the authors will need to solve a much more complex optimization problem.
- The third paragraph of section 2 claims that this paper presents a novel energy minimization framework to solve problems of the general form of eq. (2). However, this is not what the authors solve at the end. They solve a different problem that has been subject to at least two relaxations. It is not clear how solving for a local optima of this double relaxed problem is related to the original problem they want to solve.
- The paper says that Geiger et al defined non linearities on a latent space of pre-defined dimensionality. This is wrong. This paper discovers the dimensionality of the latent space by means of a regularizer that encourages the singular values to be sparse. Thus, it does not have a fixed dimensionality, the latent space is just bounded to be smaller or equal than the dimensionality of the original space.
It is not clear to me why the author say for LVMs such as GPLVM that "the latent space is learned a priority with clean training data". One can use different noise models within the GP framework. Furthermore, the proposed approach assumes Gaussian noise (see eq. 6), which is also the trivial case for GP-based LVMs.
It is not clear what the authors mean in the paper by "pre-training" or saying that techniques do not have a training phase. KPCA is trained via a closed-form update, but there is still training.