summaries: Authors propose a new method for adaptation in a few-shot learning setting. Their method comprises two different steps; first they propose a new metric-softmax loss, which aims at improving the transferability of features pre-trained on base data to novel data. They achieve this via redefining the probability score calculating function, which in practice means they replace the exponent term found in the softmax loss with a Gaussian kernel-based radial basis function. This first step improves the feature learning process at large scale but does not solve the problems found when trying to fit arbitrary novel classes. At this point comes step two, where a fast task adaptation process is proposed, i.e. Task-Adaptive Transformation based on affine transformation. This method converges fast and is learnt from the support set, vs step 1 which is trained on the training/base set. Post-training the affine transformation is applied to both support and query image sets.

summaries: This paper deals with few-shot learning from a metric-learning perspective. The authors propose replacing the softmax loss, i.e. softmax + cross-entropy loss, with a so-called "metric-softmax" loss which imitates a Guassian kernel RBF over class templates/weights. This loss is used in both stages of training on base and on novel classes and the authors argue that it helps learning more discriminative feature while preserving consistency between train and test time.

summaries: This paper develops a new few-shot image classification algorithm. It has two main contributions. The first one is to use a metric-softmax loss used to train on the meta-training dataset without episodic updates. The second is that the features learnt thereby are further modified using a linear transformation to fit the few-shot training data and the metric soft-max loss is again used for classifying the query samples. The authors provide experimental results for 5-way-1-shot and 5-way-5-shot testing on mini-Imagenet and CUB-200-2011 datasets.

abstractText: Few-shot classification is a challenging task due to the scarcity of training examples for each class. The key lies in generalization of prior knowledge learned from large-scale base classes and fast adaptation of the classifier to novel classes. In this paper, we introduce a two-stage framework. In the first stage, we attempt to learn task-agnostic feature on base data with a novel Metric-Softmax loss. The Metric-Softmax loss is trained against the whole label set and learns more discriminative feature than episodic training. Besides, the Metric-Softmax classifier can be applied to base and novel classes in a consistent manner, which is critical for the generalizability of the learned feature. In the second stage, we design a task-adaptive transformation which adapts the classifier to each few-shot setting very fast within a few tuning epochs. Compared with existing fine-tuning scheme, the scarce examples of novel classes are exploited more effectively. Experiments show that our approach outperforms current state-of-the-arts by a large margin on the commonly used mini-ImageNet and CUB-200-2011 benchmarks.

authors:
Qi Cai
Yingwei Pan
Ting Yao
Chenggang Yan
Tao Mei

title: Memory matching networks for one-shot image recognition

venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Wei-Yu Chen
Yen-Cheng Liu
Zsolt Kira
Yu-Chiang Frank Wang
Jia-Bin Huang

title: A closer look at few-shot classification

venue: In ICLR 2019 : 7th International Conference on Learning Representations,

authors:
Jun Deng
Wenjie Dong
Richard Socher
Li-Jia Li
Kehui Li
Li Fei-Fei

title: Imagenet: A largescale hierarchical image database

authors:
Arnout Devos
Matthias Grossglauser

title: Subspace networks for few-shot classification

venue: arXiv preprint arXiv:1905.13613,

authors:
Chelsea Finn
Pieter Abbeel
Sergey Levine

title: Model-agnostic meta-learning for fast adaptation of deep networks

venue: Proceedings of the 34th International Conference on Machine Learning - Volume

title: Fast r-cnn

authors:
Nathan Hilliard
Lawrence Phillips
Scott Howland
Artëm Yankov
Courtney D. Corley
Nathan O. Hodas

title: Few-shot learning with metric-agnostic conditional embeddings

venue: arXiv preprint arXiv:1802.04376,

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural Computation,

authors:
Huaxi Huang
Junjie Zhang
Jian Zhang
Qiang Wu
Jingsong Xu

title: Compare more nuanced: Pairwise alignment bilinear network for few-shot fine-grained learning

venue: IEEE International Conference on Multimedia and Expo (ICME),

authors:
Diederik P. Kingma
Jimmy Lei Ba

title: Adam: A method for stochastic optimization

venue: In ICLR 2015 : International Conference on Learning Representations

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E. Hinton

title: Imagenet classification with deep convolutional neural networks. neural information processing

authors:
Kwonjoon Lee
Subhransu Maji
Avinash Ravichandran
Stefano Soatto

title: Meta-learning with differentiable convex optimization

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Wenbin Li
Lei Wang
Jinglin Xu
Jing Huo
Yang Gao
Jiebo Luo

title: Revisiting local descriptor based image-to-class measure for few-shot learning

authors:
Jonathan Long
Evan Shelhamer
Trevor Darrell

title: Fully convolutional networks for semantic segmentation

venue: In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Nikhil Mishra
Mostafa Rohaninejad
Xi Chen
Pieter Abbeel

title: A simple neural attentive metalearner

venue: In ICLR 2018 : International Conference on Learning Representations

authors:
Tsendsuren Munkhdalai
Xingdi Yuan
Soroush Mehri
Adam Trischler

title: Rapid adaptation with conditionally shifted neurons

venue: In ICML 2018: Thirty-fifth International Conference on Machine Learning,

authors:
Boris N. Oreshkin
Pau Rodrı́guez López
Alexandre Lacoste

title: Tadam: Task dependent adaptive metric for improved few-shot learning

venue: In NIPS 2018: The 32nd Annual Conference on Neural Information Processing Systems,

authors:
Adam Paszke
Sam Gross
Soumith Chintala
Gregory Chanan
Edward Yang
Zachary DeVito
Zeming Lin
Alban Desmaison
Luca Antiga
Adam Lerer

title: Automatic differentiation in PyTorch

venue: In NIPS Autodiff Workshop,

authors:
Siyuan Qiao
Chenxi Liu
Wei Shen
Alan L. Yuille

title: Few-shot image recognition by predicting parameters from activations

authors:
Sachin Ravi
Hugo Larochelle

title: Optimization as a model for few-shot learning

venue: In ICLR 2017 : International Conference on Learning Representations

authors:
Avinash Ravichandran
Rahul Bhotika
Stefano Soatto

title: Few-shot learning with embedded class models and shot-free meta training

venue: In The IEEE International Conference on Computer Vision (ICCV),

authors:
Adam Santoro
Sergey Bartunov
Matthew Botvinick
Daan Wierstra
Timothy P. Lillicrap

title: Meta-learning with memory-augmented neural networks

venue: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume

authors:
Victor Garcia Satorras
Joan Bruna Estrach

title: Few-shot learning with graph neural networks

authors:
Chenyang Si
Wentao Chen
Wei Wang
Liang Wang
Tieniu Tan

title: Progressive cluster purification for transductive few-shot learning

authors:
Jake Snell
Kevin Swersky
Richard S. Zemel

title: Prototypical networks for few-shot learning

venue: In Advances in Neural Information Processing Systems,

authors:
Qianru Sun
Yaoyao Liu
Tat-Seng Chua
Bernt Schiele

title: Meta-transfer learning for few-shot learning

authors:
Flood Sung
Yongxin Yang
Li Zhang
Tao Xiang
Philip H.S. Torr
Timothy M. Hospedales

title: Learning to compare: Relation network for few-shot learning

authors:
Oriol Vinyals
Charles Blundell
Timothy P. Lillicrap
Koray Kavukcuoglu
Daan Wierstra

title: Matching networks for one shot learning

venue: Proceedings of the 30th International Conference on Neural Information Processing Systems,

authors:
Catherine Wah
Steve Branson
Peter Welinder
Pietro Perona
Serge J. Belongie

title: The caltechucsd birds-200-2011 dataset

venue: Advances in Water Resources,

authors:
Yandong Wen
Kaipeng Zhang
Zhifeng Li
Yu Qiao

title: A discriminative feature learning approach for deep face recognition

venue: In European Conference on Computer Vision,

authors:
Ruixiang Zhang
Tong Che
Zoubin Ghahramani
Yoshua Bengio
Yangqiu Song

title: Metagan: An adversarial approach to few-shot learning

authors:
Xuan Zhang
Hao Luo
Xing Fan
Weilai Xiang
Yixiao Sun
Qiqi Xiao
Wei Jiang
Chi Zhang
Jian Sun

title: Alignedreid: Surpassing human-level performance in person re-identification

venue: arXiv preprint arXiv:1711.08184,

heading: 1 INTRODUCTION

text:
In recent years, deep learning models have achieved great success in many visual tasks such as image classification (Krizhevsky et al., 2012), object detection (Girshick, 2015) and semantic segmentation (Long et al., 2015). In general, training a deep network needs massively labeled instances which leads to expensive manual annotation cost. In contrast, humans can learn novel concepts from only one or a few examples. Analogously, few-shot learning aims to recognize unseen instances by accessing just a small number of labeled images in each class. Unfortunately, naive methods such as re-training or fine-tuning the model on novel data would severely suffer from overfitting and provide poor result (Finn et al., 2017). The major difficulty lies in the exceedingly limited data, which can hardly represent the class distribution.
In the task of few-shot classification, we are given three datasets, namely training set, support set and query set. The training set (also known as base dataset) normally contains large-scale labeled data for learning of prior knowledge. The support set and query set contain a small number of examples which are drawn from unseen classes disjoint with the training set. We are expected to classify the query images, leveraging the prior knowledge from the base training set and the limited cue from the support images. If the support set contains k annotated images for each of m unique classes, then this task is called m-way k-shot classification.
Previous works on few-shot learning can be roughly divided into two categories, namely metalearning based and metric learning based. The former typically learns a meta-learner model on base data and generalizes it to novel unseen data. Usually, a recurrent neural network (Santoro et al., 2016) or long short-term memory network (Hochreiter & Schmidhuber, 1997) is utilized to learn a memory network to store knowledge. The latter attempts to learn a feature embedding where samples of the same class are closer to each other than samples of different classes. To make training on base data and inference on novel data consistent, a so-called episodic training strategy is widely employed (Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018). There are two limitations in metric learning based approaches. One is that episodic training only considers local sample similarity in current data batch. As shown in Snell et al. (2017), the limited number of classes pertraining episode is harmful for discriminative feature learning, while simply increasing it leads to better performance. However, in a sampled episode, a large number of classes is infeasible due to limited GPU memory. The other limitation is that most existing methods only utilize the labeled
support images in novel classes as templates for feature similarity comparison. We argue that the support images could be better exploited to improve the performance.
In this work, we introduce a two-stage framework named Fast Task Adaption (FTA). As illustrated in Figure 1, the few-shot classification task is decomposed into two steps, i.e. task-agnostic feature learning on base data and adaptation of the classifier to each few-shot learning task. To learn more general and discriminative feature on base data, we propose a novel Metric-Softmax loss, which is an improvement of the Softmax loss. Specifically, we replace the score computing in softmax with a Gaussian kernel-based radial basis function. The Metric-Softmax loss is trained against the full base classes, overcoming the limitation of small number of classes in episodic training. Moreover, the Metric-Softmax classifier can be applied to base and novel classes in the same way. The consistency between training and inference is well preserved. After task-agnostic feature is learned, we design a novel task-adaptive transformation to adapt the classifier to current few-shot learning task. It is essentially an affine transformation applied to features of novel images. Notably, compared with fine-tuning re-initialized weights, it exhibits the desirable characteristics of fast convergence and superior performance. In this way, the aforementioned weaknesses of existing metric learning based methods are well addressed, leading to significant accuracy improvement. Experimental results on the mini-ImageNet and CUB-200-2011 datasets show that our approach consistently outperforms current state-of-the-art methods with respect to various backbone networks.
The main contributions of this paper can be summarized as follows.
• We propose the Metric-Softmax loss, which learns highly discriminative feature and ensures the consistency between training and inference as well.
• We design a novel task-adaptive transformation to adapt the classifier to novel classes efficiently and effectively.
• We improve few-shot classification significantly and achieve state-of-the-art performance on two common few-shot learning benchmarks.

heading: 2 RELATED WORK

text:
In this section, we briefly review the previous approaches relevant to ours and current state-of-the-art methods we compare with in experiments.
Meta-learning based methods aim to learn task-agnostic knowledge from a sequence of similar tasks which have enough training data and apply it to new tasks for fast adaption. Ravi & Larochelle
(2017) utilize an LSTM-based network to control the parameter optimization process, which results in better generalization than SGD when training on a few labeled samples. Qiao et al. (2018) propose to adapt a pre-trained network to novel classes by directly predicting the parameters from the activations. MAML (Finn et al., 2017) learns sensitive and general initial parameters, which can fast adapt to a novel task with only one or a few steps of gradient-descent update. MTL (Sun et al., 2019) learns to adapt a deep neural network for few-shot learning tasks by learning scaling and shifting functions of weights for each task. Cai et al. (2018) use a contextual learner to predict the parameters of an embedding network for unlabeled data by using memory slots. Mishra et al. (2018) propose a meta-learner architecture that combines temporal convolutions and soft attention.
Metric learning based methods aim to learn a feature embedding that preserves the class neighborhood structure. Specifically, samples of the same class are closer than samples of different classes in the learned embedding space. Matching networks (Vinyals et al., 2016) combine attention and memory together by learning a network that maps a small labeled support set and an unlabeled example to its label. They also first introduce the episodic training strategy, where the training process mimics the test scenario based on support-query metric learning. Prototypical network (Snell et al., 2017) uses the mean feature of each class as its corresponding prototype representation to learn a similarity metric space. Relation network (Sung et al., 2018) learns a deep non-linear distance metric by considering the relation between query images and support images. Satorras & Estrach (2018) propose a graph neural network to learn the feature similarity between query and support set. Our approach belongs to this category. Compared with existing methods, our major improvements are the Metric-Softmax loss for generic feature learning and the task-adaptive transformation for fast task adaptation.
Other methods. Zhang et al. (2018) propose a GAN-based approach to help few-shot classifiers to learn sharper decision boundary. Si et al. (2019) propose a progressive cluster purification method for transductive few-shot learning. Li et al. (2019) propose a deep nearest neighbor neural network for few-shot learning. Chen et al. (2019) give a consistent comparative analysis of several representative few-shot classification algorithms and show that the depth of backbone network matters. Besides, they attempt to fine-tune a Softmax classifier on the support set. On the contrary, we first apply an affine transformation to features of the support images, and compute the class centroids of the transformed features as the classfier’s weights. Concretely, we learn the affine transforming matrix rather than directly fine-tune the randomly initialized weights. Optimizing the transformation is computationally more stable even when one example per-class is available.

heading: 3 METHOD

text:
As shown in Figure 1, the proposed FTA framework is composed of two stages, namely task-agnostic feature learning stage and fast task adaptation stage. In the first stage, we train the network on the base data with the proposed Metric-Softmax loss. After that, we fix the parameters of the network backbone and use it as a feature extractor. In the second stage, we learn a task-adaptive classifier, which is also a Metric-Softmax classifier with its weights replaced with affine transformed features of the support images. The affine transformation g is named Task-Adaptive Transformation (TAT), which is learned on the scarce support set. With proper initialization, training of g converges very fast within a few epochs. After training finishes, g is applied to both support and query images.
In this section, we first elaborately describe the proposed Metric-Softmax loss. Since it is an improvement of the Softmax loss, we first give a brief review of the Softmax loss and reveal the issue of discrepancy between training and inference. Then we explain how to address this issue with the Metric-Softmax loss. Finally, we introduce the task-adaptive transformation for fast task adaptation.

heading: 3.1 SOFTMAX CLASSIFIER

text:
Feature learning with the Softmax loss has achieved great success in many feature ranking tasks like person re-identification (Zhang et al., 2017) and face verification (Wen et al., 2016). Given an image X of base classes, a backbone network f is applied to project X into feature space h = f(X),h ∈ Rd. Then, the logits z are computed by feeding h into an affine transformation:
z = W Th + b, (1)
where W ∈ Rd×c and b ∈ Rc are the transforming weights. c is the number of classes in base data. The Softmax classifier predicts the probability score that the image belongs to class i by
si = softmax(z)i = exp(zi)∑c j exp(zj)
(2)
The Softmax loss is an assembly of the Softmax classifier and cross-entropy loss. Suppose the one-hot category label of the image is y ∈ {0, 1}c, the training cross-entropy loss is defined as
L = − c∑ j yj log(sj) (3)
During inference, if a test image belongs to one of the training classes, its label can be predicted by
ŷ = arg max i si (4)
In this case, the scores in both training and inference are computed by an affine transformation and softmax function applied to the feature vector h. However, in the context of few-shot learning, the query image for testing belongs to a novel class unseen in training. A common way to classify the novel query image is to compare its similarities to the few labeled support images, which share the same label space with the query image. Concretely, for m-way k-shot classification, we can extract features of the support images, and compute the mean feature for each class by
h̄i = 1
k k∑ j f(Xij) (5)
Here, Xij denotes the j-th image of class i in the support set. The label of a query image X can be predicted by
ŷ = arg min i ||f(X)− h̄i|| (6)
Rather than an inner-product between h and column vectors of the weight matrix (zi = W T:,ih+bi) followed by softmax function in the previous case, the prediction is based on the euclidean distance between the query and support features. We argue that this discrepancy hurts the transferability of features pre-trained on base data to novel data.

heading: 3.2 METRIC-SOFTMAX CLASSIFIER

text:
To eliminate this discrepancy, we improve the Softmax classifier by redefining the probability score calculating function. Specifically, we replace the exp(zi) term in Equation 2 with a Gaussian kernel-based radial basis function, whose center is the column vector of a learnable weight matrix W:,i,W ∈ Rd×c. Then the score can be computed by
si = exp(−α||h−W:,i||)∑c j exp(−α||h−W:,j ||) , (7)
where α ∈ R, α > 0 is a hyper-parameter for scaling and h is an L2-normalized feature vector. In this way, it can be easily derived that arg max
i si is equal to arg min i ||h −W:,i||. Here W:,i can
be interpreted as the centroid of class i in the learned embedding space. That means the training procedure is essentially optimizing the euclidean distance-based similarity between images to their corresponding class centroids. During training, the cross-entropy loss is adopted as in the Softmax loss.
During inference, we replace W:,i with the mean feature (see Equation 5) of class i in the support set. Then classification of a query image is conducted according to Equation 4. Like episodic training, the consistency between training and inference is well preserved. And we easily enjoy the benefit of more general and discriminative feature learned with the cross-entropy loss. It is worth noting that the number of classes during inference on novel data is independent from the base dataset, which makes it flexible enough to adapt to different few-shot classification tasks.

heading: 3.3 FAST TASK ADAPTATION

text:
Although the feature learned on the large-scale base data is highly general and discriminative, it is unlikely to perfectly fit arbitrary novel classes. A proper adaptation to each few-shot task is beneficial. To leverage the valuable cue provided by the support images, a straightforward choice is to fine-tune a specific Softmax classifier as in Chen et al. (2019). However, even with the feature extractor being fixed, it is still difficult to train the classifier due to easy overfitting on small dataset. In contrast, we design a parameterized transformation function g to adapt the general feature to current few-shot learning task,
h′ = g(h) (8)
Concretely, g is simply a zero-offset affine transformation: g(h) = MTh,M ∈ Rd×d. For mway k-shot classification, we reconstruct an m-class Metric-Softmax classifier whose weight matrix W̄ ∈ Rd×m is derived from the transformed features of the support images
W̄:,i = 1
k k∑ j g(h(i,j)) (9)
Here h(i,j) denotes the feature of j-th image in class i. It is worth mentioning that both the transformed feature vector g(h) and each column of the weight matrix W̄ should be L2-normalized. And now, the computation of probability score is defined as
si = exp(−α||g(h)− W̄:,i||)∑c j exp(−α||g(h)− W̄:,j ||)
(10)
The transformation g is trained on all labeled images in the support set with the cross-entropy loss. To ease optimization, the transforming matrix M is initialized with an identity matrix. After training of g finishes, it is applied to both support and query images. And the final prediction of a query image X is computed by
ŷ = arg min i ||g(f(X))− W̄:,i|| (11)

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS

text:
We conduct few-shot classification experiments on two common benchmarks, namely miniImageNet (Vinyals et al., 2016) and CUB-200-2011 (Wah et al., 2011).
mini-ImageNet is first proposed by Vinyals et al. (2016) for few-shot classification evaluation. It consists of 100 classes and 600 images per class, which is a subset sampled from the ImageNet dataset (Deng et al., 2009). In our experiments, we follow the dataset splits proposed in Ravi & Larochelle (2017), which takes 64, 16 and 20 classes for training, validation and testing respectively.
CUB-200-2011 is initially proposed for fine-grained classification of birds. It contains 200 categories and 11,788 images in total. We follow the evaluation protocol used in Hilliard et al. (2018), which randomly splits the dataset into 100, 50 and 50 classes for training, validation and testing.

heading: 4.2 IMPLEMENTATION DETAILS

text:
To make a fair comparison with existing works, we evaluate our method for three commonly used backbone networks, namely Conv-4 (Vinyals et al., 2016), ResNet-10 (Chen et al., 2019) and ResNet-12 (Oreshkin et al., 2018). Besides difference in depth and architecture, Conv-4 and ResNet12 expect an input size of 84 × 84, while ResNet-10 takes 224 × 224 images as input, following the setting in existing works. In our experiments, we mainly focus on the 5-way 1-shot and 5-way 5-shot classification settings which are the most commonly used in existing works. The test episode contains 5 classes and each class contains 1 (or 5) support image(s) and 15 query images. For all experiments, we report the mean accuracy over 1200 randomly sampled episodes and the 95% confidence interval.
Our implementation is based on PyTorch (Paszke et al., 2017). We apply the same data augmentation as Chen et al. (2019), including random cropping, horizontal flipping and color jittering in both feature learning and fast task adaptation stages. The Adam (Kingma & Ba, 2015) optimizer with = 10−3, β1 = 0.9 and β2 = 0.999 is used. In the feature learning stage, the backbone network is trained from scratch for 300 epochs in total with a batch size of 32. We decay the learning rate by a factor of 0.1 every 75 epochs. The scaling factor α used in Metric-Softmax is set to 15 for mini-ImageNet and 1 for CUB-200-2011. In the task adaptation stage, we use the same optimizer with different learning rate. We set the learning rate to 0.005 for the 1-shot setting and 0.05 for the 5-shot setting. α is set to 0.25 and 2 for the 1-shot and 5-shot settings respectively. We fine-tune the weights of the transformation g for 20 epochs. Note that all the hyper-parameters are determined by the performance on the validation set of each dataset.

heading: 4.3 COMPARISONS WITH THE STATE-OF-THE-ARTS

text:
Results on mini-ImageNet. Table 1 shows a comparison of our method to current state-of-the-arts on the mini-ImageNet dataset. Classification accuracies are reported for two backbone networks, i.e. Conv-4 and ResNet-12. For Conv-4 backbone, we achieve the best accuracy in both 5-way 1-shot and 5-way 5-shot settings. Our method boosts the performance for the 5-shot setting, surpassing the second best by 9.33% and 2.10% for the Conv-4 and ResNet-12 backbones respectively. A closer analysis on the larger improvement for the 5-shot setting than 1-shot setting is given in the discussion section.
Results on CUB-200-2011. On this dataset, we find that there is no work reporting the few-shot performance for the ResNet-12 backbone. For a fair comparison, we report the accuracies of the Conv-4 and ResNet-10 backbones. As shown in Table 2, similar to mini-ImageNet, our method outperforms the second best method by a large margin (87.92% vs. 81.90% for Conv-4 and 92.89% vs. 87.45% for ResNet-10) in the 5-way 5-shot setting. In the 1-shot setting, we achieve the best performance for ResNet-10 and comparable accuracy to PABN (65.11% vs. 66.71%) for Conv-4. It is worth mentioning that in PABN, the max-pooling layers of the backbone network are replaced with bilinear pooling, which is specially optimized for fine-grained classification tasks. We expect further accuracy improvement by incorporating such techniques in our method.
Results under Domain Shift. To evaluate the generalizability of the task-agnostic feature learned with the proposed Metric-Softmax loss, we perform the cross domain experiments. In the cross
domain scenario, we swap the training sets of the two datasets. Specifically, in the miniImageNet→CUB-200-2011 setting, the training set is comprised of the 64 base classes from miniImageNet, while the validation and test data come from CUB-200-2011. Similarly, in the CUB200-2011→mini-ImageNet setting, we use the 100-class base data of CUB-200-2011 for general feature learning and evaluate its performance on the mini-ImageNet dataset. We report the results under the same setting as Devos & Grossglauser (2019) in Table 3. Compared with existing methods, our approach achieves the highest accuracy in all settings. Notably, we improve SubspaceNet by 22.67% in the mini-ImageNet→CUB-200-2011 setting. The strong cross domain performance is mainly attributed to the task-adaptive transformation g of our FTA. If it is disabled, the accuracy drops drastically from 85.38% to 67.22%. Another observation is that the performance degrades more in CUB-200-2011→mini-ImageNet than mini-ImageNet→CUB-200-2011. This is because mini-ImageNet contains more abundant classes than CUB-200-2011 and the network can learn more general and discriminative feature.

heading: 5 DISCUSSION

text:
Ablation Study of FTA. As shown in Table 4, compared with the Softmax classifier, the proposed Metric-Softmax classifier improves the accuracy by 4.77% and 3.41% in the 5-way 1-shot and 5- way 5-shot settings respectively. This improvement is attributed to elimination of the discrepancy between training and inference brought by our Metric-Softmax classifier. By incorporating the taskadaptive transformation, the performance is further boosted, especially in the 5-shot setting.
TAT vs. Fine-tuning. To leverage the cue from the supported set, an alternative method is to fine-tune a re-initialized Metric-Softmax classifier similar to Chen et al. (2019). Figure 2 shows a comparison of the proposed task-adaptive transformation to the alternative fine-tuning scheme. In terms of convergence speed, TAT takes only 5 epochs to get an appreciable performance. However,
direct fine-tuning takes over 40 epochs to converge for the 5-way 5-shot setting. For the 5-way 1- shot setting, it barely improves the accuracy. In terms of final performance, TAT is far more superior over direct fine-tuning.
Why does FTA improve the 5-shot setting more than 1-shot setting? As shown in Section 4.3, our FTA outperforms existing methods more significantly in the 5-shot setting than 1-shot setting. It can be explained by the mechanism of the task-adaptive transformation. In the 1-shot setting, only one image is available in each class. Then Equation 9 degenerates to W̄:,i = g(h(i,1)). For an image with label y, y ∈ {1, 2, 3, 4, 5}, the numerator of Equation 10 becomes a constant: exp(−α||g(h(y,1)) − g(h(y,1))||) = 1. Now minimizing the cross-entropy means minimizing the denominator of Equation 10, which is equivalent to maximizing the total distance of this feature to others: ∑5 j ||g(h(y,1)) − g(h(j,1))||. As illustrated in Figure 3(a), task-adaptive transformation only enlarges the distance among different classes in the 1-shot setting. However, for the 5-shot setting (see Figure 3(b)), the transformation not only enlarges inter-class distance but also shrinks the intra-class distance, which helps to improve the classifier.

heading: 6 CONCLUSIONS

text: In this paper, we propose the Metric-Softmax loss which can adequately explore the feature similarities in the whole base dataset with the premise of keeping the classifier consistent between training and inference. In addition, we take full advantage of the few labeled novel data by introducing a parameterized transformation to adapt the learned general feature to each few-shot classification task. Our approach is particularly helpful for the 5-shot setting. We leave further improvement for the 1-shot setting as our future work.


summaries: The authors aim at improving the accuracy of numerical solvers (e.g. for simulations of partial differential equations) by training a neural network on simulated reference data. The neural network is used to correct the numerical solver. For different tasks they set up an approximation scheme via minimizing a square loss plus a task specific regularization (e.g. volume preservation in the Navier-Stokes equation example). This is then trained in a supervised manner. They also explore an unsupervised version by back-progagating through a differentiable numberical solver.

summaries: The paper proposes learning NN to correct for inaccuracies in numerical solvers of PDEs, with experimental focus on fluid flow simulation. It lists two approaches: (1) compute correction in high resolution simulation from reference states, convert to low-resolution correction, and train NN to predict low-res correction (optionally with temporal regularization, and (2) directly simulate forward using correction prediction and differentiable PDE solver and optimize to match the given reference states. It shows empirical results on better approximating fluid flow simulation.

summaries: Numerical solvers for partial differential equations take a lot of time to get high resolution results since they have to explore high dimensional grid in function domain. Thus, it is important to interpolate between grids to get high resolution results. In this paper, the authors propose the model that assists PDE solver by correcting residuals in a data-driven way. Specifically, they try to approximate NN to correction function in supervised and unsupervised manners. They also propose a temporal regularization method that smooths behavior of fluid between times. As a result, proposed method can generate high resolution results in efficient way with smoke rising simulation dataset.

abstractText: Improving the accuracy of numerical methods remains a central challenge in many disciplines and is especially important for nonlinear simulation problems. A representative example of such problems is fluid flow, which has been thoroughly studied to arrive at efficient simulations of complex flow phenomena. This paper presents a data-driven approach that learns to improve the accuracy of numerical solvers. The proposed method utilizes an advanced numerical scheme with a fine simulation resolution to acquire reference data. We, then, employ a neural network that infers a correction to move a coarse thus quickly obtainable result closer to the reference data. We provide insights into the targeted learning problem with different learning approaches: fully supervised learning methods with a naive and an optimized data acquisition as well as an unsupervised learning method with a differentiable Navier-Stokes solver. While our approach is very general and applicable to arbitrary partial differential equation models, we specifically highlight gains in accuracy for fluid flow simulations.

authors:
Yohai Bar-Sinai
Stephan Hoyer
Jason Hickey
Michael P Brenner

title: Data-driven discretization: a method for systematic coarse graining of partial differential equations

authors:
Andrea D. Beck
David G. Flad
Claus-Dieter Munz

title: Deep neural networks for datadriven turbulence models

authors:
J.U. Brackbill
D.B. Kothe
H.M. Ruppel

title: FLIP: A low-dissipation, particle-in-cell method for fluid flow

venue: Computer Physics Communications,

authors:
Steven L Brunton
Joshua L Proctor
J Nathan Kutz

title: Discovering governing equations from data by sparse identification of nonlinear dynamical systems

venue: Proceedings of the National Academy of Sciences,

authors:
Mengyu Chu
Nils Thuerey

title: Data-driven synthesis of smoke flows with CNN-based feature descriptors

venue: ACM Trans. Graph.,

authors:
Cathal Cummins
Madeleine Seale
Alice Macente
Daniele Certini
Enrico Mastropaolo
Ignazio Maria Viola
Naomi Nakayama

title: A separated vortex ring underlies the flight of the dandelion

authors:
Karthikeyan Duraisamy
Ze J. Zhang
Anand Pratap Singh

title: New Approaches in Turbulence and Transition Modeling Using Data-driven Techniques

authors: deep learning techniques for multiphase flows. Journal of Computational Physics

title: ISSN 0021-9991

title: tutorial: Generative adversarial networks

authors:
Xiaoxiao Guo
Wei Li
Francesco Iorio

title: Convolutional neural networks for steady flow approximation

venue: In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Byungsoo Kim
Vinicius C. Azevedo
Nils Thuerey
Theodore Kim
Markus Gross
Barbara Solenthaler

title: Deep fluids: A generative network for parameterized fluid simulations

venue: Computer Graphics Forum,

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: Imagenet classification with deep convolutional neural networks

venue: Advances in Neural Information Processing Systems

title: Deep learning in fluid dynamics

venue: Journal of Fluid Mechanics,

authors:
Lubor Ladický
SoHyeon Jeong
Barbara Solenthaler
Marc Pollefeys
Markus Gross

title: Datadriven fluid simulations using regression forests

authors:
Elmer Eugene Lewis
Warren F Miller

title: Computational methods of neutron transport

authors:
Zichao Long
Yiping Lu
Xianzhong Ma
Bin Dong

title: PDE-Net: Learning PDEs from data

authors:
Martin Magill
Faisal Qureshi
Hendrick de Haan

title: Neural networks trained to solve differential equations learn general representations

authors:
R. Maulik
O. San

title: A neural network approach for the blind deconvolution of turbulent flows

authors:
Arvind Mohan
Don Daniel
Michael Chertkov
Daniel Livescu

title: Compressed convolutional LSTM: An efficient deep learning framework to model high fidelity 3d turbulence, 2019

authors:
Jeremy Morton
Antony Jameson
Mykel J Kochenderfer
Freddie Witherden

title: Deep dynamical modeling and control of unsteady fluid flows

authors:
James M Murphy
David MH Sexton
David N Barnett
Gareth S Jones
Mark J Webb
Matthew Collins
David A Stainforth

title: Quantification of modelling uncertainties in a large ensemble of climate change

authors:
Lukas Prantl
Boris Bonev
Nils Thuerey

title: Generating liquid simulations with deformationaware neural networks

venue: In International Conference on Learning Representations,

authors:
David A Randall
Richard A Wood
Sandrine Bony
Robert Colman
Thierry Fichefet
John Fyfe
Vladimir Kattsov
Andrew Pitman
Jagadish Shukla
Jayaraman Srinivasan

title: Climate models and their evaluation

venue: In Report of the IPCC (FAR),

authors:
Deep Ray
Jan S. Hesthaven

title: An artificial neural network as a troubled-cell indicator

venue: Journal of Computational Physics,

authors:
Samuel H. Rudy
Steven L. Brunton
Joshua L. Proctor
J. Nathan Kutz

title: Data-driven discovery of partial differential equations

venue: Science Advances,

title: Learning partial differential equations via data discovery and sparse optimization

venue: Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences,

authors:
Andrew Selle
Ronald Fedkiw
ByungMoon Kim
Yingjie Liu
Jarek Rossignac

title: An unconditionally stable maccormack method

venue: Journal of Scientific Computing,

title: Stable fluids

venue: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques,

title: Finite difference schemes and partial differential equations, volume 88

authors:
Nils Thuerey
Konstantin Weissenow
Lukas Prantl
Xiangyu Hu

title: Deep learning methods for reynolds-averaged navier-stokes simulations of airfoil flows, 2018

authors:
Jonathan Tompson
Kristofer Schlachter
Pablo Sprechmann
Ken Perlin

title: Accelerating eulerian fluid simulation with convolutional networks

venue: In Proceedings of Machine Learning Research,

authors:
Kiwon Um
Xiangyu Hu
Nils Thuerey

title: Liquid splash modeling with neural networks

authors:
Siddhartha Verma
Guido Novati
Petros Koumoutsakos

title: Efficient collective swimming by harnessing vortices through deep reinforcement learning

authors:
You Xie
Erik Franz
Mengyu Chu
Nils

title: Thuerey. tempoGAN: A temporally coherent, volumetric gan for super-resolution fluid flow

authors:
Jonas Zehnder
Rahul Narain
Bernhard Thomaszewski

title: An advection-reflection solver for detail-preserving fluid simulation

authors:
Yao Zhang
Woong Je Sung
Dimitri N. Mavris

title: Application of convolutional neural network to predict airfoil lift coefficient. 2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Jan 2018

heading: 1 INTRODUCTION

text:
Numerical methods are a central component of many disciplines and widely used for solving a variety of linear and nonlinear problems. One of the long-standing targets is fluid flow, which is renowned for its great diversity and complexity in terms of dynamics. Studies in computational fluid dynamics have focused on numerical simulations for such problems and invested huge efforts in solving spatio-temporal partial differential equations (PDEs) such as the Navier-Stokes equations, which represent the well-established physical model for fluids.
Traditional methods typically improve accuracy with fine discretizations both in space and time. While the methods and computing power for numerical simulation have seen advances in recent years, there is still a pressing need for better efficiency and accuracy. For most practical applications of computer simulations, we are still far away from fully resolving all necessary scales of nature around us (Verma et al., 2018; Cummins et al., 2018).
To tackle this problem, we propose a data-driven approach that “assists” a given numerical method to improve its accuracy. To this end, we introduce a first learning-based approach that puts special emphasis on the time dimension. We demonstrate two variants to achieve this goal in the context of fluids: a supervised version with an optimization algorithm for acquisition of temporally constrained correction data and an unsupervised version with a differentiable PDE solver that allows us to autonomously takes into account temporal information when training. We compare advantages and disadvantages of both approaches, and our experiments show that, using our trained models, the simulation accuracy of the given solver can be significantly improved. In all cases, our trained models yield improved dynamics, and the learned assistance function lets a coarse simulation reproduce the behavior of the reference data more closely. In particular, we demonstrate the improvements of our approach over ad-hoc learning approaches.

heading: 2 RELATED WORK

text:
Data-driven approaches were shown to be highly effective for inferring unknown PDEs and coefficients of corresponding terms (Brunton et al., 2016). Their capabilities were investigated for different problems, among others the Navier-Stokes equations (Rudy et al., 2017; Schaeffer, 2017). Deep
learning methods were successfully applied for learning stencils of advection diffusion problems (Bar-Sinai et al., 2018), to discover PDE formulations (Long et al., 2017), and to analyze families of Poisson equations (Magill et al., 2018). While identifying governing equations represents an interesting and challenging task, we instead focus on a general method to improve the solutions of chosen solution spaces.
Due to the complexity and nonlinear nature of fluid dynamics, they remain a challenging problem, and accordingly data-driven approaches including deep learning have received a lot of attention to reach an efficient solution for such problems (Kutz, 2017). Particularly, a turbulence modeling has been a representative objective in this context (Duraisamy et al., 2015; Maulik & San, 2017; Beck et al., 2018; Mohan et al., 2019). Moreover, both steady (Guo et al., 2016) and unsteady (Morton et al., 2018) flows as well as multiphase (Gibou et al., 2018) flows have been investigated with deep learning based approaches. Additionally, the convolutional neural network (CNN) based methods were studied for airfoil flow problems (Thuerey et al., 2018; Zhang et al., 2018).
For fluid simulation, particularly in the field of computer graphics, data-driven approaches have been considered as an efficient alternative to replace computationally expensive steps of numerical processes (Ladický et al., 2015; Tompson et al., 2017). Moreover, deep learning, particularly with generative adversarial models (Goodfellow, 2016), has been used for efficiently synthesizing details within coarsely resolved simulations (Chu & Thuerey, 2017; Xie et al., 2018). In addition to scalar transport and smoke flows, liquid droplets have been efficiently tackled by learning stochastic models (Um et al., 2018). Due to their capabilities for transforming space-time data into reduced latent spaces, fluid simulations were processed and re-simulated with neural network (NN) models (Kim et al., 2019; Prantl et al., 2019). Motivated by the success of previous studies regarding datadriven approaches in numerical simulations, we aim for enabling the learning of general correction functions. Our aim differs from methods that synthesize details for low-resolution input data. As such, our method is orthogonal to super-resolution methods and shares similarities with methods to identify discontinuities in finite difference solutions (Ray & Hesthaven, 2018). However, we investigate the problem for challenging two-dimensional flow problems and propose a new approach for learning the correction bby taking into account temporal dynamics.

heading: 3 CORRECTING NUMERICAL SIMULATIONS

text:
Numerical methods yield approximations of a smooth function f(x) in a discrete setting with an approximation error of the form O(hk) where h is the step size of the discretization. Naturally, higher order methods with larger k are preferable but difficult to arrive at in practice. In our setting, the motions of fluids can be represented by a series of vector fields v(x) where bold vectors will denote two- or three-dimensional vector-valued functions in the following. The accuracy of the fluid motion likewise depends on the discretization error for v. In the following, we will focus on Cartesian Eulerian representations, meaning axis aligned grids with square cells. Discretizing functions like v on such a grid yields approximations that scale with the grid spacing h. While small h can yield accurate representations, the computational requirements typically increase superlinearly with the number of unknowns, and as such, all practical methods strive for keeping h as large as possible in order to compute solutions quickly.
Due to the nonlinear nature of flow equations and the inherent truncation errors of numerical schemes, varying resolutions often lead to very significant differences in solutions. Such differences are not readily interchangeable across the different resolutions, as especially higher frequency content can be difficult to represent on a coarse grid. In order to improve the accuracy of a numerical method, we propose to learn a correction function that compensates for discretization errors on coarse grids and is representable by a NN. More specifically, we employ a CNN (Krizhevsky et al., 2012) as the correction should be translation invariant for generalization. We target time-dependent problems for which we infer a per time step correction. This is particularly challenging as approximation errors accumulate over time. In this work, we will focus on finite difference approximations (Strikwerda, 2004).
A discretized representation s of a smooth target function f typically consists of a set of discrete physical quantities. Together, we call these discrete quantities s a state with which our correction approach can be formulated as follows:
sn+1 = F (sn) + cn (1)
where F denotes a numerical solver for the target function f moving a state sn forward in time to a new state sn+1. Here, cn denotes the correction function. Note that cn potentially updates only a part of the full state, e.g., the velocity field v.
We target solutions of the Navier-Stokes equations, which for incompressible Newtonian fluids takes the form
∂v ∂t + v · ∇v = −1 ρ ∇p+ ν∇ · ∇v + g subject to ∇ · v = 0, (2)
where ρ, p, ν, and g denote density, pressure, viscosity, and external forces, respectively. The constraint,∇ · v = 0, is particularly important as it restricts motions to the space of divergence-free (i.e., volume preserving) velocities.

heading: 3.1 CORRECTION FUNCTION

text:
The natural follow-up question is how to compute and represent the potentially highly nonlinear correction function of Eq. 1. Thanks to their flexibility and ability to represent nonlinearities, we use fully convolutional networks as our model for the correction function. A fine discretization, optionally with advanced numerical schemes, is employed to compute the reference data. Our goal is, then, to find an optimal approximation of the correction function for the given reference data. We will first approach this problem in a supervised manner with human intervention and then explain an improved approach that enables a NN model to learn the target function in an unsupervised manner.
As the velocity v is the key quantity for fluids, we target corrections c of this function in the following. However, our approach likewise could be applied to other functions. The correction is applied additively, i.e., v+c, and should be computed such that this sum matches the reference velocity vR, i.e., ideally v + c = vR. In practice, we will not aim for an equality but rather match the reference as closely as possible. As the reference is typically best defined in terms of a fine discretization with reduced approximation errors, it might have an altogether different resolution. In this case, care is needed when transferring functions between different discretizations especially when additional constraints such as physical laws of conservation need to be taken into account. We address this problem via an optimization and aim for minimizing the distance between representations on both grids with respect to a suitable metric.
For two different dimensionalities ξ, χ ∈ N with ξ < χ, consider two vector spaces H ∈ Rχ and L ∈ Rξ that both conserve volume (following Eq. 2), i.e., ∇ · h = 0 for ∀h ∈ H, and ∇ · l = 0 for ∀l ∈ L. Having a finer vector field cH , which contains the necessary information, we aim to find the closest vector field cL (∈ L) to cH (∈ H). We first describe how to transfer functions between both spaces in general before explaining how to find those that are particularly amenable for learning. Consider an interpolation operator W that introduces new data points within a vector field cL (∈ L), i.e., WcL ∈ Rχ. We, then, strive to minimize the distance between WcL and cH such that cL can best represent the information of cH without loosing its volume conserving properties. Thus, we aim for computing cL with
argmin cL ||WcL − cH ||2 subject to ∇ · cL = 0. (3)
This represents a constrained optimization problem with equality constraints, which we can solve via Lagrange multipliers λ as follows:
Φ = ||WcL − cH ||2 + (∇ · cL)>λ. (4)
This results in a system of equations as follows:[ W>W −∇ −∇> 0 ] [ cL λ ] = [ W>cH 0 ] . (5)
Using the Schur complement, we can simplify this system to speed up calculations:
∇>(W>W)−1∇λ = ∇>(W>W)−1W>cH , (6) cL = (W >W)−1(W>cH −∇λ). (7)
A NN, then, infers the correction ĉL by minimizing the supervised loss, Lsup = ∑ ||ĉL − cL||, for a pre-computed data set. While this method can efficiently yield divergence-free changes of
resolution, i.e., move a function such as the aforementioned correction from fine to coarse grids, the solutions obtained in this way are not necessarily optimal for deep learning methods as we will evaluate in more detail below. Hence, we first introduce an extension that takes into account the temporal evolution of the correction function.

heading: 3.2 TEMPORAL REGULARIZATION OF CORRECTIONS

text:
The vector fields we target are obtained from a numerical simulation. Here, the underlying flow equations are solved for a finite number of steps from an initial condition. The simulation produces a series of states, which represent the spatio-temporal changes of fluid volume and velocity. Given the setup as described so far, we can acquire the correction of a basic numerical simulation by running the same simulation with a reduced step size in space and time to reduce approximation errors. In addition, we can employ more accurate numerical schemes for the reference as we will demonstrate below. Having both the basic simulation velocity field vB and the reference simulation velocity field vR, we can compute the correction vector field cL via Eq. 7 for cH = vR −WvB . As our aim is to find a learned representation of the corrections, a crucial aspect to consider is the sensitivity (Murphy et al., 2004) of the targeted function (i.e., the correction) with respect to the data at hand, i.e., in our case, the state of a coarse simulation. Here, we observe that the correction vector fields vary strongly in time even for smooth changes of the basic simulation. That means the correction function has a very nonlinear and difficult to learn relationship with the observable data in a simulation.
In order to address this difficulty, here, we introduce supervision in data acquisition by employing a temporal regularization that makes the correction function learnable without sacrificing its correcting properties. As we are dealing with continuous models, we know that the flow motion changes smoothly in time if it is resolved finely enough. As our solution changes in space as well as time, a learned function has to be able to represent spatial as well as temporal changes of the target function correctly. We focus on the inference of per-time step corrections, and thus it is especially important to obtain correction functions that change smoothly over time (spatial regularization could potentially also be incorporated during learning implicitly or explicitly). Consequently, we regularize our correction vector fields such that they change smoothly in time by penalizing temporal change of the correction vector field. These are given by dcL/dt, which we minimize together with the transfer from fine to coarse discretizations:
argmin cL
( ||WcL − cH ||2 + β|| dcL dt ||2 ) subject to ∇ · cL = 0. (8)
Here, β is the regularization coefficient. This yields a new system of equations as follows:[ W>W + β 2∆tI −∇
−∇> 0 ] [ cL λ ] = [ W>cH + β 2 ∆tc n−1 L 0 ] , (9)
where ∆t is the timestep size, I is the identity matrix, and the superscript n − 1 of cL denotes the state at the previous step in time. We used the β = 0 in the following experiments and found it effective for both the correction accuracy and training.

heading: 3.3 UNSUPERVISED LEARNING WITH DIFFERENTIABLE PHYSICS

text:
So far, we have described how to acquire a learnable data set obtained with human support for introducing temporal regularization. The NN can, then, learn the corrections given the low-resolution input simulations, i.e., minimizes the error of inferred corrections. This naturally requires special
care in terms of choosing right amount of temporal regularization, which in practice is strongly problem dependent. To reduce the reliance on human intervention, we propose a training approach with which the model can directly learn the target function by recurrently observing the temporal dynamics of the physical model as illustrated in Fig. 1. The recurrent training can be achieved via differentiable physics solvers that allow for a back-propagation of gradients through the discretized physical model. In this way, we can let a NN learn the sought-after correction function in an unsupervised manner. As we will demonstrate below, the main advantage of this approach is that the NN experiences how corrections influence the evolution of the dynamics and receive gradients in order to improve inference accuracy. In the following, we employ a differentiable Navier-Stokes solver from concurrent work (anonymous, 2020). This solver builds on the automatic differentiation of a machine learning framework to compute analytic derivatives and augments them with custom derivatives where built-in derivatives would be inefficient. This setup allows for straightforward integration of solver functionality with machine learning models and enables end-to-end training in recurrent settings.
Denoting the discretized physical model as Fm, we can define our loss asLun = ∑m i ||(Fm(st0+i)+ cL)−Mst0+iH ||2, where the notation of the NN inference for cL is omitted for brevity, M is a sampling operator to make the spatial discretizations consistent, and m denotes the number of recurrent steps. Above, each loss evaluation starts at time t0. Note that a state st0+i depends on all i− 1 previous states, and thus requires a back-propagation of the gradients through i− 1 solver evaluations. In contrast to the supervised approach, we can directly evaluate the difference between the corrected simulation result, as provided by the unsupervised loss formulation, and the reference state sH . In practice, this can be achieved without explicitly preparing for learnable input-output pairs, which are acquired using our optimization for training. Once sequences of reference simulations are collected, we can directly use the sequences by selecting an initial frame t0, which is downsampled to the basic solver’s resolution and fed to the architecture as an initial input. During each recurrent step, the intermediate results are evaluated via the differentiable solver and the current state of the correction model. These intermediate results are compared with the given consecutive reference frames, and the model can update its weights accordingly. With this setup, the model can directly learn from simulation results that have experienced inferred corrections via the unrolled recurrent steps, each of which incorporates the physics solver.

heading: 4 EXPERIMENTS

text: To acquire our data sets, we generate a set of simulation sequences, which are started from randomized initial setups according to the given parameters. These sequences are used for obtaining pairs of input and correction velocity fields to training our NN. Anonymized and time-stamped supplemental material for our submission can be downloaded at https://www.dropbox.com/sh/ bl2xmrtc2loot14/AADChTpswLYmy7Yq PG1zfXka?dl=0.

heading: 4.1 SUPERVISED CORRECTION OF RISING SMOKE

text:
This example encompasses a volume of hot smoke rising from the bottom of an enclosed container. The motion of the smoke volume is driven by buoyancy forces computed from the density field
using the Boussinesq model. We randomize the initial size of the smoke volume. Fig. 2 shows two selected randomized initial setups at the left most. Additionally, two different results of the reference and basic simulations after 1,000 simulation steps are shown in the subsequent columns, respectively. It is worth noting that the reference simulations are significantly different from the basic ones, moreover, the small variations in the initial setup result in significant differences even after short periods of time. We use a regular MacCormack scheme (Selle et al., 2008) for the basic simulation and the more accurate and compute-intensive advection-reflection scheme (Zehnder et al., 2018) for the reference. The grid resolution of the basic simulation is 32×64, and a four times finer grid is used for the reference.
In the following, we denote the basic version corrected by the full correction function computed with Eq. 9 as the “ground truth” version. The coarse version naturally cannot exactly represent the high-resolution reference, and as such, we consider the corrected version without NN inference as the “ground truth” version we are aiming for with our model. The ground truth of our selected two simulations is shown at the right most in Fig. 2.
We train our model using the data collected from the 20 simulations; each simulation performs 1,000 steps, thus we collected 20,000 samples in total. From each simulation, we extract pairs of input features xL and a correction vector field vL. For the input features, we can use multiple channels depending on the complexity of our target function. From a pilot test, we experimentally found that nine channels, i.e., three subsequent states give best results for our problem (see Fig. 10 in Appx. A.2). These consist of the velocity and density fields, i.e., xL = {vn−2L ,v n−1 L ,v n L, ρ n−2 L , ρ n−1 L , ρ n L}. We randomly split the data into a training data set of 95% and a validation data set of 5%. We use network model A (see Appx. A.1 and Fig. 9a) and train for 200 epochs with a batch size of 32. We use the Lsup loss and an Adam optimizer (Kingma & Ba, 2014) with an adaptive learning rate starting at 0.001.
We test the trained model within two simulations that are not part of the training data set. A selected frame for these tests is shown in Fig. 3, which visualizes the density fields. Our model produces significantly closer results to the ground truth than the basic simulation. Since the error accumulates over time, differences become more apparent in later frames. Nevertheless, we find that, despite the large number of nonlinear simulation steps, the inferred correction field manages to move the solution close to the desired state. The average errors in both density and velocity fields are shown in Fig. 4. Here, we show relative errors with respect to ground truth correction. The errors show that our supervised model corrects the simulations
very well; errors stay very close to the ground truth values and start to slightly deviate only after 400 steps of simulation.

heading: 4.2 NAIVE CORRECTION

text:
We now evaluate the effect of the proposed optimization approach to compute the correction function on the learning process. In our context, a straightforward, i.e., “naive”, way to get a correction is to directly downsample the reference correction (i.e., cH ) to the target domain. We find that this
correction function produces comparable ground truth results for the rising smoke, and hence, we can train a network in a supervised manner with this downsampled correction data. Fig. 5a shows an example. Here, we directly apply a regularized and a “naive” correction function without involving the learning step. This figure highlights that both versions yield almost identical results and, hence, the regularization also retains the important characteristics of the high-resolution reference. Apart from omitting our optimization step, i.e., providing an unregularized set of target data for the supervised training, all steps are performed as described above.
Figs. 5b–5d show the results of training with both versions. The “naive” NN model converges to a much higher overall level of error, and the resulting function approximation is unusable in practice. The inference errors quickly cause the basic simulation to deteriorate and become un-
stable. This behavior contrasts with to our approach where the temporal regularization makes it possible to accurately represent the correction function with the CNN. We also demonstrate that our approach generalizes to other problems in the context of fluid flows. See Appx. A.3 for a liquid stream example.
We have shown that the smooth changes of a correction field with respect to smoothly changing inputs are crucial for successfully learning the correction function. Despite obtaining gains in accuracy from the correction function, it requires manual input to make the correction function amenable to learning via suitable temporal regularization. In the following, we will show how our unsupervised learning approach alleviates this difficulty.

heading: 4.3 LEARNING WITH A DIFFERENTIABLE PDE SOLVER

text:
We start with a test for the reference data that presents relatively smooth motions close to the basic simulation. Hence, we first test the correction from a low-resolution to a high-resolution simulations with the same standard numerical scheme for both versions. In this test, we use a semi-Lagrangian advection (Stam, 1999), and the reference data is generated for 100 steps from 20 randomized initial sizes of the smoke volume, i.e., 2,000 samples in total. It is worth noting that the actual amount of samples the NN model sees is much larger, because the intermediate output states from the recurrent iterations are likewise seen by the model. These states vary over the course of the training steps since the applied correction function changes.
Fig. 6 shows the evaluation of our trained model. It is apparent that our model with 48 recurrent steps yields a model that successfully removes the majority of the errors as shown on the left of Fig. 6. The graphs of Fig. 6a show velocity errors relative to the ground truth version for each of three models trained with different numbers of recurrent steps. Here, we can see that, with more recurrent steps, the model has a higher chance to learn the correction function for long term accuracy. Fig. 6b shows errors of inferences for the given 100 ground truth steps, where the ground truth correction is evaluated via the difference between the solver result and the downsampled reference data. These graphs demonstrate that the model with 48 recurrent steps yields the best long term accuracy despite
containing larger per step differences with respect to ground truth corrections. This indicates that, by looking ahead more steps from the solver, the model learns to anticipate the future dynamics and corrects the target function accordingly instead of relying on the seemingly ideal corrections for each time step.
Next, we test our model in a more complex settings using the reference data used in Sec. 4.1. Fig. 8 shows the corrected results of our unsupervised model, which was again trained with 48 recurrent steps, and relative errors are given in Fig. 7. It is worth noting that, within the first ca. 300 steps, the unsupervised model learns even more accurate corrections than the target ground truth version. This illustrates that the model can learn from the gradients provided over the course of the many timesteps by the differentiable solver to find a correction function that performs better than the carefully precomputed supervised version. On the other hand, we also see that the gains
disappear over the course of very long sequences (more than ca. 500 steps). Presumably, the unsupervised model would need to be trained with more recurrent steps to learn to anticipate very long-term changes.

heading: 5 DISCUSSION AND CONCLUSION

text:
A key benefit of our approach is the gain in performance resulting from our trained models. A correction velocity field for a given input is inferred only on the basic simulation grid, i.e., the lowresolution grid. This inference happens for each solving step to assist the underlying numerical solver and only requires a fixed O(n) cost for n degrees of freedom. For example, for our rising smoke test, a simulation involving the trained NN model took ca. 20 seconds for 1,000 steps whereas its corresponding high-resolution counterpart took ca. 104 seconds to compute. See Table 1 in Appx. A.4 for more details.
At training time, the unsupervised learning approach leads to significantly longer training times since each recurrent step of the architecture can require evaluating a complex numerical procedure. A potential remedy and interesting topic for future work would be to use larger timestep size in the solver such that the model could directly learn the correction for longer horizons. As the excellent performance of the unsupervised model for the initial stages of our simulations suggests, this is a very promising avenue.
To summarize, we introduced a novel approach that assists numerical methods by learning a correction function to improve the accuracy of the solution. We demonstrated that taking into account the temporal information is crucial for our goal and an optimization step can ensure that sequences of corrections can be learned accurately by a NN model. The model successfully improves the accuracy for previously unseen PDE solves. Additionally, an unsupervised training via a differentiable solver can be employed to further improve the learned correction for time spans that do not strongly exceed the number of steps seen during training. Despite focusing on fluids, we envision that our approach can be applied to a variety of other application domains that involve numerical methods for spatio-temporal problems, from plasma physics Lewis & Miller (1984) to climate modeling Randall et al. (2007).

heading: A APPENDIX

text:
A.1 NEURAL NETWORK MODEL
For the NN architecture to represent the correction function cn of Eq. 1, we use a fully convolutional network with residual blocks (He et al., 2015). The two network models, which are used for our experiments, are shown in Fig. 9. The models A and B consist of 405K and 265K training parameters, respectively. Hence, we use the model B for setups with a smaller amount of training data. Our models are implemented using the TensorFlow framework (Abadi et al., 2015).
A.2 RISING SMOKE EXPERIMENTS
In order to decide an effective input feature for a NN model, we compare two models trained with different numbers of input channel. Fig. 10 shows the errors of inferred correction between two models and indicates that the model trained with three subsequent states (i.e., k = 3) performs slightly better particularly for longer steps. Nevertheless, we observe that the difference is negligible.
A.3 LIQUID STREAM WITH STEP OBSTACLES
As an extension of our method to other types of fluid simulations, we target a liquid stream flowing over a backward facing step. To make the setup slightly more complex, we introduce a second step on the right side of the domain. Height and width of both steps are randomized to generate data sets. We note that Lagrangian approaches, i.e., particle-based discretizations, form an important class of fluid-related problems and our approach also generalizes to such approaches via an auxiliary grid. This example demonstrates how to use our method in conjunction with the methods such as the fluid implicit particle algorithm (Brackbill et al., 1988). In this method, the flow motion is transferred to a helper grid in every simulation step and follow the remaining steps as explained before. Fig. 13 shows this setup for two different initial conditions. Here, the color indicates the direction of velocity such that it highlights the locations of vortex structures in the flow. All simulations adopt the APIC method without surface tension but instead with a gravitational force and a fixed inflow velocity.
The grid resolution for the basic and corrected simulations is 32×16, and the reference grid is four times finer.
Comparing the reference and basic simulations, we can find that the basic simulation has difficulties resolving the rotating motions that form behind the step geometries. We use the network model B (see Appx. A.1 and Fig. 9b) and train the model with the data from ten simulations. Fig. 14 shows the results of our model applied to two test cases. A notable improvement of our model is that it manages to reintroduce the vortex above the right step that was lost in the basic version. The shape of the left vortex also improves to match the ground truth version.
A.4 PERFORMANCE
Here, we give details about the performance of our method to illustrate the gains that can be achieved by evaluating our trained model. Table 1 shows the timings measured from different simulations of each example. All experiments were performed on an Intel Xeon E5-1620 3.70 GHz processor with 128 GB memory. The trained NN models were evaluated with the CUDA support and TensorFlow on an NVIDIA GeForce GTX 960 GPU with 4 GB video memory. The reference is a full simulation with four times higher resolution evaluated with an optimized CPU-based solver (the same one using to generate the input for our method).