## Accelerating Full Waveform Inversion By Transfer Learning

Divya Shyam Singh ∗ 1 , Leon Herrmann , Qing Sun , Tim B¨rchner , Felix 1 2 u 1 Dietrich 2 and Stefan Kollmannsberger 3

1 Chair of Computational Modeling and Simulation, Technical University of Munich, School of Engineering and Design, Arcisstraße 21, Munich, 80 333, Germany

2 Chair of Scientific Computing in Computer Science (SCCS), School of Computation, Information and Technology,

Technical University of Munich, Boltzmannstrasse 3, Garching, 85748, Germany

3 Chair of Data Science in Engineering, Bauhaus-Universit¨t Weimar, Coudraystraße 13 b, 99423, Weimar, Germany a

## Abstract

Full waveform inversion (FWI) is a powerful tool for reconstructing material fields based on sparsely measured data obtained by wave propagation. For specific problems, discretizing the material field with a neural network (NN) improves the robustness and reconstruction quality of the corresponding optimization problem. We call this method NN-based FWI. Starting from an initial guess, the weights of the NN are iteratively updated to fit the simulated wave signals to the sparsely measured data set. For gradient-based optimization, a suitable choice of the initial guess, i.e., a suitable NN weight initialization, is crucial for fast and robust convergence.

In this paper, we introduce a novel transfer learning approach to further improve NN-based FWI. This approach leverages supervised pretraining to provide a better NN weight initialization, leading to faster convergence of the subsequent optimization problem. Moreover, the inversions yield physically more meaningful local minima. The network is pretrained to predict the unknown material field using the gradient information from the first iteration of conventional FWI. In our computational experiments on two-dimensional domains, the training data set consists of reference simulations with arbitrarily positioned elliptical voids of different shapes and orientations. We compare the performance of the proposed transfer learning NN-based FWI with three other methods: conventional FWI, NN-based FWI without pretraining and conventional FWI with an initial guess predicted from the pretrained NN. Our results show that transfer learning NN-based FWI outperforms the other methods in terms of convergence speed and reconstruction quality.

Keywords: transfer learning, neural network, deep learning, adjoint optimization, Full waveform inversion

## 1. Introduction

Civil infrastructure is often deteriorated due to exposure to loads, temperature fluctuations, natural disasters, and potential faults during construction. Assessing the damage present in the infrastructure requires non-destructive testing (NDT) [1-3]. In recent years, attempts have been made to exploit the potential of full waveform inversion (FWI) in NDT. Originating in seismology [4], FWI infers material models of the earth's interior by fitting simulated seismic waves to measured data sets from active or passive sources [5, 6]. The approach is adopted in engineering applications [710] utilizing ultrasonic waves to reveal information about unknown defects such as voids, cracks, and inclusions. Extracting detailed information about damage location and severity using FWI is

∗ divya.singh@tum.de , Corresponding author

difficult, as the problem is ill-posed and computationally expensive [11, 12].

Deep learning-based approaches are investigated to enrich FWI, potentially offering a chance to overcome the challenges mentioned above. There are several methods to incorporate neural networks (NNs) into FWI, the most common being using NNs in a supervised manner. In this context, the NN learns to map time series signals, such as displacements or pressures at sensor locations within the domain, to the corresponding field of material coefficients, often referred to as distribution, which describes wave velocities or density [13-16]. Another approach uses NNs to enrich the signals with low-frequency data [17-20]. Kleman et al. [21] use an NN to predict the actual velocity distribution by inputting the velocity distribution after the first iteration of conventional FWI. A common challenge with supervised approaches is the need for large training data sets. To address this limitation, generative adversarial networks can be utilized [22]. However, due to the absence of an accuracy measure in the supervised learning approach, the resulting predictions are often unreliable, thereby limiting their practical applicability.

Another way to take advantage of deep learning methods in FWI is to use them with conventional FWI [23-26]. Herrmann et al. [25] show that inflating the optimization space by using a massively overparameterized NN to predict the coefficients of a piecewise constant material field can serve as an implicit regularization of the optimization problem. Solving the corresponding problem with the Adam optimizer results in fast convergence to physically reasonable local minima. One reason for this may be the inherent bias of NN to learn the low-frequency content of the data [27]. Therefore, the algorithm converges to an image that has fewer high-frequency artifacts. The gradients used for this case are calculated first by the continuous adjoint method, where the derivative of the cost function with respect to the coefficients of the piecewise constant material field is calculated. Then, the gradient of the cost function with respect to the NN parameters is determined by automatic differentiation (see the gradient computation in Figure 2). In the sequel, we will refer to this approach of using the NN as a parameterization of the material field as NN-based FWI .

It is possible to further improve NN-based FWI using transfer learning. Transfer learning is a technique in machine learning where an NN trained on one data set is used for a different but related learning task. By doing so, the NN starts the downstream task with weights already adjusted to recognize relevant features, significantly reducing the training time. Simply put, transfer learning uses the knowledge from previous data to accelerate learning for related tasks. Transfer learning has already been utilized in various fields, such as for the classification of text [28, 29] and images [30, 31], machine translation [32, 33] and robotics [34-36]. This approach can also be used in conjunction with NN-based Full Waveform Inversion (FWI), where pretraining the NN provides a suitable initial guess for the optimization task. The training data set for this includes wave simulations on samples with arbitrary defects. In a recent study by Muller et al. [26], their NN was pretrained directly on sensor data to predict an appropriate initial velocity distribution. A similar approach is described in [37].

In this paper, although a similar idea is used, a significant improvement in the pretraining of the NN is proposed. Instead of simply using raw sensor data to predict an initial material distribution as in [37], the gradient computed by the adjoint method with respect to the material coefficients of the piecewise constant representation (obtained during the first iteration with a homogeneous material) is utilized. This simplifies the architecture of the NN because the gradient of the conventional FWI from the first iteration has the same shape as that of the domain. The gradient contains an approximation of the shape of the damage (Figure 3), which renders the learning task easier. Furthermore, using pretrained NN removes the dependency of the convergence of the NN on the initial weight initialization. The paper at hand thereby proposes a method for accelerating the NN-based FWI [25] by pretraining the NN. Using the pretrained NN for FWI is called transfer learning with NN-based FWI. The proposed method is compared with three other approaches: conventional FWI, NN-based FWI without pretraining, and conventional FWI with an initial guess from the pretrained NN.

The remainder of the paper is organized as follows: In section 2, the background of conventional and NN-based FWI is discussed, including the proposed transfer learning approach. In section 3, we demonstrate the superior performance of the transfer learning NN-based FWI with the help of several numerical examples. Finally, section 4 concludes the paper's main findings.

## 2. Methodology

## 2.1. Physical model

In the sequel, the scalar wave equation is used. It describes the propagation of the scalar wave field u ( x , t ) in space x and time t in a medium whose spatial distribution of the density is given by ρ ( x ). The spatial distribution of the wave speed is denoted as c ( x ), the external force as f ( x , t ). With this notation, the 2D scalar wave equation reads

$$\rho ( x ) \ddot { u } ( x, t ) - \nabla \cdot ( \rho ( x ) c ( x ) ^ { 2 } \nabla u ( x, t ) ) = f ( x, t ), \ \ x = [ x, y ] \in \Omega, t \in \mathcal { T },$$

where Ω is the 2D spatial domain and T = [0 , T max ] the time interval of interest. Appendix A of [25] gives a detailed derivation of this equation. B¨ urchner et al. [9] show that using mono-parameter FWI to only invert for density leads to better identification of defects with strong contrasts such as voids. Therefore, a dimensionless scaling function γ ( x ) is introduced to parameterize the scalar wave equation and the following optimization problem. Assuming ρ 0 and c 0 correspond to undamaged background material, the unknown material field γ ( x ) scales the density, i.e., ρ ( x ) = γ ( x ) ρ 0 , while the wave speed remains constant, i.e., c ( x ) = c 0 . This leads to the density-scaled version of the scalar wave equation, which is used throughout the paper,

$$\gamma ( x ) \rho _ { 0 } \ddot { u } ( x, t ) - \nabla \cdot ( \gamma ( x ) \rho _ { 0 } c _ { 0 } ^ { 2 } \nabla u ( x, t ) ) = f ( x, t ), \ \ x = [ x, y ] \in \Omega, t \in \mathcal { T }.$$

The initial and boundary conditions are

$$u ( x, 0 ) = i _ { \L } ( x, 0 ) = 0 \quad \text{ on } \Omega,$$

$$\frac { \partial u ( x, t ) } { \partial x } = 0 \quad \text{ on } \Gamma _ { y } \times \mathcal { T },$$

$$\frac { \partial u ( x, t ) } { \partial y } = 0 \quad \text{ on } \Gamma _ { x } \times \mathcal { T }.$$

## 2.2. Numerical configuration

In the paper, all investigations are performed with the same numerical configuration. The setup is shown in Figure 1. Considering a 2D domain of 0 1 m . × 0 05 m, a finite differences (FD) solver is . used to solve the corresponding wave equation (as discussed in detail in Appendix A of [25]). The domain is excited with a two-cycle sine burst at the positions x s . Introducing the central frequency k c with the corresponding angular frequency ω = 2 πk c and amplitude A 0 (Figure 1b), the external force reads

$$f ( x, t ) = \begin{cases} A _ { 0 } \sin ( \omega t ) \sin ^ { 2 } ( \frac { \omega t } { 2 n _ { c } } ) \delta ( x - x ^ { s } ) & \text{for $0\leq t\leq \frac{2\pi n _ { c } } { \omega } \\ 0 & \text{for $\frac{\pi n _ { c } } { \omega } < t$} \end{cases}$$

where n c = 2 is the number of cycles and x s is the source position of the underlying experiment.

The undamaged density and wave speed of the material is 2700 kg m 3 and 6000 m s . The central excitation frequency is 500 kHz. The synthetic reference data is generated with simulations on a grid with 512 × 256 grid points and 2000 time steps with dt = 3 × 10 -8 s. For the FWI, a different grid configuration has been used to avoid the 'inverse crime' [38]. The domain is discretized with 256 × 128 grid points, and the time integration is performed with 1000 time steps and a time step

Figure 1: (a) Set up of the 2D domain. The black dots indicate the position of the sensors in the domain, and the red dots indicate the point of excitation. The blue ellipsoid represents the damage, parameterized using the semi-major and minor axes a,b, the center ( x , y c c ), and the angle ϕ . (b) Excitation force sine burst with two cycles

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-f2f1d326-c4ca-40df-87a6-efb4ed4c751a.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-b0247cb5-7fe9-4b83-9bf8-0867609f5436.jpg)

length of dt = 6 × 10 -8 s. The sources and sensors are positioned along the top edge of the domain. There are four sources in total and the simulation with the excitation from one sensor is equivalent to carrying one experiment. The sensors are distributed symmetrically about the center of the plate, 36 grid points apart. 24 sensors spaced 6 grid points apart are placed between the rightmost and leftmost sources. The excitation of one source is equivalent to conducting one experiment.

## 2.3. Conventional FWI

Using conventional FWI, the problem at hand is parameterized by a set of coefficients ˆ representing γ the scaling function γ ( x ) at all finite difference grid points. Introducing a misfit functional L (ˆ) γ the corresponding optimization problem is described by

$$\hat { \gamma } ^ { * } = \underset { \hat { \gamma } } { \arg \min } \, \mathcal { L } ( \hat { \gamma } ),$$

where ˆ γ ∗ is the optimal solution. The misfit functional sums up the squared error at all N r receiver positions x r for all experiments N s

$$\mathcal { L } ( \hat { \gamma } ; u ^ { o } ) = \frac { 1 } { 2 } \int _ { T } \int _ { \Omega } \sum _ { s = 1 } ^ { N _ { s } } \sum _ { r = 1 } ^ { N _ { r } } \left [ ( \hat { u } _ { s } ( \hat { \gamma } ; x, t ) - u _ { s } ^ { o } ( x ^ { r }, t ) ) ^ { 2 } \, \delta ( x - x ^ { r } ) \right ] d \Omega d t,$$

where ˆ (ˆ; u s γ x, t ) denotes a single simulated wave field given a set of coefficients ˆ and γ u o s ( x , t r ) a observed wave field. Measurements across all experiments are summarized by u o = { u o s } N s s =1 .

For the computation of the derivative of the cost function with respect to the material coefficients, the adjoint method is used. For a detailed derivation, see e.g. [9]. Given a wave field u , the derivative with respect to one coefficient ˆ γ i is

$$\frac { d \mathcal { L } } { d \hat { \gamma } _ { i } } = \sum _ { s = 1 } ^ { N _ { s } } \int _ { \mathcal { T } } \left [ - \rho _ { 0 } \dot { u } _ { s, i } ^ { \dagger } \dot { u } _ { s, i } + \rho _ { 0 } c _ { 0 } ^ { 2 } \nabla u _ { s, i } ^ { \dagger } \cdot \nabla u _ { s, i } \right ] d t,$$

where the subindex i denotes the solution of the corresponding primal and adjoint solution at the grid point i . The derivatives at all grid points form the gradient is ∇ L γ (ˆ; γ u o ) and can readily be used in gradient-based optimization.

Figure 2: Workflow of the NN-based FWI. The generator network outputs a density scaling function, which is used to solve the equation using a forward solver. A cost function is calculated based on wavefields from the generator network output and the observed wavefields from the true density scaling function, which is then used to calculate the gradient for backpropagation Equation (12).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-185ae545-8039-4711-bdd6-6a5aa0b7b18d.jpg)

## 2.4. NN-based FWI

## 2.4.1. Hybrid gradient computation

In the NN-based FWI, an NN parameterized by θ is used to predict the scaling function coefficients, i.e., ˆ( γ θ ). To this end, consider an NN N γ that predicts the material coefficients at all grid points:

$$\hat { \gamma } ( \theta ) = N _ { \gamma } ( \theta ; k ),$$

where θ contains all parameters of the NN and k is the input as shown in ?? . Therefore, the optimization problem becomes

$$\theta ^ { * } = \underset { \theta } { \arg \min } \, \mathcal { L } ( \hat { \gamma } ( \theta ) ; u ^ { o } ).$$

The gradient of the cost function with respect to the NN parameters ∇ θ can be computed with the chain rule

$$\nabla _ { \theta } \mathcal { L } ( \hat { \gamma } ; u ^ { 0 } ) = \underbrace { \nabla _ { \hat { \gamma } } \mathcal { L } ( \hat { \gamma } ; u ^ { 0 } ) } _ { \text{adjoint method} } \cdot \underbrace { \nabla _ { \theta } \hat { \gamma } } _ { \text{automatic} } \text{differentiation}$$

As described in detail in [25], the first term ∇ L γ (ˆ( γ θ ); u o ) is obtained through the adjoint method, while the latter ∇ θ ˆ( γ θ ) with automatic differentiation. The NN used to predict the density scaling function γ (Equation (10)) has a generator type architecture [39]. The details of the generator architecture can be found in the Appendix Section A.1. The input k to the generator network is a tensor of size 128 × 8 × 4 filled with random values sampled from a normal distribution with zero mean and variance one. The output is the spatial distribution of the density scaling function, a tensor of size 1 × 256 × 128 equivalent to the size of the FD grid. The Adam optimizer [40] available in the PyTorch library [41] is used to solve the optimization problem. The NN-based FWI method is summarized in Figure 2.

## 2.4.2. Transfer learning NN-based FWI

Transfer learning NN-based FWI uses a pretrained U-Net (Figure 3) to parameterize the density scaling function γ . The knowledge from a training data set is transferred to provide good weight initialization of the NN for the downstream task of FWI.

Pretraining: In [37, 26], the pretraining was carried out using observed wavefields as inputs and the true density distribution as the output. This already improved NN-based FWI. Yet, further improvements are possible by using the adjoint gradient ∇ L γ (ˆ γ 0 ; u o ) from the first iteration as

the input where the material field ˆ γ 0 is equal to 1 throughout the entire domain. This simplifies the design of the NN architecture since the shape of the gradient has the identical dimension as the discretized domain of computation itself, thus allowing the choice of the same shape for input and output (Figure 4). The gradient also already contains an approximation of the damage. This causes the initial guess to be closer to the sought density scaling function γ true . Therefore, the training data set is D = {∇ L γ (ˆ γ 0 ; u o i ) , γ true i } N i =1 where N is the number of training samples. The supervised pretraining is carried out using the mean squared error (MSE) over all N samples as loss function given as

$$\mathcal { L } _ { D } = \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \| \gamma _ { i } ^ { \text{true} } - \hat { \gamma } _ { i } \| _ { 2 } ^ { 2 },$$

where ˆ is the output of the NN γ N γ . For this purpose, the NN, as shown in Figure 3, is used. Note that a different architecture is used in transfer learning NN-based FWI as compared to NN-based FWI, since the input tensor for pretraining is the same size as the output tensor, which was not the case earlier. Supervised training aims to minimize the average loss over the entire training data set. Pretraining the U-Net in such a manner requires one full gradient computation of the first conventional FWI iteration. Therefore, preparing the dataset for pretraining is more expensive than the one carried out in [37] since one adjoint simulation has to be carried out in addition to every forward simulation. However, it leads to better results as demonstrated in Section 3 because it provides a better initialization of the NN.

A U-Net-based convolutional NN architecture [42] is employed for pretraining. More details about this architecture are in the Appendix (A.2). The U-Net employs skip connections (Figure 3) to address the vanishing gradient problem. As previously mentioned, the input tensor is the adjoint gradient from the first conventional FWI iteration L (ˆ γ 0 ; u o ), and the output is the predicted density scaling function ˆ. γ The U-Net is trained on a data set containing one ellipsoid-shaped damage parameterized using the semi-major and minor axes a and b, the center x , y c c , and the rotation θ , see Figure 1a. The mean squared loss function is used along with RMSprop [43] optimization to train the network. The training of the U-Net is carried out with 800 samples for 100 epochs using a batch size of 80 and requires around 240 seconds on a Nvidia Quadro RTX 8000. More details about the optimal number of epochs and samples used for pretraining can be found in Appendix A.3.1 and A.3.2, respectively.

Figure 3: Pretraining of the U-Net. The supervised training uses the adjoint gradient from the first iteration of the conventional FWI as the input, and the true density scaling function is the output The U-Net is trained on 800 samples.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-dba3736c-a49a-4c9b-9ec7-2a6eb253a074.jpg)

Once the U-Net is trained, it is used for the downstream task of FWI. The optimization then updates the weights of N γ based on the FWI loss function Equation (12). The updated Equation (10) is:

Figure 4: Workflow of the transfer learning NN-based FWI. The pretrained U-Net is used for the FWI. The weights of the pretrained U-Net are updated over iterations such that the misfit between the observed and simulated wavefields are minimized.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-2997cf9b-6fbe-4727-a040-4f9412de59b1.jpg)

.

$$\hat { \gamma } ( \theta ^ { p } ) = N _ { \gamma } ( \theta ^ { p } ; \nabla _ { \gamma } \mathcal { L } ( \hat { \gamma } _ { 0 } ; u ^ { o } ) ).$$

In the following, the pretrained U-Net is used to perform NN-based FWI (Figure 4). The input to the pretrained network is the adjoint gradient from the first conventional FWI iteration. However, the loss function is changed to Equation (8), and the gradient used to update θ p is used for the NN-based FWI outlined in Equation (12).

## 2.4.3. Conventional FWI with initial guess

For comparison, the initial guess provided by the pretrained network is used with conventional FWI. Therefore, the prediction ˆ from the pretrained network γ N γ ( θ p ; ∇ L γ (ˆ γ 0 ; u o )) is used as starting model.

## 2.4.4. Importance of NN initialization

Weight initialization for NNs is important to achieve fast convergence and good accuracy. Using the weight initialization suggested by Glorot in [44] is common practice. This avoids the vanishing gradient problem but does not form an optimal starting point for FWI, as the random weights lead to the noisy initialization patterns depicted in the upper left part of Figure 5. On the contrary, an expected outcome is that the material remains undamaged almost everywhere except for locally confined flaws. An approximation to undamaged material can be provided by additionally initializing the weights of the last layer using a standard normal distribution with a narrow standard deviation of 0.01 and setting a rather high bias of, e.g. three. This removes most of the noise from the initialization of the density scaling function and provides a better starting point for NN-based FWI. As a last step, a sigmoid function maps the values to the range between zero and one. Furthermore, setting the weight of the last layer causes the method to be less dependent on the initialization of the weights. Figure 5 shows the effect of the weight initialization on the reconstruction of a damage case within 35 iterations. This weight initialization is used for the generator network for NN-based FWI (Figure 2) as well as for the U-Net employed for transfer learning NN-based FWI (Figure 3). In the case of transfer learning NN-based FWI, the weights of the last layer are initialized in this way during pretraining, which improves the convergence of the training process.

## 3. Results and discussion

## 3.1. Damage cases

To compare all four methods, i.e., conventional FWI (Section 2.3), NN-based FWI (Section 2.4.1), conventional FWI with an initial guess from pretraining (Section 2.4.3), and transfer learning NN-

Figure 5: Effect of the weight initialization on the reconstruction of the density scaling function γ over iterations.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-31d6be90-3a40-4680-aafa-e565b6146607.jpg)

based FWI (Section 2.4.2), four damage cases are used, see Figure 6. The aim is to quantify the methods with respect to their reconstruction quality and convergence speed. The red color represents the region where the density scaling function has a value of 1, and the blue color represents a void/damage, i.e., a value of 10 -5 the scaling function.

Figure 6: Four damage cases used for the case study to compare the four methods listed above. The cases are designed in increasing order of complexity.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-b9bbb73f-5063-4cfa-b9bd-bfc629010e06.jpg)

The four case studies are presented to assess the performance of transfer learning NN-based FWI. Then, the four methods are compared quantitatively using an average of over 100 test cases. Apart from the four case studies, we also compare the methods quantitatively with an average performance over 100 test cases. These cases were not part of the training data set used to pre-train the U-Net. All methods were executed on an Nvidia Quadro RTX 8000, and each case required ∼ 100 seconds to complete.

Figure 7: Case 1: Comparison of the four methods for the shown true density scaling function with one elliptical damage for 35 iterations.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-a4071abe-837f-48df-a656-62ea5ae1208f.jpg)

## 3.2. Case 1

Figure 7 compares the four methods for the first damage case. The test case is similar to but not a sample from the data used for the pretraining of the network described in Section 2.4.2. Therefore, it is expected that the pretrained network will perform better than other methods for this case. The first column in Figure 7 shows that conventional FWI recovers the damage within 10-15 iterations. However, the reconstruction suffers from artifacts, which only reduce slightly with increasing iterations. The second column shows the individual snapshots of NN-based FWI without pretraining, which recovers the damage within 10 iterations with good accuracy and without any artifacts in the reconstruction. The third column shows the results of the corresponding iterations for conventional FWI with an initial guess from the pretrained network. As expected, providing a good initial guess improves the adjoint method's reconstruction and helps recover the damage faster than without pretraining. Finally, the fourth column shows the results from transfer learning NNbased FWI. The reconstruction of the damage is much faster; within 5 iterations, the reconstructed shape of the damage is very accurate, leading to the best results for this case.

## 3.3. Case 2

Figure 8 shows the comparison of the four methods for a damage case containing a rectangularshaped hole and a circular hole. Note that the training set does not include a square hole; only elliptical holes were included (Section 2.4.2). The first column shows conventional FWI, which recovers the shape approximately within 20-25 iterations. However, there are strong artifacts in the area just above the rectangular damage. The NN-based FWI recovers the rectangular-shaped hole and the circular hole within 20 iterations with good accuracy. The conventional FWI with an initial guess from the pretrained network performs much better than without pretraining and achieves a good reconstruction within 20 iterations with less artifacts in the reconstructed image. Nevertheless, the circular hole is not reconstructed very well. Finally, the transfer learning NNbased FWI performs the best. It achieves an almost perfect reconstruction within 10 iterations without any artifacts. As compared to the NN-based FWI without pretraining, the reconstructed shape is close to the true shape, thus demonstrating that the pretrained network performs very well for cases slightly outside the distribution of data on which it is trained.

## 3.4. Case 3

For the third case study (Figure 9), three holes of different sizes need to be reconstructed. This renders the task to be quite different from the training data. The conventional FWI can roughly recover the rightmost damage while failing to recover the damages on the left side within 35 iterations. There are a lot of artifacts in the reconstruction and it is difficult to infer the correct shape and location of the damages from the reconstruction. The NN-based FWI can recover one damage shape within 15 iterations but struggles to reconstruct the other two. However, in this case, the reconstruction is better than conventional FWI, although the reconstructed shape of the damages is not accurate. The conventional FWI with an initial guess from the pretrained NN, delivers a similar reconstruction of the damage without pretraining. By contrast, the transfer learning NN-based FWI can quickly recover all the locations of the holes with good accuracy within 20 iterations. It improves the reconstructed shape of the damages in the subsequent iterations. The damaged shape is recovered more accurately than other methods, and the artifacts are negligible. Therefore, the transfer learning NN-based FWI performs best among all the methods in this case.

## 3.5. Case 4: Is transfer learning NN-based FWI always the best?

In very few cases, the transfer learning NN-based FWI does not perform as well as the other methods. The initial guess provided by the network is too inaccurate in some cases where the sought damage is very different from those of the training data. One such example is shown in Figure 10. The figure depicts two intersecting rectangular damages with three circular holes with a decreasing radius located on a straight line. Furthermore, there are two additional circular holes

on the left and the right sides, resulting in six different damages to be recovered. This is a difficult case for FWI in general, specifically for the pre-trained network, because the training data included holes with only a single damage. Additionally, the smaller circular holes are located in the bigger circular hole's shadow region, making it more challenging to reconstruct the second and third holes.

The conventional FWI can partially recover the rectangular-shaped damage and one hole on the side. The circular holes behind the rectangular damage are not recovered. The NN-based FWI delivers a fast reconstruction of the whole damaged region using only 25-30 iterations to provide high accuracy. In the case of the conventional FWI with an initial guess from the pretrained network, the initial guess is inaccurate. The recovered damages are partially accurate for the rectangular-shaped damages. The circular holes on each side are also partially recovered. For the transfer learning NN-based FWI, the method could only recover the shape of the rectangular-shaped damage and missed all the other damages. Upon increasing the total number of iterations, it is observed that it can recover the circular holes on both sides after 55 iterations. Still, it cannot recover the two intersecting rectangular damages with three circular holes behind each other with a decreasing radius ever after 100 iterations. This is one of the few cases where the damage reconstruction is unsatisfactory. However, the NN-based FWI without pretraining performs the best in this case, clearly indicating the advantage of using NN to discretize the material field.

Figure 8: Case 2: Comparison of the four methods for the shown true density scaling function with a square and a circular damage for 35 iterations.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-da952413-b27e-465f-91d7-d4fc0e12922b.jpg)

Figure 9: Case 3: Comparison of the four methods for the shown true density scaling function with 3 elliptical damages for 35 iterations.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-0e03af70-e0e5-4cc4-8a4e-a69da1a12fe4.jpg)

Figure 10: Case 4: Comparison of all the methods for a manually generated case with multiple damages for 35 iterations.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-0e04f537-4a88-4c57-99e7-d2a8a04e0992.jpg)

| Iterations   | Conventional FWI   | Conventional FWI with NN initial guess   | Transfer learning FWI   |
|--------------|--------------------|------------------------------------------|-------------------------|
| 1            |                    |                                          |                         |
| 5            |                    |                                          |                         |
| 10           |                    |                                          |                         |
| 15           |                    |                                          |                         |
| 20           |                    |                                          |                         |
| 25           |                    |                                          |                         |
| 30           |                    |                                          |                         |
| 35           |                    |                                          |                         |
|              |                    | Target                                   |                         |

## 3.6. Comparison over 100 test cases

FWI is carried out for 100 test cases to compare the four methods quantitatively, each undergoing 35 iterations. The cost function and the mean squared error between the reconstructed and true

density scaling function over the iterations are used as a measure of comparison. Results are depicted in Figure 11.

Figure 11: The left figure shows the log of the cost for four methods over the iterations averaged over 100 cases containing one elliptical damage (similar to Section 3.2). Similarly, the right figure shows the log of the mean squared error between the predicted density scaling function ˆ and the true density scaling function γ γ true at each iteration.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-5bdf7e7c-f14b-4559-a693-5ba8f3256440.jpg)

The cost function for the transfer learning NN-based FWI reduces the fastest till 20 iterations. After 35 iterations, the cost is comparable for NN-based FWI and the transfer learning NN-based FWI. However, the mean squared error is the lowest for transfer learning NN-based FWI, followed by NN-based FWI. The mean squared error at the start of the iteration is the same for conventional FWI with an initial guess from the pretrained NN and transfer learning NN-based FWI. However, the MSE for conventional FWI without and with initial guess from pretrained NN increases slightly over the iterations due to the artifacts in the reconstructed density scaling function. Conversely, the MSE consistently decreases for NN-based FWI and transfer learning NN-based FWI, indicating good reconstruction of the density scaling function with fewer artifacts.

In the examples shown here (as well as in [25]), the reconstruction of a void using NN-based FWI contains considerably fewer artifacts. This is because NNs contain an inherent spectral bias [27] and prefer to learn the low-frequency content of data. On the contrary, the frequencies of the artifacts in the conventional FWI reconstructions are rather high. Therefore, the reconstruction of the NN-based FWI contains a particular kind of high-frequency filter that surprisingly does not inhibit the formation of sharp changes from material to void and vice versa, and, yet, does not exhibit artifacts.

## 3.7. Can transfer learning FWI recover from bad initial guess?

When using a transfer learning framework, a potential issue arises if the pretrained network provides an incorrect initial guess. This can occur when the data given to the pretrained network is different from the training data. To this end, case 2 from Figure 8 is revisited. To emulate a bad pretraining, an incorrect initial guess is set manually for Case 2 (Figure 6) with a subsequent inversion. In Figure 12, it is evident that the initial guess in the first iteration differs strongly from the target damage shape. Although the transfer learning FWI required roughly 70 iterations (compared to the 10 iterations with Transfer learning FWI and 25 iterations with NN-based FWI experienced in Figure 8), it accurately recovers the damage. Similar results were observed for many cases, including

Case 1, whereas for some more complicated cases, such as Case 3 and Case 4, the recovered shape was not as good as before. In general, a greater number of iterations is needed to satisfactorily recover the damages when the initial guess is inaccurate.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-b179a309-c675-4934-a66d-1f97a7f422f1.jpg)

guess

Figure 12: Performance of the transfer learning FWI for bad initial guess.

## 3.8. Drawbacks and limitations

Transfer learning NN-based FWI also suffers from some drawbacks that are common to any machine learning-based method. It requires extra effort to tune all the hyperparameters involved in the process, such as the learning rate, the learning rate scheduler, gradient clipping, cost scaling, and the number of samples for pretraining. This can be a time-intensive process.

As in any other machine learning-based approach, carefully selecting training data is crucial for pretraining the network. Data that is too similar leads to overfitting, with the consequence that the network will not generalize well for out-of-sample data. However, selecting training data can be challenging, particularly if the type of expected damage is unknown. As the domain gets bigger, the task of generating sensible training data for pretraining, as well as hyperparameter tuning, will become more challenging. This indicates potential for further improvement in the architecture of NN models, which are less sensitive to the choice of hyperparameters.

## 4. Conclusion

NN-based FWI [25] uses an NN to parameterize the density scaling function. It relies on wave signals emitted at sources and received by the sensors to reconstruct flaws. This paper demonstrates that it is possible to improve the NN-based FWI using transfer learning, as introduced in [37]. The transfer learning is achieved with a U-Net, which is pretrained in a supervised manner - mapping the gradient from the first iteration of conventional FWI to the true density scaling function. The pretraining alleviates the strong dependency of the NN on the initialization of the NN weights. The combination of the NN ansatz in FWI and transfer learning through supervised pretraining greatly improves the convergence and reconstruction quality over conventional FWI. This includes conventional FWI relying on the same initial guess obtained through the pretrained NN.

In addition to evaluating the improvements compared to the original transfer-learning-based scheme presented [37], the transfer learning NN-based FWI is thoroughly compared to conventional FWI both with and without using the same pretrained network to provide a good starting point. In all cases, the presented transfer learning NN-based FWI outperforms conventional FWI. It might be argued that appropriate regularization techniques [45] lead to similar results as the NN-based FWI for conventional FWI. These can, however, equally be applied to both the NN-based FWI and transfer learning FWI. Whether additional regularization boosts the performance of transfer learning FWI and if similar results are obtainable with conventional FWI is the subject of future study.

## Acknowledgements

The authors gratefully acknowledge the funding provided by the Georg Nemetschek Institut (GNI) through the joint research project DeepMonitor, which finances Divya Shyam Singh and Qing Sun, as well as the Geothermal-Alliance Bavaria (GAB) by the Bavarian State Ministry of Science and the Arts (StMWK), which finances Leon Herrmann. Furthermore, we gratefully acknowledge funds received by the Deutsche Forschungsgemeinschaft under Grant no. KO 4570/1-1 and RA 624/29-1 which support Tim B¨rchner. u Felix Dietrich would like to acknowledge the funds received by the Deutsche Forschungsgemeinschaft - project no. 468830823.

## Declarations

Conflict of interest No potential conflict of interest was reported by the authors.

## Data Availability

We provide a PyTorch implementation of all methods in https://doi.org/10.5281/zenodo.131 50916 .

## A. Neural Network Architectures

## A.1. Generator network architecture

Table 1 shows the detailed architecture of the generator network used for the NN-based FWI method. The generator network upsamples the input tensor of size 128 × 8 × 4 through its layers to match the finite differences grid of the domain (1 × 256 × 128).

Table 1: 2D convolutional NN architecture for the prediction of the indicator γ . The total number of parameters is 526 252.

| layer                                             | shape after layer   | learnable parameters   |
|---------------------------------------------------|---------------------|------------------------|
| input                                             | 128 × 8 × 4         | 0                      |
| upsample                                          | 128 × 16 × 8        | 0                      |
| 2D convolution & PReLU                            | 128 × 16 × 8        | 147 584 +1             |
| 2D convolution & PReLU                            | 128 × 16 × 8        | 147 584 +1             |
| upsample                                          | 128 × 32 × 16       | 0                      |
| 2D convolution & PReLU                            | 64 × 32 × 16        | 73 792 +1              |
| 2D convolution & PReLU                            | 64 × 32 × 16        | 36 928 +1              |
| upsample                                          | 64 × 64 × 32        | 0                      |
| 2D convolution & PReLU                            | 64 × 64 × 32        | 36 928 +1              |
| 2D convolution & PReLU                            | 64 × 64 × 32        | 36 928 +1              |
| upsample                                          | 64 × 128 × 64       | 0                      |
| 2D convolution & PReLU                            | 32 × 128 × 64       | 18 464 +1              |
| 2D convolution & PReLU                            | 32 × 128 × 64       | 9 248 +1               |
| upsample                                          | 32 × 256 × 128      | 0                      |
| 2D convolution & PReLU                            | 32 × 256 × 128      | 9 248 +1               |
| 2D convolution & PReLU                            | 32 × 256 × 128      | 9 248 +1               |
| 2D convolution without padding & adaptive Sigmoid | 1 × 254 × 126       | 289 +1                 |

Table 2: 2D U-Net convolutional NN architecture used for the pretraining figure 3. The total number of parameters is 784 039. ∗ BN: Batch normalization

| layer                       | shape after layer   | learnable parameters   |
|-----------------------------|---------------------|------------------------|
| input                       | 1 × 256 × 128       | 0                      |
| 2D Convolution & BN & PReLU | 16 × 256 × 128      | 160 +32 +1             |
| 2D Convolution & BN & PReLU | 16 × 256 × 128      | 2320 +32 +1            |
| Maxpool 2D                  | 16 × 128 × 64       | 0                      |
| 2D Convolution & BN & PReLU | 32 × 128 × 64       | 4640 +64 +1            |
| 2D Convolution & BN & PReLU | 32 × 128 × 64       | 9248 +64 +1            |
| Maxpool 2D                  | 32 × 64 × 32        | 0                      |
| 2D Convolution & BN & PReLU | 64 × 64 × 32        | 18496 +128 +1          |
| 2D Convolution & BN & PReLU | 64 × 64 × 32        | 36928 +128 +1          |
| Maxpool 2D                  | 64 × 32 × 16        | 0                      |
| 2D Convolution & BN & PReLU | 128 × 32 × 16       | 73856 +256 +1          |
| 2D Convolution & BN & PReLU | 128 × 32 × 16       | 147584 +256 +1         |
| Maxpool 2D                  | 128 × 16 × 8        | 0                      |
| 2D Convolution & BN & PReLU | 128 × 16 × 8        | 147584 +256 +1         |
| 2D Convolution & BN & PReLU | 128 × 16 × 8        | 147584 +256 +1         |
| Upsample                    | 128 × 32 × 16       | 0                      |
| 2D Convolution & BN & PReLU | 64 × 32 × 16        | 110656 +128 +1         |
| 2D Convolution & PReLU      | 64 × 32 × 16        | 36928 +1               |
| Upsample                    | 64 × 64 × 32        | 0                      |
| 2D Convolution & BN & PReLU | 32 × 64 × 32        | 27680 +64 +1           |
| 2D Convolution & PReLU      | 32 × 64 × 32        | 9248 +1                |
| Upsample                    | 32 × 128 × 64       | 0                      |
| 2D Convolution & BN & PReLU | 16 × 128 × 64       | 6928 +32 +1            |
| 2D Convolution & PReLU      | 16 × 128 × 64       | 2320 +1                |
| Upsample                    | 16 × 256 × 128      | 0                      |
| 2D Convolution & BN & PReLU | 1 × 256 × 128       | 154 +2+1               |
| 2D Convolution & Sigmoid    | 1 × 256 × 128       | 10 +1                  |

## A.2. U-Net architecture

Since both the input and the output data are a 2D matrix, a U-Net-based convolutional NN architecture is employed; see table 3. The U-Net [42] architecture consists of a blocked structure where each block consists of a Convolution layer, a Batch normalization layer, and an activation layer repeated twice, followed by a Max pooling layer. The input and output tensor is 256 × 128 in shape. The skip connections (Figure 3) are used to overcome the vanishing gradient problem.

A small parametric study was carried out to find the best parameters for pretraining the U-Net and the optimum number of samples and epochs required for pretraining. It's worth noting that the optimum performance for the complete method is the performance of the pretrained network for the downstream task, i.e., carrying out the NN-based FWI. After each pretraining task, the network was run for 10 test cases, and the average was compared to the performance of each pretrained network. It was found that the optimum performance was achieved by continuing the pretraining further after the validation error had stopped reducing for the pretraining task. A similar observation was reported in [46].

## A.3. Hyperparameter tuning

The hyperparameters for each method are tuned separately using 35 iterations to maximize the individual performance (Table 3). A learning rate scheduler with a polynomial decay ( β · epoch+1) α was also used in this process with α = -0 5 and . β = 0 2. .

Table 3: Hyperparameters used for various methods

| Hyperparameter    | Transfer learn- ing with NN- based FWI   | Conentional FWI with NN initial guess   | NN-based FWI   | Conventional FWI   |
|-------------------|------------------------------------------|-----------------------------------------|----------------|--------------------|
| Learning rate     | 5 × 10 - 4                               | 5 × 10 - 2                              | 5 × 10 - 4     | 8 × 10 - 2         |
| Gradient clipping | 1 × 10 - 5                               | 1 × 10 - 5                              | 5 × 10 - 5     | 1 × 10 - 5         |
| Cost scaling      | 1 × 10 10                                | 1 × 10 12                               | 1 × 10 8       | 1 × 10 12          |

## A.3.1. Optimal number of epochs

To find the optimum number of epochs for pretraining, the network was trained for 50, 100, 200, and 500 epochs. This was done with three different numbers of samples for pretraining, 50 samples, 200 samples, and 500 samples, to find out if the change in the number of samples is a factor in the optimum number of epochs for pretraining. The hyperparameters used for this parametric study are listed in table 4.

Table 4: Hyperparameters used for pretraining the U-Net

| Hyperparameter    | Value               |
|-------------------|---------------------|
| Learning rate     | 8 × 10 - 4          |
| Gradient clipping | 5 × 10 - 5          |
| α                 | -0.5                |
| β                 | 0.2                 |
| Batch size        | No. of samples / 10 |

Figure 13: Performance of the pretrained network with increasing epochs of pretraining with 800 samples

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-496861b2-c710-4446-95c2-ced7733d5cd1.jpg)

Figure 13 shows the result of a parametric study for a U-Net pretrained using 800 samples and 50 100 200 300 500 epochs for pretraining averaged over 100 test cases. , , , , The graph also looks similar for 50 and 200 samples, with 100 epochs performing the best for carrying out transfer learning NN-based FWI. Increasing the number of epochs further reduces the accuracy. The initial guess of the network trained for 100 epochs was the best compared to other cases, and the error

reduced much faster than in other cases. Therefore, 100 epochs have been used to pretrain the U-Net.

## A.3.2. Optimal number of samples

A parametric study was conducted to determine the optimal number of data sets used for pretraining the U-Net. This study aims to determine the effect of increasing the number of samples on the performance of the U-Net for NN-based FWI. The number of samples used for pretraining the U-Net is 200, 300, 400, 500 and 800. The hyperparameters used for pretraining the network were kept constant for each case. These hyperparameters are shown in Table 4. Figure 14 depicts the mean squared error at each iteration averaged over 100 test cases. Each curve represents the corresponding amount of samples used for pretraining. Figure 14 demonstrates the clear pattern that more samples improve the performance of transfer learning NN-based FWI. As can be seen, the MSE error is the least for pretraining using 800 samples for the downstream task. Therefore, the network pretrained from 800 samples is used for final comparison with other methods.

Figure 14: Performance of the pretrained network with increasing samples for pretraining

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00695v1-figure-31af3338-2091-4858-929f-fd4215ca1e42.jpg)

## References

- G. Park, H. H. Cudney, D. J. Inman, Impedance-based health monitoring of civil structural doi:10.1061/(asce)1
- [1] components, Journal of Infrastructure Systems 6 (4) (2000) 153-160. 076-0342(2000)6:4(153) . URL http://dx.doi.org/10.1061/(asce)1076-0342(2000)6:4(153)
- [2] J. P. Lynch, An overview of wireless structural health monitoring for civil structures, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 365 (1851) (2006) 345-372. doi:10.1098/rsta.2006.1932 . URL http://dx.doi.org/10.1098/rsta.2006.1932
- [3] S. Kim, S. Pakzad, D. Culler, J. Demmel, G. Fenves, S. Glaser, M. Turon, Health monitoring of civil infrastructures using wireless sensor networks, in: Proceedings of the 6th international conference on Information processing in sensor networks - IPSN '07, IPSN '07, ACM Press, 2007. doi:10.1145/1236360.1236395 . URL http://dx.doi.org/10.1145/1236360.1236395
- [4] A. Tarantola, Inversion of seismic reflection data in the acoustic approximation, GEOPHYSICS 49 (8) (1984) 1259-1266. doi:10.1190/1.1441754 . URL http://dx.doi.org/10.1190/1.1441754

- [5] A. Fichtner, Full Seismic Waveform Modelling and Inversion, Springer Berlin Heidelberg, 2011. doi:10.1007/978-3-642-15807-0 . URL http://dx.doi.org/10.1007/978-3-642-15807-0
- [6] Copyright, in: W. Menke (Ed.), Geophysical Data Analysis (Fourth Edition), fourth edition Edition, Academic Press, 2018. doi:https://doi.org/10.1016/B978-0-12-813555-6.099 94-9 .
- URL https://www.sciencedirect.com/science/article/pii/B9780128135556099949
- [7] R. Seidl, E. Rank, Full waveform inversion for ultrasonic flaw identification, in: AIP Conference Proceedings, Author(s), 2017. doi:10.1063/1.4974657 . URL http://dx.doi.org/10.1063/1.4974657
- [8] J. He, J. Rao, J. D. Fleming, H. Nath Gharti, L. T. Nguyen, G. Morrison, Numerical ultrasonic full waveform inversion (fwi) for complex structures in coupled 2d solid/fluid media, Smart Materials and Structures 30 (8) (2021) 085044. doi:10.1088/1361-665x/ac0f44 . URL http://dx.doi.org/10.1088/1361-665X/ac0f44
- [9] T. B¨ urchner, P. Kopp, S. Kollmannsberger, E. Rank, Immersed boundary parametrizations for full waveform inversion, Computer Methods in Applied Mechanics and Engineering 406 (2023) 115893. doi:https://doi.org/10.1016/j.cma.2023.115893 . URL https://www.sciencedirect.com/science/article/pii/S0045782523000166
- [10] T. B¨rchner, P. Kopp, S. Kollmannsberger, E. Rank, Isogeometric multi-resolution full waveu form inversion based on the finite cell method, Computer Methods in Applied Mechanics and Engineering (2023) 116286 doi:https://doi.org/10.1016/j.cma.2023.116286 . URL https://www.sciencedirect.com/science/article/pii/S0045782523004103
- [11] J. Virieux, S. Operto, An overview of full-waveform inversion in exploration geophysics, GEOPHYSICS 74 (6) (2009) WCC1-WCC26. doi:10.1190/1.3238367 . URL http://dx.doi.org/10.1190/1.3238367
- [12] J. Virieux, A. Asnaashari, R. Brossier, L. M´tivier, A. Ribodetti, W. Zhou, 6. An introduction e to full waveform inversion, Society of Exploration Geophysicists, 2014, pp. R1-1-R1-40. doi: 10.1190/1.9781560803027.entry6 . URL http://dx.doi.org/10.1190/1.9781560803027.entry6
- [13] F. Yang, J. Ma, Deep-learning inversion: A next-generation seismic velocity model building method, GEOPHYSICS 84 (4) (2019) R583-R599. doi:10.1190/geo2018-0249.1 . URL http://dx.doi.org/10.1190/geo2018-0249.1
- [14] Y. Wu, Y. Lin, Inversionnet: An efficient and accurate data-driven full waveform inversion, IEEE Transactions on Computational Imaging 6 (2020) 419-433. doi:10.1109/TCI.2019.2 956866 .
- [15] W. Wang, J. Ma, Velocity model building in a crosswell acquisition geometry with imagetrained artificial neural networks, GEOPHYSICS 85 (2) (2020) U31-U46. doi:10.1190/geo2 018-0591.1 .
- URL http://dx.doi.org/10.1190/geo2018-0591.1
- [16] J. Rao, F. Yang, H. Mo, S. Kollmannsberger, E. Rank, Quantitative reconstruction of defects in multi-layered bonded composites using fully convolutional network-based ultrasonic inversion, Journal of Sound and Vibration 542 (2023) 117418. doi:https://doi.org/10.1016/j.jsv. 2022.117418 .
- URL https://www.sciencedirect.com/science/article/pii/S0022460X22006010
- [17] Y. Jin, W. Hu, X. Wu, J. Chen, Learn low-wavenumber information in fwi via deep inceptionbased convolutional networks, in: SEG Technical Program Expanded Abstracts 2018, Society

of Exploration Geophysicists, 2018. doi:10.1190/segam2018-2997901.1 . URL http://dx.doi.org/10.1190/segam2018-2997901.1

- [18] H. Sun, L. Demanet, Extrapolated full-waveform inversion with deep learning, Geophysics 85 (3) (2020) R275-R288. arXiv:https://pubs.geoscienceworld.org/geophysics/arti cle-pdf/85/3/R275/5018257/geo-2019-0195.1.pdf , doi:10.1190/geo2019-0195.1 . URL https://doi.org/10.1190/geo2019-0195.1
- [19] T. Yang, X. Chai, W. Nie, J. Yu, Deep-learning-based low-frequency reconstruction for fullwaveform inversion, 2022, pp. 134-137. arXiv:https://library.seg.org/doi/pdf/10.119 0/iwmg2021-35.1 , doi:10.1190/iwmg2021-35.1 . URL https://library.seg.org/doi/abs/10.1190/iwmg2021-35.1
- [20] Y. Li, A. Alali, T. Alkhalifah, High-wavenumber extrapolation for band-limited fwi results using deep learning 2022 (1) (2022) 1-5. doi:https://doi.org/10.3997/2214-4609.2022 10778 .
- URL https://www.earthdoc.org/content/papers/10.3997/2214-4609.202210778
- [21] C. Kleman, S. Anwar, Z. Liu, J. Gong, X. Zhu, A. Yunker, R. Kettimuthu, J. He, Full waveform inversion-based ultrasound computed tomography acceleration using two-dimensional convolutional neural networks, Journal of Nondestructive Evaluation, Diagnostics and Prognostics of Engineering Systems 6 (4) (Mar. 2023). doi:10.1115/1.4062092 . URL http://dx.doi.org/10.1115/1.4062092
- [22] M. Araya-Polo, S. Farris, M. Florez, Deep learning-driven velocity model building workflow, The Leading Edge 38 (11) (2019) 872a1-872a9. doi:10.1190/tle38110872a1.1 . URL http://dx.doi.org/10.1190/tle38110872a1.1
- [23] Q. He, Y. Wang, Reparameterized full-waveform inversion using deep neural networks, GEOPHYSICS 86 (1) (2021) V1-V13. doi:10.1190/geo2019-0382.1 . URL http://dx.doi.org/10.1190/geo2019-0382.1
- [24] W. Zhu, K. Xu, E. Darve, B. Biondi, G. C. Beroza, Integrating deep neural networks with full-waveform inversion: Reparameterization, regularization, and uncertainty quantification, GEOPHYSICS 87 (1) (2021) R93-R109. doi:10.1190/geo2020-0933.1 . URL http://dx.doi.org/10.1190/geo2020-0933.1
- [25] L. Herrmann, T. B¨rchner, F. Dietrich, S. Kollmannsberger, On the use of neural networks u for full waveform inversion, Computer Methods in Applied Mechanics and Engineering 415 (2023) 116278. doi:10.1016/j.cma.2023.116278 . URL http://dx.doi.org/10.1016/j.cma.2023.116278
- [26] A. P. O. Muller, J. C. Costa, C. R. Bom, M. Klatt, E. L. Faria, M. P. de Albuquerque, M. P. de Albuquerque, Deep pre-trained fwi: where supervised learning meets the physicsinformed neural networks, Geophysical Journal International 235 (1) (2023) 119-134. doi: 10.1093/gji/ggad215 . URL http://dx.doi.org/10.1093/gji/ggad215
- [27] N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio, A. Courville, On the spectral bias of neural networks, in: K. Chaudhuri, R. Salakhutdinov (Eds.), Proceedings of the 36th International Conference on Machine Learning, Vol. 97 of Proceedings of Machine Learning Research, PMLR, 2019, pp. 5301-5310. URL https://proceedings.mlr.press/v97/rahaman19a.html
- [28] K. Nigam, A. K. Mccallum, S. Thrun, T. Mitchell, Machine Learning 39 (2/3) (2000) 103-134. doi:10.1023/a:1007692713085 , [link]. URL http://dx.doi.org/10.1023/a:1007692713085

- [29] R. Qasim, W. H. Bangyal, M. A. Alqarni, A. Ali Almazroi, A fine-tuned bert-based transfer learning approach for text classification, Journal of Healthcare Engineering 2022 (2022) 1-17. doi:10.1155/2022/3498123 .
- URL http://dx.doi.org/10.1155/2022/3498123
- [30] M. Shaha, M. Pawar, Transfer learning for image classification, in: 2018 Second International Conference on Electronics, Communication and Aerospace Technology (ICECA), 2018, pp. 656-660. doi:10.1109/ICECA.2018.8474802 .
- [31] M. Hussain, J. J. Bird, D. R. Faria, A Study on CNN Transfer Learning for Image Classification, Springer International Publishing, 2018, p. 191-202. doi:10.1007/978-3-319-97982-3 \_16 . URL http://dx.doi.org/10.1007/978-3-319-97982-3\_16
- [32] B. Zoph, D. Yuret, J. May, K. Knight, Transfer learning for low-resource neural machine translation, CoRR abs/1604.02201 (2016). arXiv:1604.02201 . URL http://arxiv.org/abs/1604.02201
- [33] S. M. Lakew, A. Erofeeva, M. Negri, M. Federico, M. Turchi, Transfer learning in multilingual neural machine translation with dynamic vocabulary, CoRR abs/1811.01137 (2018). arXiv: 1811.01137 .
- URL http://arxiv.org/abs/1811.01137
- [34] H. Tang, L. Notash, Neural network based transfer learning for robot path generation, in: Volume 8B: 45th Mechanisms and Robotics Conference (MR), IDETC-CIE2021, American Society of Mechanical Engineers, 2021. doi:10.1115/detc2021-69006 . URL http://dx.doi.org/10.1115/DETC2021-69006
- [35] K. Pereida, M. K. Helwa, A. P. Schoellig, Data-efficient multirobot, multitask transfer learning for trajectory tracking, IEEE Robotics and Automation Letters 3 (2) (2018) 1260-1267. doi: 10.1109/LRA.2018.2795653 .
- [36] N. Makondo, B. Rosman, O. Hasegawa, Accelerating model learning with inter-robot knowledge transfer, in: 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 2417-2424. doi:10.1109/ICRA.2018.8461218 .
- [37] S. Kollmannsberger, D. Singh, L. Herrmann, Transfer learning enhanced full waveform inversion*, in: 2023 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 2023, pp. 866-871. doi:10.1109/AIM46323.2023.10196158 .
- [38] A. Wirgin, The inverse crime (2004). arXiv:math-ph/0401050 .
- [39] V. Badrinarayanan, A. Kendall, R. Cipolla, Segnet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (12) (2017) 2481-2495. doi:10.1109/TPAMI.2016.2644615 .
- [40] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization (2014). doi:10.48550/A RXIV.1412.6980 .
- URL https://arxiv.org/abs/1412.6980
- [41] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-performance deep learning library, in: Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 8024-8035,

http://papers.neurips.cc/paper/9015-pytorch-an

- -imperative-style-high-performance-deep-learning-library.pdf .

- [42] O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, Springer International Publishing, 2015, p. 234-241. doi:10.1007/978-3-319 -24574-4\_28 .
- URL http://dx.doi.org/10.1007/978-3-319-24574-4\_28
- [43] A. Graves, Generating sequences with recurrent neural networks (2013). doi:10.48550/ARX IV.1308.0850 . URL https://arxiv.org/abs/1308.0850
- [44] X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in: Y. W. Teh, M. Titterington (Eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Vol. 9 of Proceedings of Machine Learning Research, PMLR, Chia Laguna Resort, Sardinia, Italy, 2010, pp. 249-256. URL https://proceedings.mlr.press/v9/glorot10a.html
- [45] S. Arridge, P. Maass, O. Oktem, C.-B. Sch¨nlieb, Solving inverse problems using data-driven ¨ o models, Acta Numerica 28 (2019) 1-174. doi:10.1017/s0962492919000059 . URL http://dx.doi.org/10.1017/S0962492919000059
- [46] B. Neyshabur, H. Sedghi, C. Zhang, What is being transferred in transfer learning?, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, Vol. 33, Curran Associates, Inc., 2020, pp. 512-523. URL https://proceedings.neurips.cc/paper\_files/paper/2020/file/0607f4c705595 b911a4f3e7a127b44e0-Paper.pdf
Divya Shyam Singh Leon Herrmann Qing Sun Tim Bürchner Felix Dietrich Stefan Kollmannsberger
Accelerating Full Waveform Inversion By Transfer Learning
Full waveform inversion (FWI) is a powerful tool for reconstructing material
fields based on sparsely measured data obtained by wave propagation. For
specific problems, discretizing the material field with a neural network (NN)
improves the robustness and reconstruction quality of the corresponding
optimization problem. We call this method NN-based FWI. Starting from an
initial guess, the weights of the NN are iteratively updated to fit the
simulated wave signals to the sparsely measured data set. For gradient-based
optimization, a suitable choice of the initial guess, i.e., a suitable NN
weight initialization, is crucial for fast and robust convergence.
  In this paper, we introduce a novel transfer learning approach to further
improve NN-based FWI. This approach leverages supervised pretraining to provide
a better NN weight initialization, leading to faster convergence of the
subsequent optimization problem. Moreover, the inversions yield physically more
meaningful local minima. The network is pretrained to predict the unknown
material field using the gradient information from the first iteration of
conventional FWI. In our computational experiments on two-dimensional domains,
the training data set consists of reference simulations with arbitrarily
positioned elliptical voids of different shapes and orientations. We compare
the performance of the proposed transfer learning NN-based FWI with three other
methods: conventional FWI, NN-based FWI without pretraining and conventional
FWI with an initial guess predicted from the pretrained NN. Our results show
that transfer learning NN-based FWI outperforms the other methods in terms of
convergence speed and reconstruction quality.