text:
# Table of Contents

  1. Cover
  2. Related Titles
  3. Title
  4. Authors
  5. Copyright
  6. Dedication
  7. Preface
  8. Acknowledgments
  9. 1: Introduction
    1. 1.1 Computational Physics and Computational Science
    2. 1.2 This Book's Subjects
    3. 1.3 This Book's Problems
    4. 1.4 This Book's Language: The Python Ecosystem
    5. 1.5 Python's Visualization Tools
    6. 1.6 Plotting Exercises
    7. 1.7 Python's Algebraic Tools
  10. 2: Computing Software Basics
    1. 2.1 Making Computers Obey
    2. 2.2 Programming Warmup
    3. 2.3 Python I/O
    4. 2.4 Computer Number Representations (Theory)
    5. 2.5 Problem: Summing Series
  11. 3: Errors and Uncertainties in Computations
    1. 3.1 Types of Errors (Theory)
    2. 3.2 Error in Bessel Functions (Problem)
    3. 3.3 Experimental Error Investigation
  12. 4: Monte Carlo: Randomness, Walks, and Decays
    1. 4.1 Deterministic Randomness
    2. 4.2 Random Sequences (Theory)
    3. 4.3 Random Walks (Problem)
    4. 4.4 Extension: Protein Folding and Self-Avoiding Random Walks
    5. 4.5 Spontaneous Decay (Problem)
    6. 4.6 Decay Implementation and Visualization
  13. 5: Differentiation and Integration
    1. 5.1 Differentiation
    2. 5.2 Forward Difference (Algorithm)
    3. 5.3 Central Difference (Algorithm)
    4. 5.4 Extrapolated Difference (Algorithm)
    5. 5.5 Error Assessment
    6. 5.6 Second Derivatives (Problem)
    7. 5.7 Integration
    8. 5.8 Quadrature as Box Counting (Math)
    9. 5.9 Algorithm: Trapezoid Rule
    10. 5.10 Algorithm: Simpson's Rule
    11. 5.11 Integration Error (Assessment)
    12. 5.12 Algorithm: Gaussian Quadrature
    13. 5.13 Higher Order Rules (Algorithm)
    14. 5.14 Monte Carlo Integration by Stone Throwing (Problem)
    15. 5.15 Mean Value Integration (Theory and Math)
    16. 5.16 Integration Exercises
    17. 5.17 Multidimensional Monte Carlo Integration (Problem)
    18. 5.18 Integrating Rapidly Varying Functions (Problem)
    19. 5.19 Variance Reduction (Method)
    20. 5.20 Importance Sampling (Method)
    21. 5.21 von Neumann Rejection (Method)
    22. 5.22 Nonuniform Assessment
  14. 6: Matrix Computing
    1. 6.1 Problem 3: N–D Newton–Raphson; Two Masses on a String
    2. 6.2 Why Matrix Computing?
    3. 6.3 Classes of Matrix Problems (Math)
    4. 6.4 Python Lists as Arrays
    5. 6.5 Numerical Python (NumPy) Arrays
    6. 6.6 Exercise: Testing Matrix Programs
  15. 7: Trial-and-Error Searching and Data Fitting
    1. 7.1 Problem 1: A Search for Quantum States in a Box
    2. 7.2 Algorithm: Trial-and-Error Roots via Bisection
    3. 7.3 Improved Algorithm: Newton–Raphson Searching
    4. 7.4 Problem 2: Temperature Dependence of Magnetization
    5. 7.5 Problem 3: Fitting An Experimental Spectrum
    6. 7.6 Problem 4: Fitting Exponential Decay
    7. 7.7 Least-Squares Fitting (Theory)
    8. 7.8 Exercises: Fitting Exponential Decay, Heat Flow and Hubble's Law
  16. 8: Solving Differential Equations: Nonlinear Oscillations
    1. 8.1 Free Nonlinear Oscillations
    2. 8.2 Nonlinear Oscillators (Models)
    3. 8.3 Types of Differential Equations (Math)
    4. 8.4 Dynamic Form for ODEs (Theory)
    5. 8.5 ODE Algorithms
    6. 8.6 Runge–Kutta Rule
    7. 8.7 Adams–Bashforth–Moulton Predictor–Corrector Rule
    8. 8.8 Solution for Nonlinear Oscillations (Assessment)
    9. 8.9 Extensions: Nonlinear Resonances, Beats, Friction
    10. 8.10 Extension: Time-Dependent Forces
  17. 9: ODE Applications: Eigenvalues, Scattering, and Projectiles
    1. 9.1 Problem: Quantum Eigenvalues in Arbitrary Potential
    2. 9.2 Algorithms: Eigenvalues via ODE Solver + Search
    3. 9.3 Explorations
    4. 9.4 Problem: Classical Chaotic Scattering
    5. 9.5 Problem: Balls Falling Out of the Sky
    6. 9.6 Theory: Projectile Motion with Drag
    7. 9.7 Exercises: 2- and 3-Body Planet Orbits and Chaotic Weather
  18. 10: High-Performance Hardware and Parallel Computers
    1. 10.1 High-Performance Computers
    2. 10.2 Memory Hierarchy
    3. 10.3 The Central Processing Unit
    4. 10.4 CPU Design: Reduced Instruction Set Processors
    5. 10.5 CPU Design: Multiple-Core Processors
    6. 10.6 CPU Design: Vector Processors
    7. 10.7 Introduction to Parallel Computing
    8. 10.8 Parallel Semantics (Theory)
    9. 10.9 Distributed Memory Programming
    10. 10.10 Parallel Performance
    11. 10.11 Parallelization Strategies
    12. 10.12 Practical Aspects of MIMD Message Passing
    13. 10.13 Scalability
    14. 10.14 Data Parallelism and Domain Decomposition
    15. 10.15 Example: The IBM Blue Gene Supercomputers
    16. 10.16 Exascale Computing via Multinode-Multicore GPUs
  19. 11: Applied HPC: Optimization, Tuning, and GPU Programming
    1. 11.1 General Program Optimization
    2. 11.2 Optimized Matrix Programming with NumPy
    3. 11.3 Empirical Performance of Hardware
    4. 11.4 Programming for the Data Cache (Method)
    5. 11.5 Graphical Processing Units for High Performance Computing
    6. 11.6 Practical Tips for Multicore and GPU Programming
  20. 12: Fourier Analysis: Signals and Filters
    1. 12.1 Fourier Analysis of Nonlinear Oscillations
    2. 12.2 Fourier Series (Math)
    3. 12.3 Exercise: Summation of Fourier Series
    4. 12.4 Fourier Transforms (Theory)
    5. 12.5 The Discrete Fourier Transform
    6. 12.6 Filtering Noisy Signals
    7. 12.7 Noise Reduction via Autocorrelation (Theory)
    8. 12.8 Filtering with Transforms (Theory)
    9. 12.9 The Fast Fourier Transform Algorithm
    10. 12.10 FFT Implementation
    11. 12.11 FFT Assessment
  21. 13: Wavelet and Principal Components Analyses: Nonstationary Signals and Data Compression
    1. 13.1 Problem: Spectral Analysis of Nonstationary Signals
    2. 13.2 Wavelet Basics
    3. 13.3 Wave Packets and Uncertainty Principle (Theory)
    4. 13.4 Short-Time Fourier Transforms (Math)
    5. 13.5 The Wavelet Transform
    6. 13.6 Discrete Wavelet Transforms, Multiresolution Analysis
    7. 13.7 Principal Components Analysis
  22. 14: Nonlinear Population Dynamics
    1. 14.1 Bug Population Dynamics
    2. 14.2 The Logistic Map (Model)
    3. 14.3 Properties of Nonlinear Maps (Theory and Exercise)
    4. 14.4 Mapping Implementation
    5. 14.5 Bifurcation Diagram (Assessment)
    6. 14.6 Logistic Map Random Numbers (Exploration)
    7. 14.7 Other Maps (Exploration)
    8. 14.8 Signals of Chaos: Lyapunov Coefficient and Shannon Entropy
    9. 14.9 Coupled Predator–Prey Models
    10. 14.10 Lotka–Volterra Model
    11. 14.11 Predator–Prey Chaos
  23. 15: Continuous Nonlinear Dynamics
    1. 15.1 Chaotic Pendulum
    2. 15.2 Visualization: Phase-Space Orbits
    3. 15.3 Exploration: Bifurcations of Chaotic Pendulums
    4. 15.4 Alternate Problem: The Double Pendulum
    5. 15.5 Assessment: Fourier/Wavelet Analysis of Chaos
    6. 15.6 Exploration: Alternate Phase-Space Plots
    7. 15.7 Further Explorations
  24. 16: Fractals and Statistical Growth Models
    1. 16.1 Fractional Dimension (Math)
    2. 16.2 The Sierpiński Gasket (Problem 1)
    3. 16.3 Growing Plants (Problem 2)
    4. 16.4 Ballistic Deposition (Problem 3)
    5. 16.5 Length of British Coastline (Problem 4)
    6. 16.6 Correlated Growth, Forests, Films (Problem 5)
    7. 16.7 Globular Cluster (Problem 6)
    8. 16.8 Fractals in Bifurcation Plot (Problem 7)
    9. 16.9 Fractals from Cellular Automata
    10. 16.10 Perlin Noise Adds Realism
    11. 16.11 Exercises
  25. 17: Thermodynamic Simulations and Feynman Path Integrals
    1. 17.1 Magnets via Metropolis Algorithm
    2. 17.2 An Ising Chain (Model)
    3. 17.3 Statistical Mechanics (Theory)
    4. 17.4 Metropolis Algorithm
    5. 17.5 Magnets via Wang–Landau Sampling
    6. 17.6 Wang–Landau Algorithm
    7. 17.7 Feynman Path Integral Quantum Mechanics
    8. 17.8 Feynman's Space–Time Propagation (Theory)
    9. 17.9 Exploration: Quantum Bouncer's Paths
  26. 18: Molecular Dynamics Simulations
    1. 18.1 Molecular Dynamics (Theory)
    2. 18.2 Verlet and Velocity–Verlet Algorithms
    3. 18.3 1D Implementation and Exercise
    4. 18.4 Analysis
  27. 19: PDE Review and Electrostatics via Finite Differences and Electrostatics via Finite Differences
    1. 19.1 PDE Generalities
    2. 19.2 Electrostatic Potentials
    3. 19.3 Fourier Series Solution of a PDE
    4. 19.4 Finite-Difference Algorithm
    5. 19.5 Assessment via Surface Plot
    6. 19.6 Alternate Capacitor Problems
    7. 19.7 Implementation and Assessment
    8. 19.8 Electric Field Visualization (Exploration)
    9. 19.9 Review Exercise
  28. 20: Heat Flow via Time Stepping
    1. 20.1 Heat Flow via Time-Stepping (Leapfrog)
    2. 20.2 The Parabolic Heat Equation (Theory)
    3. 20.3 Assessment and Visualization
    4. 20.4 Improved Heat Flow: Crank–Nicolson Method
  29. 21: Wave Equations I: Strings and Membranes
    1. 21.1 A Vibrating String
    2. 21.2 The Hyperbolic Wave Equation (Theory)
    3. 21.3 Strings with Friction (Extension)
    4. 21.4 Strings with Variable Tension and Density
    5. 21.5 Vibrating Membrane (2D Waves)
    6. 21.6 Analytical Solution
    7. 21.7 Numerical Solution for 2D Waves
  30. 22: Wave Equations II: Quantum Packets and Electromagnetic
    1. 22.1 Quantum Wave Packets
    2. 22.2 Time-Dependent Schrödinger Equation (Theory)
    3. 22.3 Algorithm for the 2D Schrödinger Equation
    4. 22.4 Wave Packet–Wave Packet Scattering
    5. 22.5 E&M Waves via Finite-Difference Time Domain
    6. 22.6 Maxwell's Equations
    7. 22.7 FDTD Algorithm
    8. 22.8 Application: Wave Plates
    9. 22.9 Algorithm
    10. 22.10 FDTD Exercise and Assessment
  31. 23: Electrostatics via Finite Elements
    1. 23.1 Finite-Element Method
    2. 23.2 Electric Field from Charge Density (Problem)
    3. 23.3 Analytic Solution
    4. 23.4 Finite-Element (Not Difference) Methods, 1D
    5. 23.5 1D FEM Implementation and Exercises
    6. 23.6 Extension to 2D Finite Elements
  32. 24: Shocks Waves and Solitons
    1. 24.1 Shocks and Solitons in Shallow Water
    2. 24.2 Theory: Continuity and Advection Equations
    3. 24.3 Theory: Shock Waves via Burgers'Equation
    4. 24.4 Including Dispersion
    5. 24.5 Shallow-Water Solitons: The KdeV Equation
    6. 24.6 Solitons on Pendulum Chain
  33. 25: Fluid Dynamics
    1. 25.1 River Hydrodynamics
    2. 25.2 Navier–Stokes Equation (Theory)
    3. 25.3 2D Flow over a Beam
    4. 25.4 Theory: Vorticity Form of Navier–Stokes Equation
  34. 26: Integral Equations of Quantum Mechanics
    1. 26.1 Bound States of Nonlocal Potentials
    2. 26.2 Momentum–Space Schrödinger Equation (Theory)
    3. 26.3 Scattering States of Nonlocal Potentials
    4. 26.4 Lippmann–Schwinger Equation (Theory)
  35. Appendix A: Codes, Applets, and Animations
  36. Bibliography
  37. Index
  38. End User License Agreement

## Guide

  1. Cover
  2. Table of Contents
  3. Begin Reading

## List of Illustrations

  1. 1 Introduction
    1. Figure 1.1 (a) A representation of the multi-disciplinary nature of computational physics as an overlap of physics, applied mathematics and computer science, and as a bridge among them. (b) Simulation has been added to experiment and theory as a basic approach in the search for scientific truth. Although this book focuses on simulation, we present it as part of the scientific process.
    2. Figure 1.2 Screen dumps of two _x_ – _y_ plots produced by `EasyVisual.py` using the Visual package. Plot (a) uses default parameters, while plot (b) uses user-supplied options.
    3. Figure 1.3 (a) Output from the program 3GraphVisual.py that places three different types of 2D plots on one graph using Visual. (b) Three frames from a Visual animation of a quantum mechanical wave packet produced with HarmosAnimate.py.
    4. Figure 1.4 Some 3D shapes created with single commands in VPython.
    5. Figure 1.5 Matplotlib plots. (a) Output of EasyMatPlot.py showing a simple, _x_ – _y_ plot. (b) Output from GradesMatPlot.py that places two sets of data points, two curves, and unequal upper and lower error bars, all on one plot.
    6. Figure 1.6 (a,b) Columns show two separate outputs, each of two figures, produced by MatPlot2figs.py. (We used the slider button to add some space between the plots.)
    7. Figure 1.7 (a) A 3D wire frame. (b) A surface plot with wire frame. Both are produced by the program Simple3dplot.py using Matplotlib.
    8. Figure 1.8 A 3D scatter plot produced by the program Scatter3dPlot.py using Matplotlib.
    9. Figure 1.9 A comparison of a saw tooth function to the sum of its Fourier components, with the number of included waves varied interactively by a Matplotlib slider. FourierMatplot.py, which produced this output, was written by Oscar Restrepo.
    10. Figure 1.10 (a) A Mayavi surface plot of the function   as seen in the screen viewer. (b) A rotatable visualization of a spherical harmonic   in which the radial distance represents the value of the function.
    11. Figure 1.11 (a) A Mayavi visualization in which tubes are used to connect a set of data points. (b) A Mayavi visualization using arrows (glyphs) to represent a vector field.
    12. Figure 1.12 (a) A Mayavi visualization of the contours of the scalar field  . (b) A volume rendering of the same scalar field.
    13. Figure 1.13 (a) A Mayavi visualization of the same scalar field   using cut planes. (b) A visualization of the same scalar field combining cut planes and contours.
  2. 2 Computing Software Basics
    1. Figure 2.1 A schematic view of a computer's kernel and shells. The hardware is in the center surrounded by increasing higher level software.
    2. Figure 2.2 A flowchart illustrating a program to compute projectile motion. Plot (a) shows the basic components of the program, and plot (b) shows are some of its details. When writing a program, first map out the basic components, then decide upon the structures, and finally fill in the details. This is called _top-down programming._
    3. Figure 2.3 The limits of single-precision floating-point numbers and the consequences of exceeding these limits (not to scale). The hash marks represent the values of numbers that can be stored; storing a number in between these values leads to truncation error. The shaded areas correspond to over- and underflow.
  3. 3 Errors and Uncertainties in Computations
    1. Figure 3.1 A schematic of the _N_ steps in a random walk simulation that end up a distance   from the origin. Notice how the Δ _x_ 's for each step add vectorially.
    2. Figure 3.2 The first four spherical Bessel functions _j l_( _x_ ) as functions of _x_. Notice that for small _x_ , the values for increasing _l_ become progressively smaller.
    3. Figure 3.3 A log–log plot of relative error vs. the number of points used for a numerical integration. The ordinate value of ∼ 10−14 at the minimum indicates that ∼ 14 decimal places of precision are obtained before round-off error begins to build up. Notice that while the round-off error does fluctuate indicating a statistical aspect of error accumulation, on average it is increasing but more slowly than did the algorithm's error decrease.
    4. Figure 3.4 The error in the summation of the series for e− _x_ vs. _N_ for various _x_ values. The values of _x_ increase vertically for each curve. Note that a negative initial slope corresponds to decreasing error with _N_ , and that the dip corresponds to a rapid convergence followed by a rapid increase in error. (Courtesy of J. Wiren.)
  4. 4 Monte Carlo: Randomness, Walks, and Decays
    1. Figure 4.1 (a) A plot of successive random numbers ( _x_ , _y_ ) = ( _r i_, _r_ _i_ +1) generated with a deliberately "bad" generator, (b) A plot generated with the built in random number generator. While the plot (b) is not the proof that the distribution is random, the plot (a) is a proof enough that the distribution is not random.
    2. Figure 4.2 A plot of a uniform pseudorandom sequence _r i_, vs. _i_. The points are connected to make it easier to follow the order. While this does not prove that a distribution is random, it at least shows the range of values and that there is fluctuation.
    3. Figure 4.3 (a) A schematic of the _N_ steps in a random walk simulation that end up a distance _R_ from the origin. Note how the Δ _x_ 's for each step add vectorially. (b) A simulated walk in 3D from _Walk3D.py._
    4. Figure 4.4 (a) The steps taken in seven 2D random walk simulations. (b) The distance covered in two walks of _N_ steps using different schemes for including randomness. The theoretical prediction (4.20) is the straight line.
    5. Figure 4.5 Two self-avoiding random walks that simulate protein chains with hydrophobic (H) monomers in light gray, and polar (P) monomers in black. The dark dots indicate regions where two H monomers are not directly connected.
    6. Figure 4.6 _Circle_ : A sample containing _N_ nuclei, each of which has the same probability of decaying per unit time, _Graphs_ : Semilog plots of the number of nuclei vs. time for five simulations with differing initial numbers of nuclei. Exponential decay would be a straight line with bumps, similar to the initial behavior for _N_ = 100 000.
  5. 5 Differentiation and Integration
    1. Figure 5.1 A trajectory of a projectile experiencing air resistance. Forward-difference approximation (slanted dashed line) and central-difference approximation (horizontal line) for the numerical first derivative at time _t_. (A tangent to the curve at _t_ yields the correct derivative.) The central difference is seen to be more accurate.
    2. Figure 5.2 The integral   d _x_ is the area under the graph of _f_ ( _x_ ) from _a_ to _b._ Here we break up the area into four regions of equal widths _h_ and five integration points.
    3. Figure 5.3 Different shapes used to approximate the areas under the curve. (a) Straight-line sections used for the trapezoid rule. (b) Two parabolas used in Simpson's rule.
    4. Figure 5.4 Log–log plots of the error in the integration of exponential decay using the trapezoid rule, Simpson's rule, and Gaussian quadrature vs. the number of integration points _N_. Approximately 15 decimal places of precision are attainable with double precision (a), and seven places with single precision (b). The algorithms are seen to stop converging when round-off error (the fluctuating and increasing part near the bottom) starts to dominate.
    5. Figure 5.5 (a) Stones into a pond as a technique for measuring its area. The ratio of "hits" to total number of stones thrown equals the ratio of the area of the pond to that of the box. (b) The evaluation of an integral via a Monte Carlo (stone throwing) technique of the ratio of areas.
    6. Figure 5.6 The area under the curve _f_ ( _x_ ) is the same as that under the horizontal line whose height  .
    7. Figure 5.7 A ring of radius _a_ carries a current _l_. Find vector potential at point P.
    8. Figure 5.8 The von Neumann rejection technique for generating random points with weight _W_ ( _x_ ). A random point is accepted if it lies below the curve of _W_ ( _x_ ) and rejected if it lies above. This generates a random distribution weighted by whatever _W_ ( _x_ ) function is plotted.
  6. 6 Matrix Computing
    1. Figure 6.1 Two weights connected by three pieces of string and suspended from a horizontal bar of length _L_. The lengths are all known, but the angles and the tensions in the strings are to be determined.
    2. Figure 6.2 A free body diagram for one weight in equilibrium. Balancing the forces in the _x_ and _y_ directions for all weights leads to the equations of static equilibrium.
    3. Figure 6.3 (a) Row-major order used for matrix storage in Python, C and Java. (b) Column-major order used for matrix storage in Fortran. The table at the bottom shows how successive matrix elements are actually stored in a linear fashion in memory.
  7. 7 Trial-and-Error Searching and Data Fitting
    1. Figure 7.1 A graphical representation of the steps involved in solving for a zero of _f_ ( _x_ ) using the bisection algorithm. The bisection algorithm takes the midpoint of the interval as the new guess for _x_ , and so each step reduces the interval size by one-half. Four steps are shown for the algorithm.
    2. Figure 7.2 A graphical representation of the steps involved in solving for a zero of _f_ ( _x_ ) using the Newton–Raphson method. The Newton–Raphson method takes the new guess as the zero of the line tangent to _f_ ( _x_ ) at the old guess. Two guesses are shown.
    3. Figure 7.3 Two examples of how the Newton–Raphson algorithm may fail if the initial guess is not in the region where _f_ ( _x_ ) can be approximated by a straight line. (a) A guess lands at a local minimum/maximum, that is, a place where the derivative vanishes, and so the next guess ends up at _x_ = ∞. (b)The search has fallen into an infinite loop. The technique know as "backtracking" could eliminate this problem.
    4. Figure 7.4 A function of the reduced magnetism _m_ at three reduced temperatures _t_. A zero of this function determines the value of the magnetism at a particular value of _t_.
    5. Figure 7.5 Three fits to data. _Dashed_ : Lagrange interpolation using an eight-degree polynomial; _Short dashes_ : cubic splines fit ; _Long dashed_ : Least-squares parabola fit.
    6. Figure 7.6 A reproduction of the experimental measurement by Stetz _et al_. (1973) giving the number of decays of a π meson as a function of time since its creation. Measurements were made during time intervals (box sizes) of 10-ns width. The dashed curve is the result of a linear least-squares fit to the log _N_ ( _t_ ).
    7. Figure 7.7 A linear least-squares best fit of a straight line to data. The deviation of theory from experiment is greater than would be expected from statistics, which means that a straight line is not a good theory to describe these data.
    8. Figure 7.8 A linear least-squares best fit of a parabola to data. Here we see that the fit misses approximately one-third of the points, as expected from the statistics for a good fit.
  8. 8 Solving Differential Equations: Nonlinear Oscillations
    1. Figure 8.1 A mass _m_ (the block) attached to a spring with restoring force _F k_( _x_ ) driven by an external time-dependent driving force (the hand).
    2. Figure 8.2 (a) The potentials of an harmonic oscillator (solid curve) and of an anharmonic oscillator (dashed curve). If the amplitude becomes too large for the anharmonic oscillator, the motion becomes unbound. (b) The shapes of the potential energy function _V_ ( _x_ ) ∝ | _x_ | _p_ for _p_ = 2 and _p_ = 6. The "linear" and "nonlinear" labels refer to the restoring force derived from these potentials.
    3. Figure 8.3 A sequence of uniform steps of length _h_ taken in solving a differential equation. The solution starts at time _t_ = 0 and is integrated in steps of _h_ until _t_ = _T_.
    4. Figure 8.4 Euler's algorithm for integration of a differential equation one step forward in time. This linear extrapolation with the slope evaluated at the initial point is seen to lead to an error Δ.
    5. Figure 8.5 The `rk2` algorithm for integration of a differential equation uses a slope (bold line segment) evaluated at the interval's midpoint, and is seen to lead to a smaller error than Euler's algorithm in Figure 8.4.
    6. Figure 8.6 The logarithm of the relative error in the solution of an ODE obtained with `rk4` using a differing number _N_ of time steps over a fixed time interval. The logarithm approximately equals the negative of the number of places of precision. Increasing the number of steps used for a fixed interval is seen to lead to smaller error.
    7. Figure 8.7 The position vs. time for oscillations within the potential   for four different initial amplitudes. Each is seen to have a different period.
  9. 9 ODE Applications: Eigenvalues, Scattering, and Projectiles
    1. Figure 9.1 Computed wave function and the square-well potential (bold lines). The wave function computed by integration in from the left is matched to the one computed by integration in from the right (dashed curve) at a point near the right edge of the well. Note how the wave function decays rapidly outside the well.
    2. Figure 9.2 Two guesses for the energy that are either too low or too high to be an eigenvalue. We see that the low- _E_ guess does not oscillate enough to match onto a dying exponential, and that the high- _E_ guess oscillates too much.
    3. Figure 9.3 A photograph of a pinball machine in which multiple scatterings occur from the bumpers.
    4. Figure 9.4 Scattering from the potential   which in some ways models a pinball machine. The incident velocity _v_ is in the _y_ direction, and the impact parameter ( _x_ value) is _b_. The velocity of the scattered particle is _v'_ and its scattering angle is _θ_.
    5. Figure 9.5 The trajectories of a projectile fired with initial velocity _V_ 0 in the _θ_ direction. The lower curve includes air resistance.
    6. Figure 9.6 (a) The gravitational force on a planet at distance _r_ from the sun. The _x_ and _y_ components of the force are indicated. (b) Output from the applet `Planet` showing the precession of a planet's orbit when the gravitational force ∝ 1/ _r_ 4 (successive orbits do not lie on top of each other).
    7. Figure 9.7 A snapshot from the animated output of the code `UranusNeptune.py` (on Instructor's disk) showing: (a) the orbits of Uranus (inner circle) and of Neptune (outer circle) with the Sun in the center. The arrows indicate the Uranus–Neptune force that causes a perturbation in the orbits. (b) The perturbation in the angular position of Uranus as a result of the presence of Neptune.
  10. 10 High-Performance Hardware and Parallel Computers
    1. Figure 10.1 The logical arrangement of the CPU and memory showing a Fortran array _A_ ( _N_ ) and matrix M( _N, N_ ) loaded into memory.
    2. Figure 10.2 Typical memory hierarchy for a single-processor, high-performance computer (B = bytes, k, M, G, T = kilo, mega, giga, tera).
    3. Figure 10.3 The elements of a computer's memory architecture in the process of handling matrix storage.
    4. Figure 10.4 (a) Multitasking of four programs in memory at one time. On a SISD computer the programs are executed in round robin order. (b) Four programs in the four separate memories of a MIMD computer.
    5. Figure 10.5 (a) A generic view of the Intel core-2 dual-core processor, with CPU-local level-1 caches and a shared, on-die level-2 cache (courtesy of D. Schmitz), (b) The AMD Athlon 64 X2 3600 dual-core CPU (Wikimedia Commons).
    6. Figure 10.6 Two views of parallel computing (courtesy of Yuefan Deng).
    7. Figure 10.7 The theoretical maximum speedup of a program as a function of the fraction of the program that potentially may be run in parallel. The different curves correspond to different numbers of processors.
    8. Figure 10.8 A master process and two slave processes passing messages. Notice how in this not-well-designed program there are more sends than receives, and consequently the results may depend upon the order of execution, or the program may even lock up.
    9. Figure 10.9 A graphical representation of weak vs. strong scaling. Weak scaling keeps each node doing the same amount of work as the problem is made bigger. Strong scaling has each node doing less work (running for less time) as the number of nodes is made bigger.
    10. Figure 10.10 A representation of the lattice in a 2D rectangular space upon which Laplace's equation is solved using a finite difference approach. The lattice sites with white centers correspond to the boundary of the physical system, upon which boundary conditions must be imposed for a unique solution. The large circles in the middle of the lattice represent the algorithm used to solve Laplace's equation in which the potential at the point ( _x, y_ ) = ( _i, j_ )Δ is set to the average of the potential values at the four nearest-neighbor points.
    11. Figure 10.11 The building blocks of Blue Gene (from Gara _et al._ (2005)).
    12. Figure 10.12 The single-node memory system (as presented by Gara _et al._ (2005)).
    13. Figure 10.13 (a) A 3D torus connecting 2 × 2 × 2 compute nodes. (b) The global collective memory system. (c) The control and Gb-ethernet memory system (from Gara _et al._ (2005)).
    14. Figure 10.14 A schematic of an exascale computer in which, in addition to each chip having multiple cores, a graphical processing unit is attached to each chip (adapted from Dongarra (2011)).
  11. 11 Applied HPC: Optimization, Tuning, and GPU Programming
    1. Figure 11.1 Running time `vs.` dimension for an eigenvalue search using `Tune.java` and `tune.f90`.
    2. Figure 11.2 The cache manager's view of RAM. Each 128-B cache line is read into one of four lines in cache.
    3. Figure 11.3 A schematic of a GPU (b) and its interactions with a CPU (a). The GPU is seen to have streaming multiprocessors, special function units and several levels of memory. Also shown is the communication between memory levels and with the host CPU.
    4. Figure 11.4 A schematic of how the contiguous elements of a matrix must be tiled for parallel processing (from Dongarra, 2011).
    5. Figure 11.5 (a) A 1 × 1 grid with a 1D block containing 32 threads, and the corresponding threadIdx.x that goes from 0 to 31. (b) A 1D 4 × 1 grid, that has 4 blocks with 8 threads each.
  12. 12 Fourier Analysis: Signals and Filters
    1. Figure 12.1 (a) A sawtooth function that repeats infinitely in time. (b) The Fourier spectrum of frequencies contained in this function (natural units). Also see Figure 1.9 in which we show the result of summing a finite number of terms of this series.
    2. Figure 12.2 A plot of the functions sin(π _t_ /2) and sin(2π _t_ ). If the sampling rate is not high enough, these signalsmay appear indistinguishable in a Fourier decomposition. If the sample rate is too low and if both signals are present in a sample, the deduced low-frequency component may be contaminated by the higher frequency component signal.
    3. Figure 12.3 (a) A function that is a signal plus noise _s_ ( _t_ ) + _n_ ( _t_ ); (b) the autocorrelation function vs. time deduced by processing this signal; (c) the power spectrum obtained from autocorrelation function; (d) the signal plus noise after passage through a low-pass filter.
    4. Figure 12.4 A schematic of an input signal _f_ ( _t_ ) passing through a filter _h_ that outputs the function _g_ ( _t_ ).
    5. Figure 12.5 (a) An RC circuit arranged as a low-pass filter. (b) An RC circuit arranged as a high-pass filter.
    6. Figure 12.6 A delay-line filter in which the signal at different times is scaled by different amounts _c i_.
    7. Figure 12.7 The rectangle function rect( _ω_ ) that is constant for a finite frequency interval. The Fourier transform of this function is sinc( _t_ ).
    8. Figure 12.8 The basic butterfly operation in which elements _y p_ and _y q_ on the left are transformed into _y p_ \+ _Zy q_ and _y p_ – _Zy q_ on the right.
    9. Figure 12.9 The butterfly operations performing a FFT on the eight data on the left leading to eight transforms on the right. The transforms are different linear combinations of the input data.
    10. Figure 12.10 A modified FFT in which the eight input data on the left are transformed into eight transforms on the right. The results are the same as in the previous figure, but now the output transforms are in numerical order whereas in the previous figure the input signals were in numerical order.
  13. 13 Wavelet and Principal Components Analyses: Nonstationary Signals and Data Compression
    1. Figure 13.1 The input time signal (13.1) we wish to analyze. The signal is seen to contain additional frequencies as time increases. The boxes are possible placements of windows for short-time Fourier transforms.
    2. Figure 13.2 Four possible mother wavelets that can be used to generate entire sets of daughter wavelets. (a–d) Morlet (real part), Mexican hat, Daub4 e6 (explained later), and Haar. The daughter wavelets are generated by scaling and translating these mother wavelets.
    3. Figure 13.3 (a) A wave packet in time corresponding to the functional form (13.4) with _ω_ 0 = 5 and _N_ = 6. (b) The Fourier transform in frequency of this same wave packet.
    4. Figure 13.4 Four wavelet basis functions (daughters) generated by scaling ( _s_ ) and translating ( _τ_ ) an oscillating Gaussian mother wavelet. (a–d) (s = 1, _τ_ = 0), ( _s_ = 1/2, _τ_ = 0), ( _s_ = 1, _τ_ = 6), and ( _s_ = 2, _τ_ = 60). Note how _s_ < 1 is a wavelet with higher frequency, while _s_ > 1 has a lower frequency than the _s_ = 1 mother. Likewise, the _τ_ = 6 wavelet is just a translated version of the _τ_ = 0 one directly above it.
    5. Figure 13.5 A schematic representation of the steps followed in performing a wavelet transformation over all time displacements and scales. The upper signal is first analyzed by evaluating its overlap with a narrow wavelet at the signal's beginning. This produces a coefficient that measures the similarity of the signal to the wavelet. The wavelet is successively shifted over the length of the signal and the overlaps are successively evaluated. After the entire signal is covered, the wavelet is expanded and the entire analysis is repeated.
    6. Figure 13.6 Comparison of an input and reconstituted signal (13.23) using Morlet wavelets (the curves overlap nearly perfectly). As expected for Fourier transforms, the reconstruction is least accurate near the endpoints.
    7. Figure 13.7 The continuous wavelet spectrum obtained by analyzing the input signal with Morelet wavelets. Observe how at small values of time _τ_ there is predominantly one frequency present, how a second, higher frequency (smaller scale) component enters at intermediate times, and how at larger times a still higher frequency components enter (figure courtesy of `Z. Dimcovic`).
    8. Figure 13.8 A graphical representation of the relation between time and frequency resolutions (the uncertainty relation). Each box represents an equal portion of the time-frequency plane but with different proportions of time and frequency.
    9. Figure 13.9 A multifrequency dyadic (power-of-2) filter tree used for discrete wavelet transformations. The L boxes represent low-pass filters and the H boxes represent high-pass filters. Each filter performs a convolution (transform). The circles containing "↓ 2" filter out half of the signal that enters them, which is called _subsampling_ or _factor-of-2 decimation_. The signal on the left yields a transform with a single low and two high components (less information is needed about the low components for a faithful reproduction).
    10. Figure 13.10 An input signal at the top is processed by a tree of high- and low-band filters. The outputs from each filtering are down-sampled with half the data kept. The process continues until there are only two data of high-band filtering and two data of low-band filtering. When complete, the total number of output data equals the total number of signal data. The process of wavelet analysis is thus equivalent to a series of filterings.
    11. Figure 13.11 In successive passes, the filtering of the original signal at the top goes through the pyramid algorithm and produces the outputs shown. The sampling is reduced by a factor of 2 in each step. Note that in the upper graphs, we have connected the points to emphasize their continuous nature while in the lower graphs we plot the individual output points as histograms.
    12. Figure 13.12 (a) The Daub4 e6 wavelet constructed by inverse transformation of the wavelet coefficients. This wavelet has been found to be particularly effective in wavelet analyses. (b) The sum of Daub4 e10 and Daub4 1e58 wavelets of different scale and time displacements.
    13. Figure 13.13 (a) The normalized data and eigenvectors of covariance matrix. (b) The normalized data using the PCA eigenvectors as basis.
  14. 14 Nonlinear Population Dynamics
    1. Figure 14.1 The insect population _x n_ vs. the generation number _n_ for the two growth rates: (a) _μ_ = 2.8, a single attractor; (b) _μ_ = 3.3, a double attractor.
    2. Figure 14.2 The insect population _x n_ vs. the generation number _n_ for two different growth rates: (a) _μ_ = 3.5, a quadruple attractor; (b) _μ_ = 3.8, a chaotic regime.
    3. Figure 14.3 Bifurcation plot of attractor population _x_ ∗ _vs_. growth rate _μ_ for the logistic map. The inset shows some details of a three-cycle window. (The colors or gray scales indicate the regimes over which we distributed the work on different CPUs when run in parallel.)
    4. Figure 14.4 Bifurcation plot for the Gaussian map _f_ ( _x_ ) = exp(− _bx_ 2) + _μ_ (courtesy of W. Hager).
    5. Figure 14.5 Lyapunov coefficient (bottom) and bifurcation values (top) for the logistic map as functions of the growth rate _μ_. Note how the Lyapunov coefficient, whose value is a measure of chaos, changes abruptly at the bifurcations.
    6. Figure 14.6 Shannon entropy (top) and Lyapunov coefficient (bottom) for the logistic map. Note the close relation between the thermodynamic measure of disorder (entropy) and the nonlinear dynamics measure of chaos (Lyapunov).
    7. Figure 14.7 (a) The time dependencies of the populations of prey _p_ ( _t_ ) (solid curve) and of predator _P_ ( _t_ ) (dashed curve) fromthe Lotka–Volterra model. (b) A phase–space plot of Prey population _p_ as a function of predator population _P_. The different orbits correspond to different initial conditions.
    8. Figure 14.8 A chaotic attractor for the 4D Lotka–Volterra model projected onto three axes.
    9. Figure 14.9 (a) The Lotka–Volterra model of prey population _p_ ( _t_ ) (solid curve) and predator population _P_ ( _t_ ) (dashed curve) vs. time when a prey population limit is included, (b) Prey population _p_ as a function of predator population _P_.
    10. Figure 14.10 Lotka–Volterra model with predation efficiency and prey limitations. From left to right: overdamping, _b_ = 0.01; damped oscillations, _b_ = 0.1, and limit cycle, _b_ = 0.3.
  15. 15 Continuous Nonlinear Dynamics
    1. Figure 15.1 (a) A pendulum of length / driven through resistive air (dotted arcs) by an external sinusoidal torque (semicircle). The strength of the external torque is given by _f_ and that of air resistance by _a_. (b) A double pendulum with neither air resistance nor a driving force. In both cases there is a gravitational torque.
    2. Figure 15.2 The data screen (a) and the output screen (b) of the applet `HearData` that converts data into sounds. Columns of [ _t i_, _x_ ( _t i_)] data are pasted into the data window, processed into the graph in the output window, and then converted to sound data that are played by Java.
    3. Figure 15.3 Three potentials and their behaviors in phase space. The different orbits below the potentials correspond to different energies, as indicated by the limits of maximum displacements within the potentials (dashed lines). (a) A repulsive potential leads to open orbits characteristic of nonperiodic motion. The trajectories cross at the hyperbolic point in the middle, an unstable equilibrium point. (b) The harmonic oscillator leads to symmetric ellipses; the closed orbits indicate periodic behavior, and the symmetric trajectories indicate a symmetric potential. (c) A nonharmonic oscillator. Notice that the ellipse-like trajectories at the bottom are neither ellipses nor symmetric.
    4. Figure 15.4 (a) Phase-space trajectories for a pendulum including "over the top" motions. (b) The corresponding _θ_ dependence of the potential.
    5. Figure 15.5 (a) Position vs. time for two initial conditions of a chaotic pendulum that end up with the same limit cycle. (b) A phase space plot of position versus velocity for the limit cycle shown in (a) (courtesy of W. Hager).
    6. Figure 15.6 Position vs. time, phase-space plot, and Fourier spectrum for a chaotic pendulum with _ω_ 0 = 1, _a_ = 0.2, _f_ = 0.52, and _ω_ = 0.666 and three different initial conditions. Column (a) displays three dominant cycles, the column (b) only one, while column (c) has multiple cycles. (Examples of chaotic behavior can be seen in Figure 15.7).
    7. Figure 15.7 Some examples of complicated behaviors of a realistic pendulum. From top to bottom in each column of the figure, the first three rows show _θ_ ( _t_ ) behavior, a broadband Fourier spectrum, a phase-space diagram containing regular patterns with dark bands, and a broad Fourier spectrum. These features are characteristic of chaos. In the bottom two rows we see how the behavior changes abruptly after a slight change in the magnitude of the force and that for _f_ = 0.54 there occur the characteristic broad bands of chaos.
    8. Figure 15.8 (a,b) Output from the applet `Pend1` that produces an animation of a chaotic pendulum, along with the corresponding position vs. time and phase-space plots. (b)The resulting Fourier spectrum produced by `Pend1`. (c,d) outputs from two runs of the applet `Pend2` producing an animation of two chaotic pendula along with the corresponding phase-space plots, and the final output with limit cycles (dark bands).
    9. Figure 15.9 A bifurcation diagram for the damped pendulum with a vibrating pivot (see also the similar diagram for a double pendulum, Figure 15.11). The ordinate is | d _θ_ /d _t_ |, the absolute value of the instantaneous angular velocity at the beginning of the period of the driver, and the abscissa is the magnitude of the driving force _f_. Note that the heavy line results from the overlapping of points, not from connecting the points (see enlargement in the inset).
    10. Figure 15.10 Photographs of a double pendulum built by a student in the OSU Physics Department. The upper pendulum consists of two separated shafts so that the lower one can rotate completely around. Both pendula can go over their tops. The first two frames show the pendulum released from rest and then moving quickly. The photography with faster shutter speeds stops the motion in various stages (photograph by R. Landau).
    11. Figure 15.11 (a) Phase-space trajectories for a double pendulum with _m_ 1 = 10 _m_ 2 and with two dominant attractors. (b) A bifurcation diagram for the double pendulum displaying the instantaneous velocity of the lower pendulum as a function of the mass of the upper pendulum (both plots courtesy of J. Danielson).
    12. Figure 15.12 A 3D plot of a Lorenz attractor output from the program `LorentzAtract.py` (on the Instructor's disk).
  16. 16 Fractals and Statistical Growth Models
    1. Figure 16.1 (a) A statistical fractal Sierpiński gasket containing 10 000 points. Note the self-similarity at different scales. (b) A geometric Sierpiński gasket constructed by successively connecting the midpoints of the sides of each equilateral triangle. The first three steps in the process are labeled as A, B, C.
    2. Figure 16.2 (a) A fractal fern generated by 30 000 iterations of the algorithm 16.14). Enlarging this fern shows that each frond with a similar structure. (b) A fractal tree created with the simple algorithm (16.17).
    3. Figure 16.3 A simulation of the ballistic deposition of 20 000 particles onto a substrate of length 200. The vertical height increases in proportion to the length of deposition time, with the top being the final surface.
    4. Figure 16.4 Examples of the use of "box" counting to determine fractal dimension. (a) In the top "boxes" are circles and the perimeter is being covered. In the bottom an entire figure is being covered. In (b) a "coastline" is being covered by boxes of two different sizes (scales). The fractal dimension can be deduced by recording the number of box of different scale needed to cover the figures.
    5. Figure 16.5 Fractal dimensions of a line, box, and coastline determined by box counting. The slope at vanishingly small scale determines the dimension.
    6. Figure 16.6 (a) A view that might be seen in the undergrowth of a forest or after a correlated ballistic deposition. (b) The probability of particle _i_ \+ 1 sticking in one column depends upon the distance _d_ from the previously deposited particle _i_.
    7. Figure 16.7 (a) A globular cluster of particles of the type that might occur in a colloid, (b) The applet Dla2en.html lets you watch these clusters grow. Here the cluster is at the center of the circle, and random walkers are started at random points around the circle.
    8. Figure 16.8 Number 8 by the American painter Jackson Pollock. (Used with permission, Neuberger Museum, State University of New York.) Some researchers claim that Pollock's paintings exhibit a characteristic fractal structure, while some other researchers question this (Kennedy, 2006). See if you can determine the fractal dimensions within this painting.
    9. Figure 16.9 The rules for two versions of the Game of Life. The rules, given graphically on the top row, create the gaskets below (output obtained from the applet `JCellAut` in the auxiliary files).
    10. Figure 16.10 The coordinates used in adding Perlin noise. (a) The rectangular grid used to locate a square in space and a corresponding point within the square. As shown with the arrows, unit vectors _g_ _i_ with random orientation are assigned at each grid point. (b) A point within each square is located by drawing the four _p_ _i_. The _g_ _i_ vectors are the same as on the left.
    11. Figure 16.11 The mapping used in adding Perlin noise. (a) The numbers _s_ , _t_ , _u_ , and _v_ are represented by perpendiculars to the four vertices, with lengths proportional to their values. (b) The function 3p2 − 2p3 is used as a map of the noise at a point like ( _x_ , _y_ ) to others close by.
    12. Figure 16.12 Perlin noise mapping, (a) The point (x,y) is mapped to point _(s x,xy)_. (b) Using (16.33). Then three linear interpolations are performed to find _c,_ the noise at (x, y).
    13. Figure 16.13 After the addition of Perlin noise, the random scatterplot in (a) becomes the clusters on (b).
    14. Figure 16.14 (a) The output from the Pov-Ray ray-tracing program that took as input the 2D coherent random noise plot in Figure 16.13 and added height and fog. (b) An image of a surface of revolution produced by Pov-Ray in which the marble-like texture is created by Perlin noise.
  17. 17 Thermodynamic Simulations and Feynman Path Integrals
    1. Figure 17.1 The 1D lattice of _N_ spins used in the Ising model of magnetism. The interaction energy between nearest-neighbor pairs _E_ = ± _J_ is shown for aligned and opposing spins.
    2. Figure 17.2 An Ising model simulation on a 1D lattice of 100 initially aligned spins (on the left). Up spins are indicated by circles, and down spins by blank spaces. Although the system starts with all up spins (a "cold" start), the system is seen to form domains of up and down spins as time progresses.
    3. Figure 17.3 Simulation results from a 1D Ising model of 100 spins. (a) Energy and specific heat as functions of temperature; (b) magnetization as a function of temperature.
    4. Figure 17.4 The energy, specific heat, and magnetization as a function of temperature from a 2D Ising model simulation with 40 000 spins. Evidence of a phase transition at the Curie temperature _kT_ =  2.5 is seen in all three functions. The values of _C_ and _E_ have been scaled to fit on the same plot as _M_ (courtesy of J. Wetzel).
    5. Figure 17.5 Wang–Landau sampling used in the Ising model 2D Ising model on an 8 × 8 lattice. (a) Logarithm of the density of states log _g_ ( _E_ ) ∝ _S_ vs. the energy per particle. (b) The histogram _H_ ( _E_ ) showing the number of states visited as a function of the energy per particle. The aim of WLS is to make this function flat.
    6. Figure 17.6 The numbering scheme used in our WLS implementation of the 2D Ising model with an 8 × 8 lattice of spins.
    7. Figure 17.7 In the Feynman path-integral formulation of quantum mechanics a collection of paths connect the initial space–time point _A_ to the final point _B_. The solid line is the trajectory followed by a classical particle, while the dashed lines are additional paths sampled by a quantum particle. A classical particle somehow "knows" ahead of time that travel along the classical trajectory minimizes the action _S_.
    8. Figure 17.8 (a) The probability distribution for the harmonic oscillator ground state as determined with a path-integral calculation (the classical result has maxima at the two turning points). (b) A space–time quantum path resulting from applying the Metropolis algorithm.
    9. Figure 17.9 (a) A path through a space–time lattice that starts and ends at _x_ = _x_ _a_ = _x_ _b_. The action is an integral over this path, while the _path integral_ is a sum of integrals over all paths. The dotted path BD is a transposed replica of path AC. (b) The dashed path joins the initial and final times in two equal time steps; the solid curve uses _N_ steps each of size _ε_. The position of the curve at time _t_ _j_ defines the position _x_ _j_.
    10. Figure 17.10 The analytic and quantum Monte Carlo solution for the quantum bouncer. The continuous line is the Airy function squared and the dashed line | _ψ_ 0( _z_ )|2 after a million trajectories.
  18. 18 Molecular Dynamics Simulations
    1. Figure 18.1 The molecule–molecule effective interaction arises from the many-body interaction of the electrons and nucleus in one molecule (circle) with the electrons and nucleus in another molecule (other circle). Note, the size of the nucleus at the center of each molecule is highly exaggerated, and real electrons have no size.
    2. Figure 18.2 The Lennard–Jones effective potential used in many MD simulations. Note the sign change at _r_ = 1 and the minimum at _r_   1.1225 (natural units). Also note that because the _r_ -axis does not extend to _r_ = 0, the infinitely high central repulsion is not shown.
    3. Figure 18.3 (a) Two frames from an animation showing the results of a 1D MD simulation that starts with uniformly spaced atoms. Note the unequal spacing resulting from an image atom moving in from the left after an atom left from the right. (b) Two frames from the animation of a 2D MD simulation showing the initial and an equilibrated state. Note how the atoms start off in a simple cubic arrangement but then equilibrate to a face-centered-cubic lattice. In both the cases, the atoms remain confined as a result of the interatomic forces.
    4. Figure 18.4 The infinite space generated by imposing periodic boundary conditions on the particles within the simulation volume (shaded box). The two-headed arrows indicate how a particle interacts with the nearest version of another particle, be that within the simulation volume or an image. The vertical arrows indicate how the image of particle 4 enters when the actual particle 4 exits.
    5. Figure 18.5 The kinetic, potential, and total energy as a function of time or number of steps for a 2D MD simulation with 36 particles (a), and 300 particles (b), both with an initial temperature of 150 K. The potential energy is negative, the kinetic energy is positive, and the total energy is seen to be conserved (flat).
    6. Figure 18.6 (a) The temperature after equilibration as a function of initial kinetic energy for a 2D MD simulation with 36 particles. The two are nearly proportional. (b) The pressure vs. temperature for a simulation with several hundred particles. An ideal gas (noninteracting particles) would yield a straight line (courtesy of J. Wetzel).
    7. Figure 18.7 A simulation of a projectile shot into a group of particles. The energy introduced by the projectile is seen to lead to evaporation of the particles (courtesy of J. Wetzel).
  19. 19 PDE Review and Electrostatics via Finite Differences and Electrostatics via Finite Differences
    1. Figure 19.1 (a) The shaded region of space within a square in which we determine the electric potential by solving Laplace's equation. There is a wire at the top kept at a constant 100 V and a grounded wire (dashed) at the sides and bottom. (b) The computed electric potential as a function of _x_ and _y_. The projections onto the shaded _xy_ plane are equipotential (contour) lines.
    2. Figure 19.2 The analytic (Fourier series) solution of Laplace's equation summing 21 terms. Gibbs overshoot leads to the oscillations near _x_ = 0, and persist even if a large number of terms is summed.
    3. Figure 19.3 The algorithm for Laplace's equation in which the potential at the point ( _x, y_ ) = ( _i, j_ )Δ equals the average of the potential values at the four nearest-neighbor points. The nodes with white centers correspond to fixed values of the potential along the boundaries.
    4. Figure 19.4 (a) A simple model of a parallel-plate capacitor within a box. A realistic model would have the plates close together, in order to condense the field, and the enclosing grounded box so large that it has no effect on the field near the capacitor. (b) A numerical solution for the electric potential for this geometry. The projection on the _xy_ plane gives the equipotential lines.
    5. Figure 19.5 A guess as to how charge may rearrange itself on finite conducting plates.
    6. Figure 19.6 (a) A visualization of the computed electric potential for a capacitor with finite width plates. (b) A visualization of the charge distribution along one plate determined by evaluating ∇2 _U_ ( _x, y_ ) (courtesy of J. Wetzel). Note the "lightening rod" effect of charge accumulating at corners and points.
    7. Figure 19.7 (a) The geometry of a capacitor formed by placing two long, square cylinders within each other. (b) The geometry of a capacitor formed by placing two long, circular cylinders within each other. The cylinders are cracked on the side so that wires can enter the region.
    8. Figure 19.8 (a) Computed equipotential surfaces and electric field lines for a realistic capacitor. (b) Equipotential surfaces and electric field lines mapped onto the surface for a 3D capacitor constructed from two tori.
  20. 20 Heat Flow via Time Stepping
    1. Figure 20.1 A metallic bar insulated along its length with its ends in contact with ice. The bar is displayed in dark gray and the insulation is of lighter gray.
    2. Figure 20.2 The algorithm for the heat equation in which the temperature at the location _x_ = _i_ Δ _x_ and time _t_ = ( _j_ \+ 1)Δ _t_ is computed from the temperature values at three points of an earlier time. The nodes with white centers correspond to known initial and boundary conditions (the boundaries are placed artificially close for illustrative purposes).
    3. Figure 20.3 A numerical calculation of the temperature vs. position and vs. time, with isotherm contours projected onto the horizontal plane.
    4. Figure 20.4 Temperature vs. position and time when two bars at differing temperatures are placed in contact at _t_ = 0. The projected contours show the isotherms.
  21. 21 Wave Equations I: Strings and Membranes
    1. Figure 21.1 (a) A stretched string of length _L_ tied down at both ends and under high enough tension so that we can ignore gravity. The vertical disturbance of the string from its equilibrium position is _y_ ( _x, t_ ). (b) A differential element of the string showing how the string's displacement leads to the restoring force.
    2. Figure 21.2 The solutions of the wave equation for four earlier space–time points are used to obtain the solution at the present time. The boundary and initial conditions are indicated by the white-centered dots.
    3. Figure 21.3 The vertical displacement as a function of position _x_ and time _t_ for a string initially placed in a standing mode when friction is included. Notice how the wave moves up and down with time (courtesy of J. Wiren).
    4. Figure 21.4 The vertical displacement as a function of position and time of a string initially plucked simultaneously at two points, as shown by arrows. Note that each initial peak breaks up into waves traveling to the right and to the left. The traveling waves invert on reflection from the fixed ends. As a consequence of these inversions, the _t_   12 wave is an inverted _t_ = 0 wave.
    5. Figure 21.5 (a) A uniform string suspended from its ends in a gravitational field assumes a catenary shape. (b) A force diagram of a section of the catenary at its lowest point. Because the ends of the string must support the entire weight of the string, the tension now varies along the string.
    6. Figure 21.6 A small part of an oscillating membrane and the forces that act on it.
    7. Figure 21.7 The standing wave pattern on a square box top at three different times.
  22. 22 Wave Equations II: Quantum Packets and Electromagnetic
    1. Figure 22.1 The position as a function of time of a localized electron confined to a square well (computed with the code `SqWell.py` available in the instructor's site). The electron is initially on the left with a Gaussian wave packet. In time, the wave packet spreads out and collides with the walls.
    2. Figure 22.2 The probability density as a function of time for an electron confined to a 1D harmonic oscillator potential well. (a) A conventional surface plot, (b) a color visualization.
    3. Figure 22.3 The probability density as a function of position and time for an electron incident upon and passing through a slit. Significant reflection is seen to occur.
    4. Figure 22.4 The probability density as a function of _x_ and _y_ at three different times for an electron confined to a 2D parabolic tube (infinite in the y direction). The electron is initially placed in a Gaussian wave packet in both the _x_ and _y_ directions, and it is to be noted how there is spreading of the wave packet in the _y_ direction, but not in the _x_ direction.
    5. Figure 22.5 (a) Six frames from an animation of the two-particle density _ρ_ ( _x_ 1, _x_ 2, _t_ )as a function of the position of particle 1 with mass _m_ and of the position of particle 2 with mass 10 _m_. (b) This same collision as seen with the single-particle densities _ρ_ ( _x_ 1, _t_ ) and _ρ_ ( _x_ 2, _t_ ). The numbers in the left-hand corners are the times in units of 100Δ _t_. Note that each plot ends at the walls of the containing box, and that particle 1 "bounces off" a wall between times 36 and 86.
    6. Figure 22.6 A time sequence of two single-particle wave packets scattering from each other. The particles have equal mass, a mean kinetic energy equal to a quarter of the well's depth, and the wavefunction has been symmetrized.
    7. Figure 22.7 A single EM pulse traveling along the _z_ -axis. The coupled _E_ and _H_ pulses are indicated by solid and dashed curves, respectively, and the pulses at different _z_ values correspond to different times.
    8. Figure 22.8 The algorithm for using the known values of _E_ x and _H_ y at three earlier times and three different space positions to obtain the solution at the present time. Note that the values of _E_ x are determined on the lattice of filled circles, corresponding to integer space indices and half-integer time indices. In contrast, the values of _H_ y are determined on the lattice of open circles, corresponding to half-integer space indices and integer time indices.
    9. Figure 22.9 The _E_ field (light) and the _H_ field (dark) at the initial time (a) and at a later time (b). Periodic boundary conditions are used at the ends of the spatial region, which means that the large _z_ wave continues into the _z_ = 0 wave.
    10. Figure 22.10 _E_ and _H_ fields at _t_ = 100 for a circularly polarized wave in free space.
    11. Figure 22.11 One frame from the program `quarterwave.py` (on Instructor's site) showing a linearly polarized EM wave entering a quarter-wave plate from the left and leaving as a circularly polarized wave on the right (the arrow on the left oscillates back and forth at 45° while the one on the right rotates).
  23. 23 Electrostatics via Finite Elements
    1. Figure 23.1 A finite element solution to Laplace's equation for two metal plates with a charge density between them. The dots are the nodes _x i_, and the lines connecting the nodes are the finite elements.
    2. Figure 23.2 Basis functions used in finite-elements solution of the 1D Laplace equation. (a) A set of overlapping basis functions _φ i_. Each function is a triangle from _x_ _i_ −1 to _x_ _i_ +1. (b) A Piecewise-linear function. (c) A piecewise-quadratic function.
    3. Figure 23.3 Exact (line) vs. FEM solution (points) for the two-plate problem for _N_ = 3 and _N_ = 11 finite elements ( _N_ = 3 displaced upwards for clarity). On this scale, the _N_ = 11 solution IS identical to the exact one.
    4. Figure 23.4 (a) Decomposition of a domain into triangular elements. Smaller triangles are used in regions of rapid variation and larger triangles are used in regions of slow variation. Discretization errors occur at boundaries. (b) A decomposition of a rectangular domain into 32 right triangles on a mesh with 25 nodes.
    5. Figure 23.5 (a) Linear triangular elements in the _x_ – _y_ plane. (b) Linear triangular element (master element) in the  _η_ plane.
  24. 24 Shocks Waves and Solitons
    1. Figure 24.1 (a) A re-creation on the Union Canal near Edinburgh of Russel's soliton (www.ma.hw.ac.uk/solitons/press.html, Nature v. 376, 3 August 1995, p. 373). (b) Two shallow-water solitary waves crossing each other computed with the code `**Soliton.py**`. The taller soliton on the left catches up with and overtakes the shorter one at _t_   5. The waves resume their original shapes after the collision.
    2. Figure 24.2 The formation of a tsunami. A single two-level waveform at time zero progressively breaks up into eight solitons (labeled) as time increases. The tallest soliton (1) is narrower and faster in its motion to the right. You can generate an animation of this with the program **SolitonAnimate.py**.
    3. Figure 24.3 A visualization showing the wave height vs. position for increasing times showing the formation of a shock wave (sharp edge) from an initial sine wave.
    4. Figure 24.4 A 1D chain of pendulums coupled with a torsion bar on top. The pendulums swing in planes perpendicular to the length of the bar.
    5. Figure 24.5 The dispersion relation for a linearized chain of pendulums.
    6. Figure 24.6 A circular ring soliton at times 8, 20, 40, 60, 80, and 120. This has been proposed as a model for an elementary particle.
  25. 25 Fluid Dynamics
    1. Figure 25.1 Side view of the flow of a stream around a submerged beam (a) and around two parallel plates (b). Both beam and plates have length _L_ along the direction of flow. The flow is seen to be symmetric about the center-line and to be unaffected at the bottom and at the surface by the submerged object.
    2. Figure 25.2 The boundary conditions for two thin submerged plates. The surrounding box is the integration volume within which we solve the PDEs and upon whose surface we impose the boundary conditions. In practice, the box is much larger than _L_ and _H_
    3. Figure 25.3 Boundary conditions for flow around the beam in Figure 25.1. The flow is symmetric about the centerline, and the beam has length _L_ in the _x_ direction (along flow).
    4. Figure 25.4 Two visualizations of the stream function _u_ for Reynold's number _R_ = 5. The one on the left uses contours, while the one on the right uses color.
    5. Figure 25.5 (a) The velocity field around the beam as represented by vectors. (b) The vorticity as a function of _x_ and _y_ for the same velocity field. Rotation is seen to be largest behind the beam.
  26. 26 Integral Equations of Quantum Mechanics
    1. Figure 26.1 A dark particle moving in a dense medium in which it interacts with all particles present. The nonlocality of the potential felt by the dark particle at _r_ arises from the interactions at all _r′_.
    2. Figure 26.2 The grid of momentum values on which the integral equation is solved.
    3. Figure 26.3 (a) A projectile (dark particle at _r_ ) scattering from a dense medium. (b) The same process viewed in the CM system where the projectile and target always have equal and opposite momenta.
    4. Figure 26.4 Three different paths in the complex _k′_ plane used to evaluate line integrals when there are singularities. Here the singularities are at _k_ and _−k,_ and the integration variable is _k′._
    5. Figure 26.5 The energy dependence of the cross section for _l_ = 0 scattering from an attractive delta-shell potential with _λb_ = 15. The dashed curve is the analytic solution (26.39), and the solid curve results from numerically solving the integral Schrödinger equation, either via direct matrix inversion or via LU decomposition.

## List of Tables

  1. 1 Introduction
    1. Table 1.1 Topics for one-quarter (10 Weeks) scientific computing course.
    2. Table 1.2 Topics for two-quarters (20 Weeks) computational physics course.
    3. Table 1.3 Some common Matplotlib commands.
  2. 2 Computing Software Basics
    1. Table 2.1 The IEEE 754 Standard for Primitive Data Types.
    2. Table 2.2 Representation scheme for normal and abnormal IEEE singles.
    3. Table 2.3 Representation scheme for IEEE doubles.
  3. 3 Errors and Uncertainties in Computations
    1. Table 3.1 Approximate values for spherical Bessel functions (from maple).
  4. 4 Monte Carlo: Randomness, Walks, and Decays
    1. Table 4.1 A table of a uniform, pseudo-random sequence _r i_ generated by Python's random method.
  5. 5 Differentiation and Integration
    1. Table 5.1 Elementary weights for uniform-step integration rules.
    2. Table 5.2 Types of Gaussian integration rules.
    3. Table 5.3 Points and weights for 4-point Gaussian quadrature (for checking computation).
  6. 6 Matrix Computing
    1. Table 6.1 The operators of NumPy and their effects.
  7. 7 Trial-and-Error Searching and Data Fitting
    1. Table 7.1 Experimental values for a scattering cross section ( _f_ ( _E_ ) in the theory), each with absolute error _±σ i_, as a function of energy ( _x i_ in the theory).
    2. Table 7.2 Temperature vs. distance as measured along a metal rod.
    3. Table 7.3 Distance vs. radial velocity for 24 extragalactic nebulae.
  8. 8 Solving Differential Equations: Nonlinear Oscillations
    1. Table 8.1 Comparison of ODE solvers for different equations.
  9. 10 High-Performance Hardware and Parallel Computers
    1. Table 10.1 Computation of _c_ = ( _a_ \+ _b_ )/( _d_ × _f_ ).
    2. Table 10.2 Computation of matrix [ _C_ ] = [ _A_ ] + [ _B_ ]-
    3. Table 10.3 Vector processing of matrix [ _A_ ] + [ _B_ ] = [ _C_ ].
    4. Table 10.4 A typical organization of a program containing both serial and parallel tasks.
  10. 12 Fourier Analysis: Signals and Filters
    1. Table 12.1 Reordering for 16 data complex points.
  11. 13 Wavelet and Principal Components Analyses: Nonstationary Signals and Data Compression
    1. Table 13.1 PCA data.
  12. 18 Molecular Dynamics Simulations
    1. Table 18.1 Parameter and scales for the Lennard-Jones potential.
  13. 19 PDE Review and Electrostatics via Finite Differences and Electrostatics via Finite Differences
    1. Table 19.1 Three categories of PDE based on the value of their discriminant _d._
    2. Table 19.2 The relation between boundary conditions and uniqueness for PDEs.
  14. Appendix A: Codes, Applets, and Animations
    1. Table A.1 Python codes, Chapters 1–26.
    2. Table A.2 Animations (A player such as VLC or QuickTime required formpeg and avi, and a Web browser for gifs).

## Pages

  1. cover
  2. contents
  3. ii
  4. iii
  5. iv
  6. v
  7. xix
  8. xx
  9. xxi
  10. xxii
  11.
  12.
  13.
  14.
  15.
  16.
  17.
  18.
  19.
  20.
  21.
  22.
  23.
  24.
  25.
  26.
  27.
  28.
  29.
  30.
  31.
  32.
  33.
  34.
  35.
  36.
  37.
  38.
  39.
  40.
  41.
  42.
  43.
  44.
  45.
  46.
  47.
  48.
  49.
  50.
  51.
  52.
  53.
  54.
  55.
  56.
  57.
  58.
  59.
  60.
  61.
  62.
  63.
  64.
  65.
  66.
  67.
  68.
  69.
  70.
  71.
  72.
  73.
  74.
  75.
  76.
  77.
  78.
  79.
  80.
  81.
  82.
  83.
  84.
  85.
  86.
  87.
  88.
  89.
  90.
  91.
  92.
  93.
  94.
  95.
  96.
  97.
  98.
  99.
  100.
  101.
  102.
  103.
  104.
  105.
  106.
  107.
  108.
  109.
  110.
  111.
  112.
  113.
  114.
  115.
  116.
  117.
  118.
  119.
  120.
  121.
  122.
  123.
  124.
  125.
  126.
  127.
  128.
  129.
  130.
  131.
  132.
  133.
  134.
  135.
  136.
  137.
  138.
  139.
  140.
  141.
  142.
  143.
  144.
  145.
  146.
  147.
  148.
  149.
  150.
  151.
  152.
  153.
  154.
  155.
  156.
  157.
  158.
  159.
  160.
  161.
  162.
  163.
  164.
  165.
  166.
  167.
  168.
  169.
  170.
  171.
  172.
  173.
  174.
  175.
  176.
  177.
  178.
  179.
  180.
  181.
  182.
  183.
  184.
  185.
  186.
  187.
  188.
  189.
  190.
  191.
  192.
  193.
  194.
  195.
  196.
  197.
  198.
  199.
  200.
  201.
  202.
  203.
  204.
  205.
  206.
  207.
  208.
  209.
  210.
  211.
  212.
  213.
  214.
  215.
  216.
  217.
  218.
  219.
  220.
  221.
  222.
  223.
  224.
  225.
  226.
  227.
  228.
  229.
  230.
  231.
  232.
  233.
  234.
  235.
  236.
  237.
  238.
  239.
  240.
  241.
  242.
  243.
  244.
  245.
  246.
  247.
  248.
  249.
  250.
  251.
  252.
  253.
  254.
  255.
  256.
  257.
  258.
  259.
  260.
  261.
  262.
  263.
  264.
  265.
  266.
  267.
  268.
  269.
  270.
  271.
  272.
  273.
  274.
  275.
  276.
  277.
  278.
  279.
  280.
  281.
  282.
  283.
  284.
  285.
  286.
  287.
  288.
  289.
  290.
  291.
  292.
  293.
  294.
  295.
  296.
  297.
  298.
  299.
  300.
  301.
  302.
  303.
  304.
  305.
  306.
  307.
  308.
  309.
  310.
  311.
  312.
  313.
  314.
  315.
  316.
  317.
  318.
  319.
  320.
  321.
  322.
  323.
  324.
  325.
  326.
  327.
  328.
  329.
  330.
  331.
  332.
  333.
  334.
  335.
  336.
  337.
  338.
  339.
  340.
  341.
  342.
  343.
  344.
  345.
  346.
  347.
  348.
  349.
  350.
  351.
  352.
  353.
  354.
  355.
  356.
  357.
  358.
  359.
  360.
  361.
  362.
  363.
  364.
  365.
  366.
  367.
  368.
  369.
  370.
  371.
  372.
  373.
  374.
  375.
  376.
  377.
  378.
  379.
  380.
  381.
  382.
  383.
  384.
  385.
  386.
  387.
  388.
  389.
  390.
  391.
  392.
  393.
  394.
  395.
  396.
  397.
  398.
  399.
  400.
  401.
  402.
  403.
  404.
  405.
  406.
  407.
  408.
  409.
  410.
  411.
  412.
  413.
  414.
  415.
  416.
  417.
  418.
  419.
  420.
  421.
  422.
  423.
  424.
  425.
  426.
  427.
  428.
  429.
  430.
  431.
  432.
  433.
  434.
  435.
  436.
  437.
  438.
  439.
  440.
  441.
  442.
  443.
  444.
  445.
  446.
  447.
  448.
  449.
  450.
  451.
  452.
  453.
  454.
  455.
  456.
  457.
  458.
  459.
  460.
  461.
  462.
  463.
  464.
  465.
  466.
  467.
  468.
  469.
  470.
  471.
  472.
  473.
  474.
  475.
  476.
  477.
  478.
  479.
  480.
  481.
  482.
  483.
  484.
  485.
  486.
  487.
  488.
  489.
  490.
  491.
  492.
  493.
  494.
  495.
  496.
  497.
  498.
  499.
  500.
  501.
  502.
  503.
  504.
  505.
  506.
  507.
  508.
  509.
  510.
  511.
  512.
  513.
  514.
  515.
  516.
  517.
  518.
  519.
  520.
  521.
  522.
  523.
  524.
  525.
  526.
  527.
  528.
  529.
  530.
  531.
  532.
  533.
  534.
  535.
  536.
  537.
  538.
  539.
  540.
  541.
  542.
  543.
  544.
  545.
  546.
  547.
  548.
  549.
  550.
  551.
  552.
  553.
  554.
  555.
  556.
  557.
  558.
  559.
  560.
  561.
  562.
  563.
  564.
  565.
  566.
  567.
  568.
  569.
  570.
  571.
  572.
  573.
  574.
  575.
  576.
  577.
  578.
  579.
  580.
  581.
  582.
  583.
  584.
  585.
  586.
  587.
  588.
  589.
  590.
  591.
  592.
  593.
  594.
  595.
  596.
  597.
  598.
  599.
  600.
  601.
  602.
  603.
  604.
  605.
  606.
  607.
  608.
  609.
  610.
  611.
  612.
  613.
  614.
  615.
  616.
  617.
  618.
  619.
  620.

# _Related Titles_

Paar, H.H.

**An Introduction to Advanced Quantum Physics**

2010

Print ISBN: 978-0-470-68675-1; also available in electronic formats

Har, J.J., Tamma, K.K.

**Advances in Computational Dynamics of Particles, Materials and Structures**

2012

Print ISBN: 978-0-470-74980-7; also available in electronic formats ISBN: 978-1-119-96589-3

Cohen-Tannoudji, C., Diu, B., Laloe, F.

**Quantum Mechanics**

2 Volume Set

1977

Print ISBN: 978-0-471-56952-7; also available in electronic formats

Schattke, W., Díez Muiño, R.

**Quantum Monte-Carlo Programming**

for Atoms, Molecules, Clusters, and Solids

2013

Print ISBN: 978-3-527-40851-1; also available in electronic formats

Zelevinsky, V.

**Quantum Physics 1 &2**

2 Volume Set

2011

Print ISBN: 978-3-527-41057-6; also available in electronic formats

# Computational Physics

## Problem Solving with Python

3rd completely revised edition

**_Rubin H. Landau_**

**_Manuel J. Páez_**

**_Cristian C. Bordeianu_**

# Authors

**_Rubin H. Landau_**

Oregon State University
97331 Corvallis OR
United States

**_Manuel J. Páez_**
Universad de Antioquia
Departamento Fisica
Medellin
Colombia

**_Cristian C. Bordeianu_**

National Military College "Ştefan cal Mare"
Campulung Moldovenesc
Romania
All books published by Wiley-VCH are carefully produced. Nevertheless, authors, editors, and publisher do not warrant the information contained in these books, including this book, to be free of errors. Readers are advised to keep in mind that statements, data, illustrations, procedural details or other items may inadvertently be inaccurate.

**Library of Congress Card No.:**
applied for

**British Library Cataloguing-in-Publication Data:**

A catalogue record for this book is available from the British Library.

**Bibliographic information published by the Deutsche Nationalbibliothek**

The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data are available on the Internet at <http://dnb.d-nb.de>.

© 2015 WILEY-VCH Verlag GmbH & Co. KGaA,
Boschstr. 12, 69469 Weinheim, Germany

All rights reserved (including those of translation into other languages). No part of this book may be reproduced in any form – by photoprinting, microfilm, or any other means – nor transmitted or translated into a machine language without written permission from the publishers. Registered names, trademarks, etc. used in this book, even when not specifically marked as such, are not to be considered unprotected by law.

**Print ISBN** 978-3-527-41315-7

**ePDF ISBN** 978-3-527-68466-3

**ePub ISBN** 978-3-527-68469-4

**Mobi ISBN** 978-3-527-68467-0
To the memory of Jon Maestri

# Preface

Seventeen years have past since Wiley first published Landau and Páez's _Computational Physics_ and twelve years since Cristian Bordeianu joined the collaboration for the second edition. This third edition adheres to the original philosophy that the best way to learn computational physics (CP) is by working on a wide range of projects using the text and the computer as partners. Most projects are still constructed using a computational, scientific problem-solving paradigm:

(0.1)

Our guiding hypothesis remains that CP is a computational science, which means that to understand CP you need to understand some physics, some applied mathematics, and some computer science. What is different in this edition is the choice of Python for sample codes and an increase in the number of topics covered. We now have a survey of CP which is more than enough for a full-year's course.

The use of Python is more than just a change of language, it is taking advantage of the Python ecosystem of base language plus multiple, specialized libraries to provide all computational needs. In addition, we find Python to be the easiest and most accessible language for beginners, while still being excellent for the type of interactive and exploratory computations now popular in scientific research. Furthermore, Python supplemented by the Visual package (VPython) has gained traction in lower division physics teaching, and this may serve as an excellent segue to a Python-based CP course. Nevertheless, the important aspects of computational modeling and thinking transcends any particular computer language, and so having a Python alternative to our previous use of Fortran, C and Java may help promote this view (codes in all languages are available).

As before, we advocate for the use of a compiled or interpreted programming language when learning CP, in contrast to a higher level problem-solving environment like Mathematica or Maple, which we use in daily work. This follows from our experiences that if you want to _understand_ how to compute scientifically, then you must look inside a program's black box and get your hands dirty. Otherwise, the algorithms, logic, and the validity of solutions cannot be ascertained, and that is not a good physics. Not surprisingly, we believe all physicists should know how to read programs how to write them as well.

Notwithstanding our beliefs about programming, we appreciate how time-consuming and frustrating debugging programs often is, and especially for beginners. Accordingly, rather than make the learner write all codes from scratch, we have placed a large number of codes within the text and often ask the learner only to run, modify, and extend them. This not only leaves time for exploration and analysis, but also provides experience in the modern work environment in which one must incorporate new developments into the preexisting codes of others. Be that as it may, for this edition we have added problems in which the relevant codes are not in the text (but are available to instructors). This should permit an instructor to decide on the balance of new and second-hand codes with which their students should work.

In addition to the paper version of the text, there is also an eBook of it that incorporates many of the multimodal enhancements possible with modern technologies: video lecture modules, active simulations, editable codes, animations, and sounds. The eBook is available as a Web (HTML5) document appropriate for both PCs or mobile devices. The lecture modules, which can be viewed separately from the eBook, cover most of the topics in the text, are listed in Appendix B, and are available online. They may provide avenues for alternative understanding the text (either as a preview or a review), for an online course, or for a blended course that replaces some lecture time with lab time. This latter approach, which we recommend, provides time for the instructor to assist students more personally with their projects and their learning issues. The studio-produced lectures are truly "modules," with active slides, a dynamic table of context, excellent sound (except maybe for a Bronx accent), and with occasional demonstrations replacing the talking head.

The introductory chapter includes tables listing all of the problems and exercises in the text, their locations in the text, as well as the physics courses in which these problems may be used as computational examples. Although we think it better to have entire courses in CP rather than just examples in the traditional courses, the inclusion of examples may serve as a valuable first step towards modernization.

The entire book has been reedited to improve clarity and useability. New materials have also been added, and this has led to additional and reorganized chapters. Specific additions not found in the second edition include: descriptions of the Python language and its packages, demonstrations of several visualization packages, discussions of algebraic tools, an example on protein folding, a derivation of the Gaussian quadrature rule, searching to obtain the temperature dependence of magnetization, chaotic weather patterns, planetary motion, matrix computing with Numerical Python, expanded and updated discussion of parallel computing including scalability and domain composition, optimized matrix computing with NumPy, GPU computing, CUDA programming, principal components analysis, digital filtering, the fast Fourier transform (FFT), an entire chapter on wavelet analysis and data compression, a variety of predator–prey models, signals of chaos, nonlinear behavior of double pendulum, cellular automata, Perlin noise, ray tracing, Wang–Landau sampling for thermodynamic simulations, finite _element_ (in addition to _difference)_ solutions of 1D and 2D PDEs, waves on a catenary, finite-difference-time-domain solutions for E&M waves, advection and Shock waves in fluids, and a new chapter on fluid dynamics. We hope you enjoy it all!

Redmond, Oregon, June 2014

_RHL,_ rubin@science.oregonstate.edu

# Acknowledgments

_Immature poets imitate;
mature poets steal.
T.S. Elliot_

This book and the courses it is based upon could not have been created without continued financial support from the National Science Foundation's CCLI, EPIC, and NPACI programs, as well as support from the Oregon State University. Thank you all and we hope we have done you proud.

Our CP developments have followed the pioneering path paved by the books of Thompson, Gould and Tobochnik, Koonin and Press _et al.;_ indubitably, we have borrowed material from them and made it our own with no further thought. We wish to acknowledge valuable contributions by Hans Kowallik, Sally Haerer (video-lecture modules), Paul Fink, Michel Vallières, Joel Wetzel, Oscar A. Restrepo, Jaime Zuluaga, Pavel Snopok, and Henri Jansen. It is our pleasure to acknowledge the invaluable friendship, encouragement, helpful discussions, and experiences we have had with many colleagues and students over the years. We are particularly indebted to Guillermo Avendaño-Franco, Saturo S. Kano, Melanie Johnson, Jon Maestri (deceased), David McIntyre, Shashikant Phatak, Viktor Podolskiy, C.E. Yaguna, Zlatco Dimcovic, and Al Stetz. The new work on principal component analysis resulted from a wonderful collaboration with Jon Wright and Roy Schult in 1997. Our gratitude also goes to the reviewers for their thoughtful and valuable suggestions, and to Bruce Sherwood, who has assisted us in making the Python codes run faster and look better. And finally, Martin Preuss, Nina Stadthaus, Ann Seidel, and Vera Palmer at Wiley-VCH have been a pleasure to work with.

In spite of everyone's best efforts, there are still errors and confusing statements in the book and codes for which we are to blame.

Finally, we extend our gratitude to the wives, Jan and Lucia, whose reliable support and encouragement are lovingly accepted, as always.

# 1
Introduction

> Beginnings are hard.
>
> _Chaim Potok_

> Nothing is more expensive than a start.
>
> _Friedrich Nietzsche_

This book is really two books. There is a rather traditional paper one with a related Web site, as well as an eBook version containing a variety of digital features best experienced on a computer. Yet even if you are reading from paper, you can still avail yourself of many of digital features, including video-based lecture modules, via the book's Web sites: <http://physics.oregonstate.edu/~rubin/Books/CPbook/eBook/Lectures/> and www.wiley.com/WileyCDA.

We start this chapter with a description of how computational physics (CP) fits into physics and into the broader field of computational science. We then describe the subjects we are to cover, and present lists of all the problems in the text and in which area of physics they can be used as computational examples. The chapter finally gets down to business by discussing the Python language, some of the many packages that are available for Python, and some detailed examples of the use of visualization and symbolic manipulation packages.

## 1.1 Computational Physics and Computational Science

This book presents computational physics (CP) as a subfield of computational science. This implies that CP is a multidisciplinary subject that combines aspects of physics, applied mathematics, and computer science (CS) (Figure 1.1a), with the aim of solving realistic and ever-changing physics problems. Other computational sciences replace physics with their discipline, such as biology, chemistry, engineering, and so on. Although related, computational science is _not_ part of computer science. CS studies computing for its own intrinsic interest and develops the hardware and software tools that computational scientists use. Likewise, applied mathematics develops and studies the algorithms that computational scientists use. As much as we also find math and CS interesting for their own sakes, our focus is on helping the reader do better physics for which you need to understand the CS and math well enough to solve your problems correctly, but not to become an expert programmer.

**Figure 1.1** (a) A representation of the multi-disciplinary nature of computational physics as an overlap of physics, applied mathematics and computer science, and as a bridge among them. (b) Simulation has been added to experiment and theory as a basic approach in the search for scientific truth. Although this book focuses on simulation, we present it as part of the scientific process.

As CP has matured, we have come to realize that it is more than the overlap of physics, computer science, and mathematics. It is also a bridge among them (the central region in Figure 1.1a) containing core elements of it own, such as computational tools and methods. To us, CP's commonality of tools and its problem-solving mindset draws it toward the other computational sciences and away from the subspecialization found in so much of physics. In order to emphasize our computational science focus, to the extent possible, we present the subjects in this book in the form of a _Problem_ to solve, with the components that constitute the solution separated according to the scientific problem-solving paradigm (Figure 1.1b). In recent times, this type of problem-solving approach, which can be traced back to the post-World War II research techniques developed at US national laboratories, has been applied to science education where it is called something like _computational scientific thinking._ This is clearly related to what the computer scientists more recently have come to call _Computational Thinking_ , but the former is less discipline specific. Our computational scientific thinking is a hands-on, inquiry-based project approach in which there is problem analysis, a theoretical foundation that considers computability and appropriate modeling, algorithmic thinking and development, debugging, and an assessment that leads back to the original problem.

Traditionally, physics utilizes both experimental and theoretical approaches to discover scientific truth. Being able to transform a theory into an algorithm requires significant theoretical insight, detailed physical and mathematical understanding, and a mastery of the art of programming. The actual debugging, testing, and organization of scientific programs are analogous to experimentation, with the numerical simulations of nature being virtual experiments. The synthesis of numbers into generalizations, predictions, and conclusions requires the insight and intuition common to both experimental and theoretical science. In fact, the use of computation and simulation has now become so prevalent and essential a part of the scientific process that many people believe that the scientific paradigm has been extended to include simulation as an additional pillar (Figure 1.1b). Nevertheless, as a science, CP must hold experiment supreme, regardless of the beauty of the mathematics.

## 1.2 This Book's Subjects

This book starts with a discussion of Python as a computing environment and then discusses some basic computational topics. A simple review of computing hardware is put off until Chapter 10, although it also fits logically at the beginning of a course. We include some physics applications in the first third of this book, by put off most CP until the latter two-thirds of the book.

This text have been written to be accessible to upper division undergraduates, although many graduate students without a CP background might also benefit, even from the more elementary topics. We cover both ordinary and partial differential equation (PDE) applications, as well as problems using linear algebra, for which we recommend the established subroutine libraries. Some intermediate-level analysis tools such as discrete Fourier transforms, wavelet analysis, and singular value/principal component decompositions, often poorly understood by physics students, are also covered (and recommended). We also present various topics in fluid dynamics including shock and soliton physics, which in our experience physics students often do not see otherwise. Some more advanced topics include integral equations for both the bound state and (singular) scattering problem in quantum mechanics, as well as Feynman path integrations.

A traditional way to view the materials in this text is in terms of its use in courses. In our classes (CPUG, 2009), we have used approximately the first third of the text, with its emphasis on computing tools, for a course called _Scientific Computing_ that is taken after students have acquired familiarity with some compiled language. Typical topics covered in this one-quarter course are given in Table 1.1, although we have used others as well. The latter two-thirds of the text, with its greater emphasis on physics, has typically been used for a two-quarter (20-week) course in CP. Typical topics covered for each quarter are given in Table 1.2. What with many of the topics being research level, these materials can easily be used for a full year's course or for extended research projects.

The text also uses various symbols and fonts to help clarify the type of material being dealt with. These include:

  | Optional material
---|---
`**Monospace font**` | Words as they would appear on a computer screen
_Vertical gray line_ | Note to reader at the beginning of a chapter saying

**Table 1.1** Topics for one-quarter (10 Weeks) scientific computing course.

**Week** | **Topics** | **Chapter** | **Week** | **Topics** | **Chapter**
---|---|---|---|---|---
1 | OS tools, limits | 1, (10) | 6 | Matrices, _N-D_ search | 6
2 | Visualization, Errors | 1, 3 | 7 | Data fitting | 7
3 | Monte Carlo, | 4, 4 | 8 | ODE oscillations | 8
4 | Integration, visualization | 5, (1) | 9 | ODE eigenvalues | 8
5 | Derivatives, searching | 5, 7 | 10 | Hardware basics | 10

**Table 1.2** Topics for two-quarters (20 Weeks) computational physics course.

**Computational Physics I** | **Computational Physics II**
---|---
**Week** | **Topics** | **Chapter** | **Week** | **Topics** | **Chapter**
1 | Nonlinear ODEs | 8, 9 | 1 | Ising model, Metropolis | 17
2 | Chaotic scattering | 9 | 2 | Molecular dynamics | 18
3 | Fourier analysis, filters | 12 | 3 | Project completions | **–**
4 | Wavelet analysis | 13 | 4 | Laplace and Poisson PDEs | 19
5 | Nonlinear maps | 14 | 5 | Heat PDE | 19
6 | Chaotic/double pendulum | 15 | 6 | Waves, catenary, friction | 21
7 | Project completion | 15 | 7 | Shocks and solitons | 24
8 | Fractals, growth | 16 | 8 | Fluid dynamics | 25
9 | Parallel computing, MPI | 10, 11 | 9 | Quantum integral equations | 26
10 | More parallel computing | 10, 11 | 10 | Feynman path integration | 17

## 1.3 This Book's Problems

For this book to contribute to a successful learning experience, we assume that the reader will work through what we call the _Problem_ at the beginning of each discussion. This entails studying the text, writing, debugging, and running programs, visualizing the results, and then expressing in words what has been performed and what can be concluded. As part of this approach, we suggest that the learner write up a mini lab report for each problem containing sections on

Equations solved | Numerical method | Code listing
---|---|---
Visualization | Discussion | Critique

Although we recognize that programming is a valuable skill for scientists, we also know that it is incredibly exacting and time-consuming. In order to lighten the workload, we provide "bare bones" programs. We recommend that these be used as guides for the reader's own programs, or tested and extended to solve the problem at hand. In any case, they should be understood as part of the text.

While we think it is best to take a course, or several courses, in CP, we recognize that this is not always possible and some instructors may only be able to include some CP examples in their traditional courses. To assist in this latter endeavor, in this section we list the location of each problem distributed throughout the text and the subject area of each problem. Of course this is not really possible with a multidisciplinary subject like CP, and so there is an overlap. The code used in the table for different subjects is: QM = quantum mechanics or modern physics, CM = classical mechanics, NL = nonlinear dynamics, EM = electricity and magnetism, SP = statistical physics, MM = mathematical methods as well as tools, FD = fluid dynamics, CS = computing fundamentals, Th = thermal physics, and BI = biology. As you can see from the tables, there are many problems and exercises, which reflects our view that you learn computing best by doing it, and that many problems cover more than one subject.

**Problems and exercises in computational basics**
---
**Subject** | **Section** | **Subject** | **Section** | **Subject** | **Section**
MM, CS | 1.6 | CS | 2.2.2 | CS | 2.2.2
CS | 2.4.3 | CS | 2.4.5 | CS | 2.5.2
CS | 3.1.2 | CS | 3.2 | CS | 3.2.2
CS | 3.3 | CS | 3.3.1 | CS | 4.2.2
MM, CS | 6.6 | CS | 10.13.1 | CS | 10.14.1
CS | 11.3.1 | CS | 11.1.2 | CS | 11.2.1
**Problems and exercises in thermal physics and statistical physics**
---
**Subject** | **Section** | **Subject** | **Section** | **Subject** | **Section**
SP, MM | 4.3 | SP, MM | 4.5 | QM, SP | 4.6
Th, SP | 7.4 | Th, SP | 7.4.1 | NL, SP | 16.3.3
NL, SP | 16.4.1 | NL, SP | 16.7.1 | NL, SP | 16.7.1
NL, SP | 16.8 | NL, SP | 16.11 | SP, QM | 17.4.1
SP, QM | 17.4.2 | SP, QM | 17.6.2 | Th, MM | 20.2.4
Th, MM | 20.3 | TH, MM | 20.4.2 | TH, MM | 20.1
TH, MM | 17.1 | SP | 16.2 | SP, BI | 16.3
SP | 16.4 | SP, MM | 16.5 | SP | 16.6
SP | 16.7 |  |  |  |
**Problems and exercises in electricity and magnetism**
---
**Subject** | **Section** | **Subject** | **Section** | **Subject** | **Section**
EM, MM | 19.6 | EM, MM | 19.7 | EM, MM | 19.8
EM, MM | 19.9 | EM, MM | 23.2 | EM, MM | 23.5
EM, MM | 23.5.1 | EM, MM | 23.6.6 | EM, MM | 22.7.2
EM, MM | 22.10 | EM, MM | 19.2 |  |
**Problems and exercises in quantum mechanics**
---
**Subject** | **Section** | **Subject** | **Section** | **Subject** | **Section**
QM, SP | 4.6 | QM, MM | 7.1 | QM, MM | 7.2.1
QM, MM | 7.3.2 | QM, MM | 9.1 | QM, MM | 9.2
QM, MM | 9.2.1 | QM, MM | 9.3 | QM | 13.6.3
QM, MM | 17.7 | QM, MM | 26.1 | QM, MM | 26.3
QM, MM | 22.1 |  |  |  |
**Problems and exercises in classical mechanics and nonlinear dynamics**
---
**Subject** | **Section** | **Subject** | **Section** | **Subject** | **Section**
CM, NL | 5.16 | CM | 6.1 | CM, NL | 8.1
CM, NL | 8.7.1 | CM, NL | 8.8 | CM, NL | 8.9
CM, NL | 8.10 | CM, NL | 9.4 | CM, NL | 9.4.3
CM | 9.5 | CM | 9.7 | CM | 9.7
NL, FD | 9.7 | CM | 9.7 | CM, MM | 6.6.2
CM, MM | 6.6.1 | CM, NL | 12.1 | BI, NL | 14.3
CM, MM | 6.6.1 | BI, NL | 14.4 | BI, NL | 14.5.2
BI, NL | 14.5.3 | BI, NL | 14.10 | BI, NL | 14.11.1
BI, NL | 14.11.4 | BI, NL | 14.11.5 | CM, NL | 15.1.3
CM, NL | 15.1 | NL, BI | 14.1 | NL, BI | 14.9
CM, NL | 15.2.2 | CM, NL | 15.3 | CM, NL | 15.4
CM, NL | 15.5 | CM, NL | 15.6 | CM, NL | 15.7
CM, NL | 15.7 | NL, MM | 16.2.1 | NL, MM | 16.3.3
NL, MM | 16.4.1 | NL, MM | 16.5.3 | NL, MM | 16.7.1
NL, MM | 16.7.1 | NL, MM | 16.8 | NL, MM | 16.11
CM, MM | 21.2.4 | CM, MM | 21.3 | CM, MM | 21.4.3
CM, MM | 24.6 | CM, MM | 21.1 | CM, MM | 21.5
**Problems and exercises in fluid dynamics**
---
**Subject** | **Section** | **Subject** | **Section** | **Subject** | **Section**
NL, FD | 9.7 | FD, MM | 24.3.2 | FD, MM | 24.5.3
FD, MM | 24.5.4 | FD, MM | 25.1 | FD, MM | 25.2.3
FD, MM | 25.4.4 | FD, MM | 25.4.5 |  |
**Problems and exercises in mathematical methods and computational tools**
---
**Subject** | **Section** | **Subject** | **Section** | **Subject** | **Section**
MM, CS | 1.6 | MM, SP | 4.3 | SP, MM | 4.3.2
BI, MM | 4.4 | MM, SP | 4.5 | MM | 5.12.3
MM | 5.16 | MM | 5.17.2 | MM | 5.5
MM | 5.5 | QM, MM | 7.1 | QM, MM | 7.2.1
QM, MM | 7.3.2 | MM, QM | 9.1 | QM, MM | 9.2
QM, MM | 9.2.1 | QM, MM | 9.3 | CM, NL | 9.4
MM, CS | 6.6 | CM, MM | 6.6.2 | CM, MM | 6.6.1
MM | 7.5.1 | MM | 7.5.2.1 | MM | 7.8
MM | 7.8.1 | MM | 7.8.2 | MM | 12.3
MM | 12.5.3 | MM | 12.7.1 | MM | 12.11
MM | 13.3.1 | MM | 13.5.2 | MM | 13.6.3
CM, MM | 15.5 | NL, MM | 16.2.1 | NL, MM | 16.3.3
NL, MM | 16.4.1 | NL, MM | 16.5.3 | NL, MM | 16.7.1
NL, MM | 16.7.1 | NL, MM | 16.8 | NL, MM | 16.11
Th, MM | 20.2.4 | Th, MM | 20.3 | TH, MM | 20.4.2
EM, MM | 19.6 | EM, MM | 19.7 | EM, MM | 19.8
EM, MM | 19.9 | EM, MM | 23.5 | EM, MM | 23.5.1
EM, MM | 23.6.6 | CM, MM | 21.2.4 | CM, MM | 21.3
CM, MM | 21.4.3 | QM, MM | 22.2.2 | QM, MM | 22.2.2
QM, MM | 22.2.3 | EM, MM | 22.7.2 | EM, MM | 22.10
FD, MM | 24.3.2 | FD, MM | 24.5.3 | FD, MM | 24.5.4
FD, MM | 25.2.3 | FD, MM | 25.4.4 | FD, MM | 25.4.5
QM, MM | 26.2.3 | QM, MM | 26.2.4 | QM, MM | 26.4.5
QM, MM | 26.4.6 | MM, NL | 13.1 | MM, CM | 12.1
MM | 12.6 | MM | 12.8.1 | MM | 7.5
MM | 7.5.2.1 | MM | 7.6 | MM | 7.8.2
MM | 13.7.2 |  |  |  |
BI, MM | 4.4 | BI, NL | 14.3 | BI, NL | 14.4
BI, NL | 14.5.2 | BI, NL | 14.5.3 | BI, NL | 14.10
BI, NL | 14.11.1 | BI, NL | 14.11.4 | BI, NL | 14.11.5
SP, BI | 16.3 | BI, NL | 14.1 | BI, NL | 14.9
MD, QM | 18.3 | MD, QM | 18.4 | MD | 18.4
MM, SP | 18 |  |  |  |

## 1.4 This Book's Language: The Python Ecosystem

The codes in this edition of _Computational Physics_ employ the computer language _Python._ Previous editions have had their examples in Java, Fortran and C, and used post-simulation tools for visualization. Although we have experienced no general agreement in the computational science community as to the best language for scientific computing, this has not stopped many of the users of each language from declaring it to be the best. Even so, we hereby declare that we have found Python to be the best language yet for teaching CP. Python is free, robust (not easily broken), portable (program run without modifications on various devices), universal (available for most every computer system), has a clean syntax that lets students learn the language quickly, has dynamic typing and high-level, built-in data types that enable getting programs to work quickly without having to declare data types or arrays, count matching braces, or use separate visualization programs. Because Python is interpreted, students can learn the language by executing and analyzing individual commands within an interactive shell, or by running the entire program in one fell swoop. Furthermore, Python brings to scientific computing the availability of a myriad of free packages supporting numerical algorithms, state-of the art, or simple, visualizations and specialized toolkits that rival those in Matlab and Mathematica/Maple. And did we mention, all of this is free?

There are literally thousands of Python packages available, but not to worry, we use only a few for numerical and visualization purposes. Because it is essential to be able to run and modify the example codes in this book, we suggest that you spend the time necessary to get Python to function properly on your computer (and then leave notes as to what you did). For learning Python, we recommend the online tutorials (Ptut, 2014; Pguide, 2014; Plearn, 2014), the books by Langtangen Langtangen (2008) and Langtangen (2009), and the _Python Essential Reference_ (Beazley, 2009). For general numerical methods, a book by Press _et al_. (1994) is the standard, and most fun to read, while the NIST Digital Library of Mathematical Functions (NIST, 2014) is probably the most convenient.

Python has developed rapidly since its first implementation in December 1989 (History, 2009). Python's combination of language plus packages is now the standard for the explorative and interactive computing that typifies the present-day scientific research. These rapid developments of Python have also led to a succession of new versions, and the inevitable incompatibilities. Most of the codes in this book were written using Python 2, which was released in 2000, and specifically Python 2.6 with the Visual package (also known as "VPython"). However, there have been major changes to the Python development process as well as in features, and this has led to the release of Python 3.0 in December 2008. Unfortunately, some of the changes in Python 3 were not backward compatible with Python 2.6 and 2.7, and so advances in both Python 2 and 3 and their associated packages have been occurring in parallel. (For our codes, the major difference is in the print statement using a parenthesis in 3, which is not hard to correct.) Furthermore, there have been new versions of operating systems and processors from 32- to 64-bit CPUs, and this also has led to the variety of Python versions and associated packages.

To be honest, we have sometimes felt frustrated by these changes and resulting incompatibilities; however, we are intent on not sharing that! While we will describe the packages and distribution briefly, we indicate here that we have adapted to the real world by having both independent Python 2 and 3 implementations exist on our computers. Specifically, our Visual package programs use Python 3.2, while the others use the _Enthought Canopy Distribution Version 1.3.0_ , which at present uses _Python 2.7.3._ (The Visual package is not available in Enthought.)

### 1.4.1 Python Packages (Libraries)

The Python language plus its family of packages comprise a veritable ecosystem for computing. A package or module is a collection of related methods or classes of methods that are assembled together into a subroutine library.1) Inclusion of the appropriate packages extends the language to meet the specialized needs of various science and engineering disciplines, and lets one obtain state-of-the-art computing for free. In fact, the May/June 2007 and March/April 2011 issues of _Computing in Science and Engineering_ (Perez _et al._ , 2010) focus on scientific computing with Python, and we recommend them.

To use a package named `**PackageName**`, you include in your Python program either an `**import PackageName**` or a `**from PackageName**` statement at the beginning of your program. The import statement loads the entire package, which is efficient, but may require you to include the package name as a prefix to the method you want. For example,

Here   represents the prompt for a Python shell. Some of the typing can be avoided by assigning a symbol to the package name:

There is also a starred version of `**from**` that copies _all_ of the methods of a package (here Matplotlib called pylab) so that you can leave off prefixes:

### 1.4.2 This Book's Packages

We are about to describe some of the packages that make Python such a rich environment. If you are anxious to get started now, or worry about getting overwhelmed by the Python packages, you may just want to load _VPython_ now and move on to the next chapter. You will need some more stuff to do visualizations and matrices, but you can always upgrade your knowledge when you feel more comfortable with Python.

Because all too often you do not know what you do not know, or what you need to know, we list here a few, basic Python packages and what each does. The packages used in the text are underlined and described more fully later.

**Boost.Python** A C++ library that enables seamless interoperability between C++ and Python, thus extending the lifetime of legacy codes and making use of the speed of C, www.boost.org/doc/libs/1_55_0/libs/python/doc/.

**Cython: C Extensions for Python** A superset of the Python language that supports calling C functions and intermixing Python and C for legacy purposes and for high performance, <http://cython.org/>.

**f2py: Fortran to Python Interface Generator** that provides connection between Python and Fortran languages; great for steering legacy codes, <http://cens.ioc.ee/projects/f2py2e/>.

_**IPython: Interactive Python**_ An advanced _shell_ (command line interpreter) that extends Python's basic interpreter IDLE. IPython has enhanced interactivity and interactive visualization capabilities that encourage exploratory computing. IPython also has a browser-based notebook like Mathematica that permits embedded code executions, as well as capabilities for parallel computing, <http://ipython.org/>.

_**Matplotlib: Mathematics Plotting Library**_ A 2D and 3D graphics library that uses NumPy (Numerical Python), and produces publication quality figures in a variety of hard copy formats, and permits interactive graphics. Similar to MATLAB's plotting (except Matplotlib is free and doesn't need its license renewed yearly). See Section 1.5.3 for examples and discussion, <http://matplotlib.sf.net>.

**Mayavi** Interactive and simplified 3D visualization. Also contains TVTK, a wrapper for the more basic Visualization Tool Kit VTK. ("Mayavi" is Sanskrit for magician.) See Section 1.5.6 for examples and discussion, <http://mayavi.sf.net>.

**Mpmath: Multiprecision Floating Point Arithmetic** A pure-Python library for multiprecision floating-point arithmetic for transcendental functions, unlimited exponent sizes, complex numbers, interval arithmetic, numerical integration and differentiation, root-finding, linear algebra, and more, <https://code.google.com/p/mpmath/>.

_**NumPy: Numerical Python**_ Permits the use of fast, high-level multidimensional arrays in Python, which are used as the basis for many of the numerical procedures in Python libraries (NumPy, 2013; SciPy, 2014) – the successor to both _Numeric_ and _Numarray._ Used by Visual and Matplotlib. _SciPy_ extends NumPy. See Sections 6.5, 6.5.1, and 11.2 for examples of NumPy array use.

**Pandas: Python Data Analysis Library** A collection of high-performance, user-friendly data structures and data analysis tools, <http://pandas.pydata.org/>.

**PIL: Python Imaging Library** Image processing and graphics routines for various file formats, www.pythonware.com/products/pil/.

_**Python**_ The Python standard library, <http://python.org>.

**PyVISA** Wrappers for the VISA library providing controls for measurement equipment through various busses from within Python programs, <http://pyvisa.readthedocs.org/en/latest/>.

**SciKits: SciPy Toolkits** A collection of toolkits that extend SciPy to special disciplines such as audio processing, financial computation, geosciences, time series analysis, computer vision, engineering, machine learning, medical computing, and bioinformatics, https://scikits.appspot.com/.

**SciPy: Scientific Python** A basic library for mathematics, science, and engineering. (See SciKits for further extensions.) Provides user-friendly and efficient numerical routines for linear algebra, optimization, integration, special functions, signal and image processing, statistics, genetic algorithms, ODE solutions, and others. Uses NumPy's _N_ -dimensional arrays but also extends NumPy. SciPy essentially provides wrapper for many existing libraries in other languages, such as LAPACK (Anderson _et al._ , 2013) and FFT. The SciPy distribution usually includes Python, NumPy, and f2py, <http://scipy.org>.

**Sphinx** Python documentation generator for output in various formats, <http://sphinx-doc.org/>.

**SWIG** An interface compiler that connects programs written in C and C++ with scripting languages such as Perl, Python, Ruby, and TCL. Useful for extending the lifetime of legacy codes or for making use of the speed of C, <http://swig.org>.

**SyFi: Symbolic Finite Elements** Built on top of the symbolic math library GiNaC, SyFi is used in the finite element solution of PDEs. It provides polygonal domains, polynomial spaces and degrees of freedom as symbolic expressions that are easily manipulated, <https://pypi.python.org/pypi/SyFi/>.

**SymPy: Symbolic Python** A system for symbolic mathematics using pure Python (no external libraries) to provide a simple computer algebra system that also includes calculus, differential equations, etc. Similar to Maple or Mathematica, with the Sage package being even more complete. Examples in Section 1.7. See also mpmath, <http://sympy.org/>.

**VisIt** Distributed, parallel, visualization tool for visualizing data defined on 2D and 3D structured and unstructured meshes, <https://wci.llnl.gov/codes/visit/>.

_**Visual (VPython)**_ Python programming language plus the _visual_ 3D graphics module, with the _VIDLE_ interactive shell replacing Python's standard _IDLE._ Particularly helpful, even for novices, in creating 3D demonstrations and animations for education. We often use Visual for 2D plots of numerical data and animations. Can be installed separately from Canopy, <http://vpython.org/>.

### 1.4.3 The Easy Way: Python Distributions (Package Collections)

Although most Python packages are free, there is a true value for both users and vendors to distribute a collection of packages that have been engineered and tuned to work well together, and that can be installed in one fell swoop. (This is similar to what Red Hat and Debian do with Linux.) These distributions can be thought of as complete Python ecosystems assembled for specific purposes, and are highly recommended. Here we mention four with which we are familiar:

**Anaconda** A free Python distribution including more than 125 packages for science, mathematics, engineering, and data analysis, including Python, NumPy, SciPy, Pandas, IPython, Matplotlib, Numba, Blaze, and Bokeh. Anaconda is self-described as enterprise-ready for large-scale data processing, predictive analytics, and scientific computing, and permits easy switching between Python 2.6, 2.7, and 3.3. As also true for Canopy, Anaconda installs in its own directory and so runs independently from other Python installations on your computer, <https://store.continuum.io/cshop/anaconda/>.

_**Enthought Canopy**_ A comprehensive and complete Python analysis environment with easy installation and updates. The commercial distribution includes more than 150 packages, yet is available for free to academic users. In any case, there is an Express version containing more than 50 packages that is free to everyone. The packages include the IPython, NumPy, SciPy, Matplotlib, Mayavi, scikit, SymPy, Chaco, Envisage, and Pandas, /https://www.enthought.com/products/canopy/.

**Python XY** A free scientific and engineering development collection of packages for numerical computations, data analysis, and data visualization employing the Qt graphical libraries for GUI development and the Spyder interactive scientific development environment, https://code.google.com/p/pythonxy/.

**Sage** An amazingly complete collection of open-source packages for mathematical computations, both numerically and symbolically using the IPython interface and notebooks. Sage's stated mission is to create a viable, free, open-source alternative to Magma, Maple, Mathematica, and Matlab, www.sagemath.org/.

## 1.5 Python's Visualization Tools

> _If I can't picture it, I can't understand it._
>
> _Albert Einstein_

In the sections to follow we discuss tools to visualize data produced by simulations and measurements. Whereas other books may choose to relegate this discussion to an appendix, or not to include it at all, we believe that visualization is such an integral part of CP, and so useful for your work in the rest of this book, that we have placed it here, right up front. We describe the use of Matplotlib, Visual (VPython), and Mayavi. VPython makes easy 2D plot, solid geometric figures, and animations. Matplotlib makes very nice 3D(surface) plots, while Mayavi can create state-of-the-art visualizations.

**Generalities** One of the most rewarding aspects of computing is visualizing the results of calculations. While in the past this was performed with 2D plots, in modern times it is a regular practice to use 3D (surface) plots, volume rendering (dicing and slicing), animations, and virtual reality (gaming) tools. These types of visualizations are often breathtakingly beautiful and may provide deep insights into problems by letting us see and "handle" the functions with which we are working. Visualization also assists in the debugging process, the development of physical and mathematical intuition, and the all-around enjoyment of work.

In thinking about ways to view your results, keep in mind that the point of visualization is to make the science clearer and to communicate your work to others. Then it follows that you should make all figures as clear, informative, and self-explanatory as possible, especially if you will be using them in presentations without captions. This means labels for curves and data points, a title, and labels on the axes.2) After this, you should look at your visualization and ask whether there are better choices of units, ranges of axes, colors, style, and so on, that might get the message across better and provide better insight. And try to remember that those colors which look great on your monitor may turn into uninformative grays when printed. Considering the complexity of human perception and cognition, there may not be a single best way to visualize a particular data set, and so some trial and error may be necessary to "see" what works best.

**Listing 1.1** **EasyVisual.py** produces two different 2D plot using the Visual package.

### 1.5.1 Visual (VPython)'s 2D Plots

As indicated in the description of packets, VPython (Python plus the Visual package) is a simple way to get started with Python and visualizations.3) The Visual package is useful for creating 3D solids, 2D plots, and animations. For example, in Figure 1.2, we present two plots produced by the program `**EasyVisual.py**` in Listing 1.1. Notice that the plotting technique is to create first a plot object, and then to add the points to the object, one by one. (In contrast, Matplotlib creates a vector of points and then plots the entire vector.)

**Figure 1.2** Screen dumps of two _x_ – _y_ plots produced by `**EasyVisual.py**` using the Visual package. Plot (a) uses default parameters, while plot (b) uses user-supplied options.

`**EasyVisual.py**` is seen to create two plot objects, `**Plot1**` and `**Plot2**`, with the plot method used to plot each object. `**Plot1**` uses the `**gcurve**` method with no options specified other than the color of the curve (white). We obtain (Figure 1.2a) a connected curve by default, but no labels. In contrast, `**Plot2**` uses the `**gdisplay**` method to set the display characteristics for the plot to follow, and then `**gdots**` to draw the data as points (Figure 1.2b). You use a `**gdisplay**` plotting object before you create a `**gcurve**` graph object, in order to set the size, position, and title for the graph window, to specify titles for the _x_ and _y_ axes, and to specify maximum values for each axis. The `**gdisplay**` arguments are self-explanatory, with the width and height given in pixels. Because we set `**Plot2= gdots()**`, only points are plotted.

**Listing 1.2** **3GraphVisual.py** produces a 2D _x_ – _y_ plot with the Matplotlib and NumPy packages.

Note that the Python codes are listed within shaded boxes with some formatting to improve readability. For example, see Listing 1.2. Note that we have structured the codes so that a line is skipped before major elements like functions, and that indentations indicate structures in Python (where Java and C may use braces).

It is often a good idea to place several plots in the same figure. The program `**3GraphVisual.py**` in Listing 1.2 does that and produces the graph in Figure 1.3a. There are white vertical bars created with `**gvbars,**` red dots created with `**gdots**`, and a yellow curve created with `**gcurve**` (colors appear only as shades of gray in the paper text). Also note in `**3GraphVisual.py**` that we avoid having to include the package name as a prefix to the commands by starting the program with import `**visual.graph as vg**`. This both imports Visual's graphing package and assigns the symbol `**vg**` to `**visual.graph**`.

#### 1.5.1.1 VPython's 3D Objects

**Listing 1.3** **3Dshapes.py** produces a sample of VPython's 3D shapes.

One way to make simulations appear more realistic is to use 3D solid shapes, for example, a sphere for a bouncing ball rather than just a dot. VPython can produce a variety of 3D shapes with one-line commands, as shown in Figure 1.4, and as produced by the code in Listing 1.3. To make the ball bounce, you would need to vary the `**position**` variable according to some kinematic equations.

**Figure 1.3** (a) Output from the program 3GraphVisual.py that places three different types of 2D plots on one graph using Visual. (b) Three frames from a Visual animation of a quantum mechanical wave packet produced with HarmosAnimate.py.

**Figure 1.4** Some 3D shapes created with single commands in VPython.

### 1.5.2 VPython's Animations

Creating animations with Visual is essentially just making the same 2D plot over and over again, with each one at a slightly differing time, and then placing the plots on top of each other. When performed properly, this gives the impression of motion. Several of our sample codes produce animations, for example, `**HarmosAnimate.py**` and `**3Danimate.py**`. Three frames produced by `**HarmosAnimate.py**` are shown in Figure 1.3b. The major portions of these codes deal with the solution of PDEs, which need not concern us yet. The part which makes the animation is simple:

Here `**PlotObj**` is a curve that continually gets built from within a while loop and thus appears to be moving. Note that being able to plot points individually without having to store them all in an array for all times keeps the memory demand of the program quite small and leads to fast programs.

**Listing 1.4** **EasyMatPlot.py** produces a, 2D _x_ – _y_ plot using the Matplotlib package (which includes the NumPy package).

### 1.5.3 Matplotlib's 2D Plots

_Matplotlib_ is a powerful plotting package that lets you make 2D and 3D graphs, histograms, power spectra, bar charts, error charts, scatter plots, and more, all directly from within your Python program. Matplotlib is free, uses the sophisticated numerics of NumPy and LAPACK (Anderson _et al._ , 2013), and, believe it or not, is easy to use. Specifically, Matplotlib uses the NumPy `**array**` (vector) object to store the data to be plotted. In Chapter 6, we talk at more length about NumPy arrays, so you may want to go there soon to understand arrays better.

Matplotlib commands are by design similar to the plotting commands of MATLAB, a commercial problem-solving environment that is particularly popular in engineering. As is true for MATLAB, Matplotlib assumes that you have placed the _x_ and _y_ values that you wish to plot into 1D arrays (vectors), and then plots these vectors in one fell swoop. This is in contrast to Visual, which first creates a plot object and then adds points to the object one by one. Because Matplotlib is not part of standard Python, you must import the entire Matplotlib package, or the individual methods you use, into your program. For example, on line 2 of `**EasyMatPlot.py**` in Listing 1.4 (line numbers are in the dark shading on the right), we import Matplotlib as the `**pylab**` library:

Then, on lines 6 and 7 we calculate and input arrays of the _x_ and _y_ values

As you can see, NumPy's arange method constructs an array covering "a range" between `**Xmax**` and `**Xmin**` in steps of `**DelX**`. Because the limits are floating-point numbers, so also will be the _x_ _i_ 's. And because `**x**` is an array, `**y = -sin(x)*cos(x)**` is automatically one too! The actual plotting is performed on line 14 with a dash "–" used to indicate a line, and `**lw = 2**` to set its width. The result is shown in Figure 1.5a with the desired labels and title. The `**show()**` command produces the graph on your desktop. More commands are given in Table 1.3. We suggest you try out some of the options and types of plots possible.

**Figure 1.5** Matplotlib plots. (a) Output of EasyMatPlot.py showing a simple, _x_ – _y_ plot. (b) Output from GradesMatPlot.py that places two sets of data points, two curves, and unequal upper and lower error bars, all on one plot.

**Table 1.3** Some common Matplotlib commands.

**Command** | **Effect** | **Command** | **Effect**
---|---|---|---
`**plot(x, y, '-', lw=2)**` | _x_ – _y_ curve, line width 2 | `**myPlot.setYRange(-8., 8.)**` | Set _y_ range
`**show()**` | Show output graph | `**myPlot.setSize(500, 400)**` | Size in pixels
`**xlabel('x')**` | _x_ -axis label | `**pyplot.semilogx**` | Semilog _x_ plot
`**ylabel('f(x)')**` | _y_ -axis label | `**pyplot.semilogy**` | Semilog _y_ plot
`**title('f vs x')**` | Add title | `**grid(True)**` | Draw grid
`**text(x, y, 's')**` | Add text _s_ at ( _x_ , _y_ ) | `**myPlot.setColor (false)**` | Black and White
`**myPlot.addPoint (0, x, y, true)**` | Add ( _x_ , _y_ ) to set 0 connect | `**myPlot.setButtons(true)**` | For zoom button
`**myPlot.addPoint (1, x, y, false)**` | Add ( _x_ , _y_ ) to 1, no connect | `**myPlot.fillPlot()**` | Fir ranges to data
`**pyplot.errorbar**` | Point + error bar | `**myPlot.setImpulses(true,0)**` | Vertical lines, set 0
`**pyplot.clf()**` | Clear current figure | `**pyplot.contour**` | Contour lines
`**pyplot. scatter**` | Scatter plot | `**pyplot.bar**` | Bar charts
`**pyplot.polar**` | Polar plot | `**pyplot.gca**` | For current axis
`**my Plot.setXRange(-1., 1.)**` | Set _x_ range | `**pyplot.acorr**` | Autocorrelation

**Listing 1.5** **GradesMatPlot.py** produces a, 2D _x_ – _y_ plot using the Matplotlib package.

In Listing 1.5, we give the code `**GradesMatplot.py**`, and in Figure 1.5b we show its output. This is not a simple plot. Here we repeat the plot command several times in order to plot several data sets on the same graph, and to plot both the data points and the lines connecting them. On line 3, we import Matplotlib (pylab), and on line 4 we import NumPy, which we need for the array command. Because we have imported two packages, we add the `**pylab**` prefix to the plot commands so that Python knows which package to use.

In order to place a horizontal line along _y_ = 0, on lines 10 and 11 we create a data set as an array of _x_ values, −1 ≤ _x_ ≤ 5, and a corresponding array of _y_ values, _y i_ ≡ 0. We then plot the horizontal on line 12. Next we place four more curves in the figure. First on lines 14–15, we create data set 0, and then plot the points as blue circles (` **'bo'**`), and connect the points with green (` **'g'**`) lines (the color will be visible on a computer screen, but will appear only as shades of gray in print). On lines 19–21, we create and plot another data set as a red (` **'r'**`) line. Finally, on lines 23–25, we define unequal lower and upper error bars and place them on the plot. We finish by adding grid lines (line 27) and _showing_ the plot on the screen.

**Listing 1.6** **MatPlot2figs.py** produces the two figures shown in Figure 1.6. Each figure contains two plots with one Matplotlib figure.

Often the science is clearer if there are several curves in one plot, and, several plots in one figures. Matplotlib lets you do this with the `**plot**` and the `**subplot**` commands. For example, in `**MatPlot2figs.py**` in Listing 1.6 and Figure 1.6, we have placed two curves in one plot, and then output two different figures, each containing two plots. The key here is the repetition of the `**subplot**` command:

**Figure 1.6** (a,b) Columns show two separate outputs, each of two figures, produced by MatPlot2figs.py. (We used the slider button to add some space between the plots.)

The listing is self-explanatory, with sections that set the plotting limits, that creates each figure, and then creates the grid.

**Listing 1.7** **PondMatPlot.py** produces the scatter plot and the curve shown in Figure 5.5 in Chapter 5.

**Scatter Plots** Sometimes we need a scatter plot of data points, and maybe even a curve thrown in as well. In Figure 5.5 in Chapter 5, we show such a plot created with the code `**PondMapPlot.py**` in Listing 1.7. The key statements here are of the form `**ax.plot(xo, yo, 'bo', markersize=3)**`, which in this case adds a blue point (on screen) of size 3.

### 1.5.4 Matplotlib's 3D Surface Plots

A 2D plot of the potential  . r is fine for visualizing the radial dependence of the potential field surrounding a single charge, but if you want to visualize a dipole potential such as  , you need a 3D visualization. We get that by creating a world in which the _z_ dimension (mountain height) is the value of the potential, and the _x_ and _y_ axes define the plane below the mountain. Because the surface we are creating is a 3D object, it is not truly possible to draw it on a flat screen, and so different techniques are used to give the impression of three dimensions to our brains. We do that by rotating the object (by grabbing it with your mouse), shading it, employing parallax, and other tricks.

**Listing 1.8** **Simple3Dplot.py** produces the Matplotlib 3D surface plots in Figure 1.7.

**Figure 1.7** (a) A 3D wire frame. (b) A surface plot with wire frame. Both are produced by the program Simple3dplot.py using Matplotlib.

In Figure 1.7a, we show a wire-frame plot and in Figure 1.7b, a surface-plus-wire-frame plot. These are obtained from the program `**Simple3Dplot.py**` in Listing 1.8. Note that there is an extra import of `**Axes3D**` from the Matplotlib tool kit needed for 3D plotting. Lines 8 and 9 are the usual creation of _x_ and _y_ arrays of floats using `**arange**`. Line 11 uses the `**meshgrid**` method to set up the entire coordinate matrix grid from the _x_ and _y_ coordinate vectors with a vector call, and line 12 constructs the entire _Z_ surface with another vector operation. The remaining of the program is self-explanatory, with `**fig**` being the plot object, `**ax**` the 3D axes object, and `**plot_wireframe**` and `**plot_surface**` creating wire frame and surface plots, respectively.

Another type of 3D plot is particularly useful when examining data of the form ( _x i_, _y j_, _z k_), is a scatter plot into the 3D ( _x_ , _y_ , _z_ ) volume. In Listing 1.9, we give the program `**Scatter3dPlot.py**` that created the plot in Figure 1.8. This program, which is taken from the Matplotlib documentation, uses the NumPy random number generator, with the 111 notation being a hand-me-down from MATLAB indicating a 1 × 1 × 1 grid.

**Listing 1.9** **Scatter3dPlot.py** produces a 3D scatter plot using Matplotlib 3D tools.

**Figure 1.8** A 3D scatter plot produced by the program Scatter3dPlot.py using Matplotlib.

Finally, the program `**FourierMatplot.py**`, written by Oscar Restrepo, performs a Fourier reconstruction of a saw tooth wave, with the number of waves included controlled by the viewer via a slider bar, as shown in Figure 1.9. (We discuss the mathematics of Fourier transforms in Chapter 12.) The slider method is included via the extra lines:

### 1.5.5 Matplotlib's Animations

Matplotlib also can do animations, although not as simply as VPython. The Matplotlib example page shows a number of them. We include some Matplotlib animation codes in the PythonCodes/Visualizations directory, and show a sample code for the heat equation in Listing 1.10. Here too most of the code deals with solving a PDE, which need not interest us yet. The animation is carried out at the bottom of the code.

**Figure 1.9** A comparison of a saw tooth function to the sum of its Fourier components, with the number of included waves varied interactively by a Matplotlib slider. FourierMatplot.py, which produced this output, was written by Oscar Restrepo.

**Listing 1.10** **EqHeatAnimateMatPlot.py** produces an animation of a cooling bar using Matplotlib.

### 1.5.6 Mayavi's Visualizations Beyond Plotting*

This section on Mayavi is indicated as optional because we do not use it in our sample programs. However, we recommend that, at least, the reader browse through it in order to obtain some ideas about the next level of Python visualization.

Although Matplotlib is excellent for plotting functions vs. one or two of its variables, it is not designed to do the sculpture-like 3D visualizations of functions of three or more variables that are often displayed by supercomputer centers. Mayavi (Sanskrit for "magician") is designed for this next level of visualization. Mayavi is open source, tightly integrated with Python and included in the Canopy distribution.

Mayavi consists of two different packages and two different interfaces to those packages. The package we illustrate here is the set of Matlab- or Mathematica-like commands that operate at a fairly high level of abstraction and works naturally with NumPy arrays. The other package is a set of VTK (Visual Tool Kit) primitives that may be more appropriate for developing your own, research-specific, visualization modules. Even with the high-level package, you have the choice of interacting with Mayavi via scripting from within your Python program (what we demonstrate) or via a stand-alone application that runs separately from your programs.

We will now show a few examples derived from the Enthought tutorial. We start by having Mayavi produce a standard surface plot of  :

You see here that we use NumPy's `**numpy.mgrid**` method to set up the _X_ and _Y_ arrays, and then set up the Z array with a vectorized evaluation of  . Then we use Mayavi to create the Z surface, to draw the axes and to outline the surface with a box. Finally, there is an important call to `**mlab.show()**` to show the visualization in a display box such as that in Figure 1.10a. This display box is seen (well, if enlarged) to contain a number of (too small) buttons that lets you produce different views and sizes, insert directional arrows, save the file in various formats to disk, edit properties of the visualization, and open the _pipeline_ window. The pipeline window shows, and lets the user control, the various stages of a visualization: loading the data into a data source object, transforming the data with filters, and visualizing it with modules.

**Figure 1.10** (a) A Mayavi surface plot of the function   as seen in the screen viewer. (b) A rotatable visualization of a spherical harmonic   in which the radial distance represents the value of the function.

Now we go beyond the direct plotting of a function's values to the creation of a visualization of a spherical harmonic function   that is defined over the surface of a sphere (Figure 1.10b):

Because we do not have four dimensions to use, we take the values of  at various grid points and plot those values as the radial distances from the origin for each value of _θ_ and _φ_. The new element here is the statement `**s = mlab.mesh(x, y, z)**` that produces a mesh throughout 3D space, and then the projection of the radius into its ( _x, y, z_ ) components.

In the next example, we start with a data set in the form of ( _x i_, _y j_, _z k_) values and connect the points with _tubes_ of various colors:

**Figure 1.11** (a) A Mayavi visualization in which tubes are used to connect a set of data points. (b) A Mayavi visualization using arrows (glyphs) to represent a vector field.

The new command here is `**plot3d**`, which is seen in Figure 1.11b to produce rainbow colored (` **'Spectral'**`) tubes connecting the data points. The `**arange**` command sets up the array of `**phi**` values, and then the arrays of `**x, y, z, mu**` and `**sin(mu)**` values all follow.

A popular style of visualization for vector fields is one in which arrows _(glyphs)_ are drawn at various points in space with the directions of the arrows indicating the directions of the field, and with the length of the arrows indicating its strengths. Here we create such a visualization and show its output in Figure 1.11b:

As before, we use NumPy to set an ( _x_ , _y_ , _z_ ) grid. Then we set up an array of _r_ values as an intermediate function of ( _x_ , _y_ , _z_ ), and finally set up arrays of the ( _u_ , _v_ , _w_ ) components of the vector field as functions of the other arrays. The new command here is `**quiver3d**` which provides a collection of arrows (cute name).

**Figure 1.12** (a) A Mayavi visualization of the contours of the scalar field  . (b) A volume rendering of the same scalar field.

If the field we wish to visualize is a scalar field, such as  , then the appropriate visualization would be an iso-surface (a 3D contour plot of equal values) throughout a 3D space. We do that with the `**contour3d**` command:

Figure 1.12a shows the output, which is periodic, but not obviously trigonometric.

We now take our visualization of the same scalar field and show how some other Mayavi methods yield different views of the field. First, a volume rendering to produce the nebulous view in Figure 1.12b:

Next, we take the same field and replace the `**mlab.contour3d(s)**` command with the pipeline command:

This produces the nebulous visualization in Figure 1.12b. Next, we produce the visualization in Figure 1.13a by having some planes cut through the scalar field:

**Figure 1.13** (a) A Mayavi visualization of the same scalar field   using cut planes. (b) A visualization of the same scalar field combining cut planes and contours.

Although we cannot show it, the user can interact with the visualization by moving the cuts and rotating the figures. And to finish, we place both contours and cut planes in the same plot to produce the interesting visualization as shown in Figure 1.13b.

## 1.6 Plotting Exercises

We encourage you to make your own plots and personalize them by trying out other commands and by including further options in the commands. The Matplotlib documentation is extensive and available on the Web. As an exercise, explore:

  1. how to zoom in and zoom out on sections of a plot?
  2. how to save your plots to files in various formats?
  3. how to print up your graphs?
  4. the options available from the pull-down menus?
  5. how to increase the space between subplots?
  6. and how to rotate and scale the surfaces.

## 1.7 Python's Algebraic Tools

While this book's focus is on the use of Python for numerical simulations, this is not to discount the importance of computational symbolic manipulations (even though that may be the way we feel). Python actually has (at least) two packages that can be used for symbolic manipulations, and they are quite different. As indicated in Section 1.4.3, the _Sage_ package is very much in the same class as Maple and Mathematica, with a notebook graphical interface that lets the user create publication quality text, within which Python programs can be run, or the equations can be manipulated symbolically. Yet Sage is a big and powerful package that goes beyond pure Python by including multiple computer algebra systems as well as visualization tools and more. Using the multiple features of Sage can get to be quite complicated, and, in fact, books have been written and workshops taught on the use of Sage. We refer the interested reader to the online Sage documentation page at www.sagemath.org/help.html.

The _SymPy_ package for symbolic manipulations runs very much like any other Python package from within your regular Python shell. It can be downloaded from <https://github.com/sympy/sympy/releases>, or you can use the Canopy distribution that includes SymPy. Now we give some simple examples of SymPy's use, but you really need to start with the _SymPy Tutorial_ <http://docs.sympy.org/latest/tutorial/> if you want to use SymPy. (Note, despite the fact that we are working within a Python shell, SymPy has automatically found our  application and used it to format the output.) To start, we will take some derivatives to show that SymPy knows calculus:

We see here that we must first import methods from SymPy and then use the `**symbols**` command to declare the variables _x_ and _y_ as algebraic. The rest is rather obvious, with `**diff**` being the derivative operator and the x argument in diff indicating what we are taking the derivative with respect to _x._ Now let us try expansions:

SymPy knows about infinite series and different expansion points:

One of the classic difficulties with computer algebra systems is that even if the answer is correct, if it is not simple, then it probably is not useful. And so, SymPy has a `**simplify**` function as well as a `**factor**` function (and `**collect**`, `**cancel**` and `**apart**` which we will not illustrate):

  1) The Python Package Index (PYPI, 2014), a repository of free Python packages, currently contains more than 40 000 packages! 2) Although this may not need saying, place the independent variable _x_ along the abscissa (horizontal), and the dependent variable _y_ = _f_ ( _x_ ) along the ordinate. 3) Because Visual is not one of the Canopy packages, to run our Visual programs you would need to install the Visual package and the version of Python that runs with it, even if you have Canopy installed. There is no problem doing this because VPython and Canopy go into different folders/directories.

# 2
Computing Software Basics

* * *

This chapter discusses some computing basics starting with computing languages, number representations, and programming tools. The limits and consequences of using floating-point numbers are explored. Related topics dealing with hardware basics are found in Chapter 10.

* * *

## 2.1 Making Computers Obey

> _The best programs are written so that computing machines can perform them quickly and human beings can understand them clearly. A programmer is ideally an essayist who works with traditional aesthetic and literary forms as well as mathematical concepts, to communicate the way that an algorithm works and to convince a reader that the results will be correct._
>
> _Donald E. Knuth_

As anthropomorphic as your view of your computer may be, keep in mind that computers always do exactly as they are told. This means that you must tell them exactly everything they have to do. Of course, the programs you run may have such convoluted logic that you may not have the endurance to figure out the details of what you have told the computer to do, but it is always possible in principle. So your first _problem_ is to obtain enough understanding so that you feel well enough in control, no matter how illusionary, to figure out what the computer is doing.

Before you tell the computer to obey your orders, you need to understand that life is not simple for computers. The instructions they understand are in a _basic machine language_1) that tells the hardware to do things like move a number stored in one memory location to another location or to do some simple binary arithmetic. Very few computational scientists talk to computers in a language computers can understand. When writing and running programs, we usually communicate to the computer through _shells,_ in _high-level languages_ (Python, Java, Fortran, C), or through _problem-solving environments_ (Maple, Mathematica, and Matlab). Eventually, these commands or programs are translated into the basic machine language that the hardware understands.

**Figure 2.1** A schematic view of a computer's kernel and shells. The hardware is in the center surrounded by increasing higher level software.

A _shell_ is a _command-line interpreter,_ that is, a set of small programs run by a computer that respond to the commands (the names of the programs) that you key in. Usually you open a special window to access the shell, and this window is called a shell as well. It is helpful to think of these shells as the outer layers of the computer's operating system (OS) (Figure 2.1), within which lies a _kernel_ of elementary operations. (The user seldom interacts directly with the kernel, except possibly when installing programs or when building an OS from scratch.) It is the job of the shell to run programs, compilers, and utilities that do things like copying files. There can be different types of shells on a single computer or multiple copies of the same shell running at the same time.

Operating systems have names such as _Unix, Linux, DOS, MacOS,_ and _MS Windows._ The _operating system_ is a group of programs used by the computer to communicate with users and devices, to store and read data, and to execute programs. The OS tells the computer what to do in an elementary way. The OS views you, other devices, and programs as input data for it to process; in many ways, it is the indispensable office manager. While all this may seem complicated, the purpose of the OS is to let the computer do the nitty-gritty work so that you can think higher level thoughts and communicate with the computer in something closer to your normal everyday language.

When you submit a program to your computer in a _high-level language_ the computer uses a compiler to process it. The _compiler_ is another program that treats your program as a foreign language and uses a built-in dictionary and set of rules to translate it into basic machine language. As you can probably imagine, the final set of instructions is quite detailed and long and the compiler may make several passes through your program to decipher your logic and translate it into a fast code. The translated statements form an _object_ or compiled code, and when _linked_ together with other needed subprograms, form a load module. A _load module_ is a complete set of machine language instructions that can be _loaded_ into the computer's memory and read, understood, and followed by the computer.

Languages such as _Fortran_ and C use compilers to read your entire program and then translate it into basic machine instructions. Languages such as _BASIC_ and _Maple_ translate each line of your program as it is entered. Compiled languages usually lead to more efficient programs and permit the use of vast subprogram libraries. Interpreted languages give a more immediate response to the user and thereby appear "friendlier." The Python and Java languages are a mix of the two. When you first compile your program, Python interprets it into an intermediate, universal _byte code_ which gets stored as a PYC (or PYO) file. This file can be transported to and used on other computers, although not with different versions of Python. Then, when you run your program, Python recompiles the byte code into a machine-specific compiled code, which is faster than interpreting your source code line by line.

## 2.2 Programming Warmup

Before we go on to serious work, we want to establish that your local computer is working right for you. Assume that calculators have not yet been invented and that you need a program to calculate the area of a circle. Rather than use any specific language, write that program in pseudocode that can be converted to your favorite language later. The first program tells the computer2):

This program cannot really work because it does not tell the computer which circle to consider and what to do with the area. A better program would be

The instruction `**calculate area of circle**` has no meaning in most computer languages, so we need to specify an _algorithm,_ that is, a set of rules for the computer to follow:

This is a better program, and so let us see how to implement it in Python (other language versions are available online). In Listing 2.1, we give a Python version of our Area program. This is a simple program that outputs to the screen, with its input built into the program.

**Listing 2.1** **Area.py** outputs to the screen, with its input built into the program.

### 2.2.1 Structured and Reproducible Program Design

Programming is a written art that blends elements of science, mathematics, and computer science into a set of instructions that permit a computer to accomplish a desired task. And now, as published scientific results increasingly rely on computation as an essential element, it is increasingly important that the source of your program itself be available to others so that they can reproduce and extend your results. Reproducibility may not be as exciting as a new discovery, but it is an essential ingredient in science (Hinsen, 2013). In addition to the grammar of a computer language, a scientific program should include a number of essential elements to ensure the program's validity and useability. As with other arts, we suggest that until you know better, you follow some simple rules. A good program should

  * Give the correct answers.
  * Be clear and easy to read, with the action of each part easy to analyze.
  * Document itself for the sake of readers and the programmer.
  * Be easy to use.
  * Be built up out of small programs that can be independently verified.
  * Be easy to modify and robust enough to keep giving correct answers after modification and simple debugging.
  * Document the data formats used.
  * Use trusted libraries.
  * Be published or passed on to others to use and to develop further.

One attraction of _object-oriented programming_ is that it enforces these rules automatically. An elementary way to make any program clearer is to _structure_ it with indentation, skipped lines, and paranetheses placed strategically. This is performed to provide visual clues to the function of the different program parts (the "structures" in structured programming). In fact, Python uses indentations as structure elements as well as for clarity. Although the space limitations of a printed page keep us from inserting as many blank lines as we would prefer, we recommend that you do as we say and not as we do!

In Figure 2.2, we present basic and _detailed flowcharts_ that illustrate a possible program for computing projectile motion. A flowchart is not meant to be a detailed description of a program but instead a graphical aid to help visualize its logical flow. As such, it is independent of a specific computer language and is useful for developing and understanding the basic structure of a program. We recommend that you draw a flowchart or (second best) write a pseudocode before you write a program. _Pseudocode_ is like a text version of a flowchart that leaves out details and instead focuses on the logic and structures:

### 2.2.2 Shells, Editors, and Execution

  1. To gain some experience with your computer system, use an editor to enter the program `**Area.py**` that computes the area of a circle (yes, we know you can copy and paste it, but you may need some exercise before getting to work). Then write your file to a disk by saving it in your home (personal) directory (we advise having a separate subdirectory for each chapter). _Note:_ Readers familiar with Python may want to enter the program `**AreaFormatted.py**` instead that uses commands that produce the formatted output.
  2. Compile and execute the appropriate version of `**Area.py**`.
  3. Experiment with your program. For example, see what happens if you leave out decimal points in the assignment statement for _r_ , if you assign _r_ equal to a blank, or if you assign a letter to _r_. Remember, it is unlikely that you will "break" anything on the computer by making a mistake, and it is good to see how the computer responds when under stress.
  4. Change the program so that it computes the volume 4/3 π _r_ 3 of a sphere and prints it out with the proper name. Save the modified program to a file it in your personal directory and give it the name `**Vol.py**`.
  5. Open and execute `**Vol.py**` and check that your changes are correct by running a number of trial cases. Good input data are _r_ = 1 and _r_ = 10.
  6. Revise `**Area.py**` so that it takes input from a file name that you have made up, then writes in a different format to another file you have created, and then reads from the latter file.
  7. See what happens when the data type used for output does not match the type of data in the file (e.g., floating point numbers are read in as integers).
  8. Revise `**AreaMod**` so that it uses a main method (which does the input and output) and a separate function or method for the calculation. Check that you obtain the same answers as before.

**Figure 2.2** A flowchart illustrating a program to compute projectile motion. Plot (a) shows the basic components of the program, and plot (b) shows are some of its details. When writing a program, first map out the basic components, then decide upon the structures, and finally fill in the details. This is called _top-down programming._

**Listing 2.2** `**AreaFormatted.py**` does I/O to and from keyboard, as well as from a file. It works with either Python 2 or 3 by switching between _raw input_ and _input._ Note to read from a file using Canopy, you must right click in the Python run window and choose _Change to Editor Directory._

## 2.3 Python I/O

The simplest I/O with Python is outputting to the screen with the print command (as seen in `**Area.py**`, Listing 2.1), and inputting from the keyboard with the `**input**` command (as seen in `**AreaFormatted.py**`, Listing 2.2). We also see in `**AreaFormatted.py**` that we can input strings (literal numbers and letters) by either enclosing the string in quotes (single or double), or by using the `**raw_input**` (Python2) or `**input**` (Python 3) command without quotes. To print a string with `**print**`, place the string in quotes. `**AreaFormatted.py**` also shows how to input both a string and numbers from a file. (But be careful if you are using Canopy: you must right click in the Python run window and choose _Change to Editor Directory_ in order to switch the working directory of Canopy's Python shell to your working directory.) The simplest output prints the value of a float just by giving its name:

This uses Python's default format, which tends to vary depending on the precision of the number being printed. As an alternative, you can control the format of your output. For floats, you need to specify two things. First, how many digits (places) after the decimal point are desired, and second, how many spaces overall should be used for the number:

Here the `**%6.3f**` formats a float (which is a double in Python) to be printed in fixed-point notation (the `**f**`) with three places after the decimal point and with six places overall (one place for the decimal point, one for the sign, one for the digit before the decimal point, and three for the decimal). The directive `**%9.6f**` produces six digits after the decimal place and nine overall.

To print an integer, we need to specify only the total number of digits (there is no decimal part), and we do that with the `**%d**` (d for digits) format. The % symbol in these output formats indicates a conversion from the computer's internal format to that used for output. Notice in Listing 2.2 how we read from the keyboard, as well as from a file, and then output to both the screen and file. Beware that if you do not create the file `**Name.dat**`, the program will issue ("throw") an error message of the sort:

**IOError: [Errno 2] No such file or directory: 'Name.dat'**

Note that we have also use a `**\n**` directive here to indicate a new line. Other directives, some of which are demonstrated `**in Directives.py**` in Listing 2.3 (and some of which like backspace may not yet work right) are:

`**\"**` double quote

`**\a**` alert (bell)

`**\f**` form feed

`**\t**` horizontal tab

`**\0NNN**` octal NNN

`**\b**` backspace

`**\n**` new line

`**\v**` vertical tab

`**\\**` backslash

`**\c**` no more output

`**\r**` carriage ret

`**%%**` a single %

**Listing 2.3** `**Directives.py**` illustrates formatting via directives and escape characters.

## 2.4 Computer Number Representations (Theory)

Computers may be powerful, but they are finite. A problem in computer design is how to represent an arbitrary number using a finite amount of memory space and then how to deal with the limitations arising from this representation. As a consequence of computer memories being based on the magnetic or electronic realizations of a spin pointing up or down, the most elementary units of computer memory are the two binary integers ( _bits_ ) 0 and 1. This means that all numbers are stored in memory in the _binary_ form, that is, as long strings of 0s and 1s. Accordingly, _N_ bits can store integers in the range [0, 2 _N_ ], yet because the sign of the integer is represented by the first bit (a zero bit for positive numbers), the actual range for _N_ -bit integers decreases to [0, 2 _N_ −1].

Long strings of 0s and 1s are fine for computers but are awkward for humans. For this reason, binary strings are converted to _octal, decimal,_ or _hexadecimal_ numbers before the results are communicated to people. Octal and hexadecimal numbers are nice because the conversion maintains precision, but not all that nice because our decimal rules of arithmetic do not work for octals and hexadecimals. Converting to decimal numbers makes the numbers easier for us to work with, but unless the original number is a power of 2, the conversion decreases precision.

A description of a particular computer's system or language normally states the _word length,_ that is, the number of bits used to store a number. The length is often expressed in _bytes,_ (a mouthful of bits) where

Memory and storage sizes are measured in bytes, kilobytes, megabytes, gigabytes, terabytes, and petabytes (1015). Some care should be taken here by those who chose to compute sizes in detail, because K does not always mean 1000:

(2.1)

This is often (and confusingly) compensated for when memory size is stated in K, for example,

Conveniently, 1 byte is also the amount of memory needed to store a single letter like "a," which adds up to a typical printed page requiring ∼3 kB.

The memory chips in some older personal computers used 8-bit words, with modern PCs using 64 bits. This means that the maximum integer was a rather small 27 = 128 (7 because 1 bits is used for the sign). Using 64 bits permits integers in the range 1−263   1019. While at first this may seem like a large range, it really is not when compared to the range of sizes encountered in the physical world. As a case in point, the size of the universe compared to the size of a proton covers a scale of 1041. Trying to store a number larger than the hardware or software was designed for ( _overflow_ ) was common on older machines, but is less so now. An overflow is sometimes accompanied by an informative error message, and sometimes not.

### 2.4.1 IEEE Floating-Point Numbers

Real numbers are represented on computers in either _fixed-point_ or _floating-point_ notation. _Fixed-point notation_ can be used for numbers with a fixed number of places beyond the decimal point (radix) or for integers. It has the advantages of being able to use _two's complement_ arithmetic and being able to store integers exactly.3) In the fixed-point representation with _N_ bits and with a two's complement format, a number is represented as

(2.2)

where _n_ \+ _m_ = _N_ − 2. That is, 1 bit is used to store the sign, with the remaining ( _N_ − 1) bits used to store the _α i_ values (the powers of 2 are understood). The particular values for _N, m,_ and _n_ are machine dependent. Integers are typically 4 bytes (32 bits) in length and in the range

(2.3)

An advantage of the representation (2.2) is that you can count on all fixed-point numbers having the same absolute error of 2− _m_ −1 (the term left off the right-hand end of (2.2)). The corresponding disadvantage is that _small_ numbers (those for which the first string of _α_ values are zeros) have large _relative_ errors. Because relative errors in the real world tend to be more important than absolute ones, integers are used mainly for counting purposes and in special applications (like banking).

Most scientific computations use double-precision floating-point numbers with 64 bits = 8 B. The _floating-point representation_ of numbers on computers is a binary version of what is commonly known as _scientific_ or _engineering notation._ For example, the speed of light _c_ = +2.997 924 58 × 108 m/s in scientific notation and +0.299 792 458 × 109 or 0.299 795 498 E09 m/s in engineering notation. In each of these cases, the number in front is called the _mantissa_ and contains nine _significant figures._ The power to which 10 is raised is called the _exponent,_ with the plus sign in front a reminder that these numbers may be negative.

Floating-point numbers are stored on the computer as a concatenation (juxtaposition) of a sign bit, an exponent, and a mantissa. Because only a finite number of bits are stored, the set of floating-point numbers that the computer can store exactly, _machine numbers_ (the hash marks in Figure 2.3), does not cover the entire the set of real numbers. In particular, machine numbers have a maximum and a minimum (the shading in Figure 2.3). If you exceed the maximum, an error condition known as _overflow_ occurs; if you fall below the minimum, an error condition known as _underflow_ occurs. In the latter case, the software and hardware may be set up so that underflows are set to zero without your even being told. In contrast, overflows usually halt a program's execution.

The actual relation between what is stored in memory and the value of a floating-point number is somewhat indirect, with there being a number of special cases and relations used over the years. In fact, in the past each computer OS and each computer language contained its own standards for floating-point numbers. Different standards meant that the same program running correctly on different computers could give different results. Although the results usually were only slightly different, the user could never be sure if the lack of reproducibility of a test case was as a result of the particular computer being used or to an error in the program's implementation.

**Figure 2.3** The limits of single-precision floating-point numbers and the consequences of exceeding these limits (not to scale). The hash marks represent the values of numbers that can be stored; storing a number in between these values leads to truncation error. The shaded areas correspond to over- and underflow.

In 1987, the Institute of Electrical and Electronics Engineers (IEEE) and the American National Standards Institute (ANSI) adopted the IEEE 754 standard for floating-point arithmetic. When the standard is followed, you can expect the primitive data types to have the precision and ranges given in Table 2.1. In addition, when computers and software adhere to this standard, and most do now, you are guaranteed that your program will produce identical results on different computers. Nevertheless, because the IEEE standard may not produce the most efficient code or the highest accuracy for a particular computer, sometimes you may have to invoke compiler options to demand that the IEEE standard be strictly followed for your test cases. After you know that the code is okay, you may want to run with whatever gives the greatest speed and precision.

**Table 2.1** The IEEE 754 Standard for Primitive Data Types.

Name | Type | Bits | Bytes | Range
---|---|---|---|---
`**boolean**` | Logical | 1 |   | **true or false**
`**char**` | String | 16 | 2 | `**'\u0000' ↔ '\uFFFF' (ISO Unicode)**`
`**byte**` | Integer | 8 | 1 | −128 ↔ +127
`**short**` | Integer | 16 | 2 | −32 768 ↔ +32 767
`**int**` | Integer | 32 | 4 | −2 147 483 648 ↔ +2 147 483 647
`**long**` | Integer | 64 | 8 | −9 223 372 036 854 775 808 ↔ 9 223 372 036 854 775 807
`**float**` | Floating | 32 | 4 | ±1.401 298 × 10−45 ↔ ±3.402 923 × 1038
`**double**` | Floating | 64 | 8 | ±4.940 656 458 412 465 44 × 10−324 ↔ ±1.797 693 134 862 315 7 × 10308

There are actually a number of components in the IEEE standard, and different computer or chip manufacturers may adhere to only some of them. Furthermore, Python as it develops may not follow all standards, but probably will in time. Normally, a floating-point number _x_ is stored as

(2.4)

that is, with separate entities for the sign _s,_ the fractional part of the mantissa _f,_ and the exponential field _e._ All parts are stored in the binary form and occupy adjacent segments of a single 32-bit word for singles or two adjacent 32-bit words for doubles. The sign _s_ is stored as a single bit, with _s_ = 0 or 1 for a positive or a negative sign. Eight bits are used to stored the exponent _e,_ which means that _e_ can be in the range 0 ≤ _e_ ≤ 255. The endpoints, _e_ = 0 and _e_ = 255, are special cases (Table 2.2). _Normal numbers_ have 0 < _e_ < 255, and with them the convention is to assume that the mantissa's first bit is a 1, so only the fractional part _f_ after the _binary point_ is stored. The representations for _subnormal numbers_ and for the special cases are given in Table 2.2.

Note that the values `**±INF**` and `**NaN**` are not numbers in the mathematical sense, that is, objects that can be manipulated or used in calculations to take limits and such. Rather, they are signals to the computer and to you that something has gone awry and that the calculation should probably stop until you straighten things out. In contrast, the value −0 can be used in a calculation with no harm. Some languages may set unassigned variables to −0 as a hint that they have yet to be assigned, although it is best not to count on that!

Because the uncertainty (error) is only in the mantissa and not in the exponent, the IEEE representations ensure that all normal floating-point numbers have the same relative precision. Because the first bit of a floating point number is assumed to be 1, it does not have to be stored, and computer designers need only recall that there is a _phantom bit_ 1 there to obtain an extra bit of precision. During the processing of numbers in a calculation, the first bit of an intermediate result may become zero, but this is changed before the final number is stored. To repeat, for normal cases, the actual mantissa (1. _f_ in binary notation) contains an implied 1 preceding the binary point.

**Table 2.2** Representation scheme for normal and abnormal IEEE singles.

Number name | Values of _s, e,_ and _f_ | Value of single
---|---|---
Normal | 0 < _e_ < 255 | (−l) _s_ × 2 _e_ −127 × 1. _f_
Subnormal | _e_ = 0, _f_ ≠ 0 | (−l) _s_ × 2−126 × 0. _f_
Signed zero (±0) | _e_ = 0, _f_ = 0 | (−l) _s_ × 0.0
+∞ | _s_ = 0, _e_ = 255, _f_ = 0 | `**+INF**`
−∞ | _s_ = l, _e_ = 255, _f_ = 0 | `**−INF**`
Not a number | _s_ = _u, e_ = 255, _f_ ≠ 0 | `**NaN**`

Finally, in order to guarantee that the stored biased exponent _e_ is always positive, a fixed number called the _bias_ is added to the actual exponent _p_ before it is stored as the biased exponent _e_. The actual exponent, which may be negative, is

(2.5)

#### 2.4.1.1 Examples of IEEE Representations

There are two basic IEEE floating-point formats: singles and doubles. _Singles or floats_ is shorthand for _single-precision floating-point numbers,_ and _doubles_ is shorthand for _double-precision floating-point numbers._ (In Python, however, all floats are double precision.) Singles occupy 32 bits overall, with 1 bit for the sign, 8 bits for the exponent, and 23 bits for the fractional mantissa (which gives 24-bit precision when the phantom bit is included). Doubles occupy 64 bits overall, with 1 bit for the sign, 10 bits for the exponent, and 53 bits for the fractional mantissa (for 54-bit precision). This means that the exponents and mantissas for doubles are not simply double those of floats, as we see in Table 2.1. (In addition, the IEEE standard also permits _extended precision_ that goes beyond doubles, but this is all complicated enough without going into that right now.)

To see the scheme in practice, consider the 32-bit representation (2.4):

The sign bit _s_ is in bit position 31, the biased exponent e is in bits 30−23, and the fractional part of the mantissa _f_ is in bits 22−0. Because 8 bits are used to store the exponent e and because 28 = 256, _e_ has the range

(2.6)

The values _e_ = 0 and 255 are special cases. With bias = 12710, the full exponent

(2.7)

and, as indicated in Table 2.1, singles have the range

(2.8)

The mantissa _f_ for singles is stored as the 23 bits in positions 22−0. For _normal numbers,_ that is, numbers with 0 < _e_ < 255, _f_ is the fractional part of the mantissa, and therefore the actual number represented by the 32 bits is

(2.9)

_Subnormal numbers_ have _e_ = 0, _f_ ≠ 0. For these, _f_ is the entire mantissa, so the actual number represented by these 32 bits is

(2.10)

The 23 bits _m_ 22 − _m_ 0, which are used to store the mantissa of normal singles, correspond to the representation

(2.11)

with 0. _f_ used for subnormal numbers. The special _e_ = 0 representations used to store ±0 and ±∞ are given in Table 2.2.

To see how this works in practice (Figure 2.3), the largest positive normal floating-point number possible for a 32-bit machine has the maximum value _e_ = 254 (the value 255 being reserved) and the maximum value for _f_ :

(2.12)

where we have grouped the bits for clarity. After putting all the pieces together, we obtain the value shown in Table 2.1:

(2.13)

Likewise, the smallest positive floating-point number possible is subnormal ( _e_ = 0) with a single significant bit in the mantissa:

(2.14)

This corresponds to

(2.15)

In summary, single-precision (32-bit or 4-byte) numbers have six or seven decimal places of significance and magnitudes in the range

(2.16)

_Doubles_ are stored as two 32-bit words, for a total of 64 bits (8 B). The sign occupies 1 bit, the exponent _e,_ 11 bits, and the fractional mantissa, 52 bits:

As we see here, the fields are stored contiguously, with part of the mantissa _f_ stored in separate 32-bit words. The order of these words, and whether the second word with _f_ is the most or least significant part of the mantissa, is machine-dependent. For doubles, the bias is quite a bit larger than for singles,

(2.17)

**Table 2.3** Representation scheme for IEEE doubles.

Number name | Values of _s, e,_ and _f_ | Value of double
---|---|---
Normal | 0 < _e_ < 2047 | (−l) _s_ × 2 _e_ −1023 × 1. _f_
Subnormal | _e_ = 0, _f_ ≠ 0 | (−l) _s_ × 2−1022 × 0. _f_
Signed zero | _e_ = 0, _f_ = 0 | (−1) _s_ × 0.0
+∞ | _s_ = 0, _e_ = 2047, _f_ = 0 | `**+INF**`
−∞ | _s_ = l, _e_ = 2047, _f_ = 0 | `**−INF**`
Not a number | _s_ = _u, e_ = 2047, _f_ ≠ 0 | `**NaN**`

so the actual exponent _p_ = _e_ − 1023.

The bit patterns for doubles are given in Table 2.3, with the range and precision given in Table 2.1. To repeat, if you write a program with doubles, then 64 bits (8 bytes) will be used to store your floating-point numbers. Doubles have approximately 16 decimal places of precision (1 part in 252) and magnitudes in the range

(2.18)

If a single-precision number _x_ is larger than 2128, a fault condition known as an _overflow_ occurs (Figure 2.3). If _x_ is smaller than 2−128, an underflow occurs. For overflows, the resulting number _x_ c may end up being a machine-dependent pattern, not a number (NAN), or unpredictable. For underflows, the resulting number _x_ c is usually set to zero, although this can usually be changed via a compiler option. (Having the computer automatically convert underflows to zero is usually a good path to follow; converting overflows to zero may be the path to disaster.) Because the only difference between the representations of positive and negative numbers on the computer is the sign bit of 1 for negative numbers, the same considerations hold for negative numbers.

In our experience, _serious scientific calculations almost always require at least 64-bit (double-precision) floats._ And if you need double precision in one part of your calculation, you probably need it all over, which means double-precision library routines for methods and functions.

### 2.4.2 Python and the IEEE 754 Standard

Python is a relatively recent language with changes and extensions occurring as its use spreads and as its features mature. It should be no surprise then that Python does not at present adhere to all aspects, and especially the special cases, of the IEEE 754 standard. Probably the most relevant difference for us is that _Python does not support single (32 bits) precision floating-point numbers._ So when we deal with a data type called a _float_ in Python, it is the equivalent of a _double_ in the IEEE standard. Because singles are inadequate for most scientific computing, this is not a loss. However be wary, if you switch over to Java or C you should declare your variables as `**doubles**` and not as `**floats**`. While Python eliminates single-precision floats, it adds a new data type _complex_ for dealing with complex numbers. Complex numbers are stored as pairs of doubles and are quite useful in science.

The details of how closely Python adheres to the IEEE 754 standard depend upon the details of Python's use of the C or Java language to power the Python interpreter. In particular, with the recent 64 bits architectures for CPUs, the range may even be greater than the IEEE standard, and the abnormal numbers (` **±INF, NaN**`) may differ. Likewise, the exact conditions for overflows and underflows may also differ. That being the case, the exploratory exercises to follow become all that more interesting because we cannot say that we know what results you should obtain!

### 2.4.3 Over and Underflow Exercises

  1. Consider the 32-bit single-precision floating-point number _A_ :
    1. a) What are the (binary) values for the sign _s,_ the exponent _e,_ and the fractional mantissa _f_? ( _Hint: e_ 10 = 14.)
    2. b) Determine decimal values for the biased exponent _e_ and the true exponent _p._
    3. c) Show that the mantissa of _A_ equals 1.625 000.
    4. d) Determine the full value of _A_.
  2. Write a program that determines the _underflow_ and _overflow_ limits (within a factor of 2) for Python on your computer. Here is a sample pseudocode
You may need to increase _N_ if your initial choice does not lead to underflow and overflow. (Notice that if you want to be more precise regarding the limits of your computer, you may want to multiply and divide by a number smaller than 2.)
    1. a) Check where under- and overflow occur for double-precision floating-point numbers (floats). Give your answer in decimal.
    2. b) Check where under- and overflow occur for double-precision floating-point numbers.
    3. c) Check where under- and overflow occur for integers. _Note:_ There is no exponent stored for integers, so the smallest integer corresponds to the most negative one. To determine the largest and smallest integers, you must observe your program's output as you explicitly pass through the limits. You accomplish this by continually adding and subtracting 1. (Because integer arithmetic uses _two's complement_ arithmetic, you should expect some surprises.)

### 2.4.4 Machine Precision (Model)

A major concern of computational scientists is that the floating-point representation used to store numbers is of limited precision. As we have shown for a 32-bit-word machine, _single-precision numbers are good to 6–7 decimal places, while doubles are good to 15–16 places._ To see how limited precision affects calculations, consider the simple computer addition of two single-precision numbers:

(2.19)

The computer fetches these numbers from memory and stores the bit patterns

(2.20)

(2.21)

in _working registers_ (pieces of fast-responding memory). Because the exponents are different, it would be incorrect to add the mantissas, and so the exponent of the smaller number is made larger while progressively decreasing the mantissa by _shifting bits_ to the right (inserting zeros) until both numbers have the same exponent:

(2.22)

(2.23)

Because there is no room left to store the last digits, they are lost, and after all this hard work the addition just gives 7 as the answer (truncation error in Figure 2.3). In other words, because a 32-bit computer stores only 6 or 7 decimal places, it effectively ignores any changes beyond the sixth decimal place.

The preceding loss of precision is categorized by defining the _machine precision _ m as the maximum positive number that, on the computer, can be added to the number stored as 1 without changing that stored 1:

(2.24)

where the subscript c is a reminder that this is a computer representation of 1. Consequently, an arbitrary number _x_ can be thought of as related to its floating-point representation _x_ c by

(2.25)

where the actual value for _ _is not known. In other words, except for powers of 2 that are represented exactly, we should assume that all single-precision numbers contain an error in the sixth decimal place and that all doubles have an error in the 15th place. And as is always the case with errors, we must assume that we really do not know what the error is, for if we knew, then we would eliminate it! Consequently, the arguments we are about to put forth regarding errors should be considered approximate, but that is to be expected for unknown errors.

### 2.4.5 Experiment: Your Machine's Precision

Write a program to determine the machine precision _ _m of your computer system within a factor of 2. A sample pseudocode is

A Python implementation is given in Listing 2.4, while a more precise one is Byte-Limit.py on the instructor's guide.

**Listing 2.4** `**Limits.py**` determines machine precision within a factorof 2.

  1. Determine experimentally the precision of double-precision floats.
  2. Determine experimentally the precision of complex numbers.

To print out a number in the decimal format, the computer must convert from its internal binary representation. This not only takes time, but unless the number is an exact power of 2, leads to a loss of precision. So if you want a truly precise indication of the stored numbers, you should avoid conversion to decimals and instead print them out in octal or hexadecimal format (` **\0NNN**`).

## 2.5 Problem: Summing Series

A classic numerical problem is the summation of a series to evaluate a function. As an example, consider the infinite series for sin _x:_

(2.26)

Your _problem_ is to use just this series to calculate sin _x_ for _x_ < 2π and _x_ > 2π, with an absolute error in each case of less than 1 part in 108. While an infinite series is exact in a mathematical sense, it is not a good algorithm because errors tend to accumulate and because we must stop summing at some point. An algorithm would be the finite sum

(2.27)

But how do we decide when to stop summing? (Do not even think of saying, "When the answer agrees with a table or with the built-in library function.") One approach would be to stop summing when the next term is smaller than the precision desired. Clearly then, if _x_ is large, this would require large _N_ as well. In fact, for really large _x,_ one would have to go far out in the series before the terms start decreasing.

### 2.5.1 Numerical Summation (Method)

Never mind that the algorithm (2.27) indicates that we should calculate   and then divide it by   This is not a good way to compute. On the one hand, both   and   can get very large and cause overflows, despite the fact that their quotient may not be large. On the other hand, powers and factorials are very expensive (time-consuming) to evaluate on the computer. Consequently, a better approach is to use a single multiplication to relate the next term in the series to the previous one:

(2.28)

While we want to establish absolute accuracy for sin _x,_ that is not so easy to do. What is easy to do is to assume that the error in the summation is approximately the last term summed (this assumes no round-off error, a subject we talk about in Chapter 3). To obtain a relative error of 1 part in 108, we then stop the calculation when

(2.29)

where "term" is the last term kept in the series (2.27) and "sum" is the accumulated sum of all the terms. In general, you are free to pick any tolerance level you desire, although if it is too close to, or smaller than, machine precision, your calculation may not be able to attain it. A pseudocode for performing the summation is

### 2.5.2 Implementation and Assessment

  1. Write a program that implements this pseudocode for the indicated _x_ values. Present the results as a table with headings `**x imax sum |sum- sin(x)|/sin(x)**`, where `**sin(x)**` is the value obtained from the built-in function. The last column here is the relative error in your computation. Modify the code that sums the series in a "good way" (no factorials) to one that calculates the sum in a "bad way" (explicit factorials).
  2. Produce a table as above.
  3. Start with a tolerance of 10−8 as in (2.29).
  4. Show that for sufficiently small values of _x,_ your algorithm converges (the changes are smaller than your tolerance level) and that it converges to the correct answer.
  5. Compare the number of decimal places of precision obtained with that expected from (2.29).
  6. Without using the identity  , show that there is a range of somewhat large values of _x_ for which the algorithm converges, but that it converges to the wrong answer.
  7. Show that as you keep increasing _x,_ you will reach a regime where the algorithm does not even converge.
  8. Now make use of the identity   to compute sin _x_ for large _x_ values where the series otherwise would diverge.
  9. Repeat the calculation using the "bad" version of the algorithm (the one that calculates factorials) and compare the answers.
  10. Set your tolerance level to a number smaller than machine precision and see how this affects your conclusions.

1) The Beginner's All-Purpose Symbolic Instruction Code (BASIC) programming language of the original PCs should not be confused with basic machine language. 2) Comments placed in the field to the right are for your information and _not_ for the computer to act upon. 3) The _two's complement_ of a binary number is the value obtained by subtracting the number from 2 _N_ for an _N_ -bit representation. Because this system represents negative numbers by the two's complement of the absolute value of the number, additions and subtractions can be made without the need to work with the sign of the number.

# 3
Errors and Uncertainties in Computations

> _To err is human, to forgive divine._
>
> _Alexander Pope_

Whether you are careful or not, errors and uncertainties are part of computation. Some errors are the ones that humans inevitably make, but some are introduced by the computer. Computer errors arise because of the limited precision with which computers store numbers or because algorithms or models can fail. Although it stifles creativity to keep thinking "error" when approaching a computation, it certainly is a waste of time, and possibly harmful, to work with results that are meaningless "garbage" because of errors. In this chapter, we examine some of the errors and uncertainties that may occur in computations. Although we do not keep repeating a mantra about watching for errors, the lessons of this chapter apply to all other chapters as well.

## 3.1 Types of Errors (Theory)

Let us say that you have a program of high complexity. To gauge why errors should be of concern, imagine that the program that has the logical flow

(3.1)

where each unit   might be a statement or a step. If each unit has probability _p_ of being correct, then the joint probability _P_ of the whole program being correct is _P_ = _p n_. Let us say, we have a medium-sized program with _n_ = 1000 steps and that the probability of each step being correct is almost 1,  . This means that you end up with  , that is, a final answer that is as likely wrong as right (not a good way to build a bridge). The problem is that, as a scientist, you want a result that is correct – or at least in which the uncertainty is small and of known size, even if the code executes millions of steps.

Four general types of errors exist to plague your computations:

  1. _Blunders or bad theory:_ Typographical errors entered with your program or data, running the wrong program or having a fault in your reasoning (theory), using the wrong data file, and so on. (If your blunder count starts increasing, it may be time to go home or take a break.)
  2. _Random errors:_ Imprecision caused by events such as fluctuations in electronics, cosmic rays, or someone pulling a plug. These may be rare, but you have no control over them and their likelihood increases with running time; while you may have confidence in a 20 s calculation, a week-long calculation may have to be run multiple times to check reproducibility.
  3. _Approximation errors:_ Imprecision arising from simplifying the mathematics so that a problem can be solved on the computer. They include the replacement of infinite series by finite sums, infinitesimal intervals by finite ones, and variable functions by constants. For example,

(3.2)

where _ _( _x_ , _N_ ) is the approximation error and __ __is the series from _N_ \+ 1 to ∞. Because the approximation error arises from the algorithm we use to approximate the mathematics, it is also called the _algorithmic error_. For every reasonable approximation, the approximation error should decrease as _N_ increases and should vanish in the limit _N_ → ∞. Specifically for (3.2), because the scale for _N_ is set by the value of _x_ , a small approximation error requires _N_   _x_. So if _x_ and _N_ are close in value, the approximation error will be large.

  4. _Round-off errors:_ Imprecision arising from the finite number of digits used to store floating-point numbers. These "errors" are analogous to the uncertainty in the measurement of a physical quantity encountered in an elementary physics laboratory. The overall round-off error accumulates as the computer handles more numbers, that is, as the number of steps in a computation increases. This may cause some algorithms to become _unstable_ with a rapid increase in error. In some cases, round-off error may become the major component in your answer, leading to what computer experts call _garbage_.
For example, if your computer kept four decimal places, then it will store 1/3 as 0.3333 and 2/3 as 0.6667, where the computer has "rounded off" the last digit in 2/3. Accordingly, if we ask the computer to do as simple a calculation as 2(1/3) – 2/3, it produces

(3.3)

So although the result is small, it is not 0, and if we repeat this type of calculation millions of times, the final answer might not even be small (small garbage begets large garbage).

When considering the precision of calculations, it is good to recall our discussion in Chapter 2 of significant figures and scientific notation given in your early physics or engineering classes. For computational purposes, let us consider how the computer may store the floating-point number

(3.4)

Because the exponent is stored separately and is a small number, we may assume that it will be stored in full precision. In contrast, some of the digits of the mantissa may be truncated. In double precision, the mantissa of _a_ will be stored in two words, the _most significant part_ representing the decimal 1.122 33, and the _least significant part_ 44 556 677. The digits beyond 7 are lost. As we shall see soon, when we perform calculations with words of fixed length, it is inevitable that errors will be introduced (at least) into the least significant parts of the words.

### 3.1.1 Model for Disaster: Subtractive Cancelation

Calculations employing numbers that are stored only approximately on the computer can only be expected to yield approximate answers. To demonstrate the effect of this type of uncertainty, we model the computer representation _x c_ of the exact number _x_ as

(3.5)

Here   is the relative error in _x c_, which we expect to be of a similar magnitude to the machine precision  . If we apply this notation to the simple subtraction  , we obtain

(3.6)

We see from (3.6) that the resulting error in _a_ is essentially a weighted average of the errors in _b_ and _c_ , with no assurance that the last two terms will cancel. Of special importance here is the observation that the error in the answer _a c_ increases when we subtract two nearly equal numbers   because then we are subtracting off the most significant parts of both numbers and leaving the error-prone least-significant parts:

(3.7)

This shows that even if the relative errors in _b_ and _c_ cancel somewhat, they are multiplied by the large number _b_ / _a_ , which can significantly magnify the error. Because we cannot assume any sign for the errors, we must assume the worst (the "max" in (3.7)).

**Theorem** If you subtract two large numbers and end up with a small one, then the small one is less significant than the large numbers.

We have already seen an example of subtractive cancelation in the power series summation for sin   for large _x_. A similar effect occurs for   for large _x_ , where the first few terms are large but of alternating sign, leading to an almost total cancelation in order to yield the final small result. (Subtractive cancelation can be eliminated by using the identity  , although the round-off error will still remain.)

### 3.1.2 Subtractive Cancelation Exercises

  1. Remember back in high school when you learned that the quadratic equation

(3.8)

has an analytic solution that can be written as either

(3.9)

Inspection of (3.9) indicates that subtractive cancelation (and consequently an increase in error) arises when _b_ 2   4 _ac_ because then the square root and its preceding term nearly cancel for one of the roots.

    1. a) Write a program that calculates all four solutions for arbitrary values of _a_ , _b_ , and _c_.
    2. b) Investigate how errors in your computed answers become large as the subtractive cancelation increases and relate this to the known machine precision. ( _Hint:_ A good test case utilizes  )
    3. c) Extend your program so that it indicates the most precise solutions.
  2. As we have seen, subtractive cancelation occurs when summing a series with alternating signs. As another example, consider the finite sum

(3.10)

If you sum the even and odd values of _n_ separately, you get two sums:

(3.11)

All terms are positive in this form with just a single subtraction at the end of the calculation. Yet even this one subtraction and its resulting cancelation can be avoided by combining the series analytically to obtain

(3.12)

Although all three summations _S_ (1), _S_ (2), and _S_ (3) are mathematically equal, they may give different numerical results.

    1. a) Write a double-precision program that calculates _S_ (1), _S_ (2), and _S_ (3).
    2. b) Assume _S_ (3) to be the exact answer. Make a log–log plot of the relative error vs. the number of terms, that is, of   Start with _N_ = 1 and work up to _N_ = 1 000 000. (Recollect that  .) The negative of the ordinate in this plot gives an approximate value for the number of significant figures.
    3. c) See whether straight-line behavior for the error occurs in some region of your plot. This indicates that the error is proportional to a power of _N_.
  3. In spite of the power of your trusty computer, calculating the sum of even a simple series may require some thought and care. Consider the two series

(3.13)

Both series are finite as long as _N_ is finite, and when summed analytically both give the same answer. Nonetheless, because of round-off error, the numerical value of _S_ (up) will not be precisely that of _S_ (down).

    1. a) Write a program to calculate _S_ (up) and _S_ (down) as functions of _N_.
    2. b) Make a log–log plot of  .
    3. c) Observe the linear regime on your graph and explain why the downward sum is generally more precise.

### 3.1.3 Round-off Errors

Let us start by seeing how error arises from a single division of the computer representations of two numbers:

(3.14)

Here we ignore the very small   terms and add errors in absolute value because we cannot assume that we are fortunate enough to have unknown errors canceling each other. Because we add the errors in absolute value, this same rule holds for multiplication. Equation 3.14 is just the basic rule of error propagation from elementary laboratory work: You add the uncertainties in each quantity involved in an analysis to arrive at the overall uncertainty.

We can even generalize this model to estimate the error in the evaluation of a general function _f_ ( _x_ ), that is, the difference in the value of the function evaluated at _x_ and at _x_ c:

(3.15)

So, for example,

(3.16)

(3.17)

If we evaluate this expression for   and assume an error in the fourth place of _x_ , we obtain a similar relative error of  .

### 3.1.4 Round-off Error Accumulation

There is a useful model for approximating how round-off error accumulates in a calculation involving a large number of steps. As illustrated in Figure 3.1, we view the error in each step of a calculation as a literal "step" in a _random walk_ , that is, a walk for which each step is in a random direction. As we derive and simulate in Chapter 4, the total distance _R_ covered in _N_ steps of length _r_ , is, on average,

(3.18)

By analogy, the total relative error   arising after _N_ calculational steps each with the machine precision error   is, on average,

(3.19)

If the round-off errors in a particular algorithm do not accumulate in a random manner, then a detailed analysis is needed to predict the dependence of the error on the number of steps _N_. In some cases there may be no cancelation, and the error may increase as  . Even worse, in some recursive algorithms, where the error generation is coherent, such as the upward recursion for spherical Bessel functions, there may be an _N_! increase in error.

## 3.2 Error in Bessel Functions (Problem)

Accumulating round-off errors often limits the ability of a program to calculate accurately. Your _problem_ is to compute the spherical Bessel and Neumann functions _j l_( _x_ ) and _n l_( _x_ ). These function are, respectively, the regular/irregular (non-singular/singular at the origin) solutions of the differential equation

(3.20)

**Figure 3.1** A schematic of the _N_ steps in a random walk simulation that end up a distance   from the origin. Notice how the Δ _x_ 's for each step add vectorially.

**Table 3.1** Approximate values for spherical Bessel functions (from maple).

**_x_** | **_j_ 3( _x_ )** | **_j_ 5( _x_ )** | **_j_ 8( _x_ )**
---|---|---|---
0.1 | +9.518 519 719 × 10−6 | +9.616 310 231 × 10−10 | +2.901 200 102 × 10−16
1 | +9.006 581 118 × 10−3 | +9.256 115 862 × 10−05 | +2.826 498 802 × 10−08
10 | − 3.949 584 498 × 10−2 | −5.553 451 162 × 10−02 | +1.255 780 236 × 10−01

The spherical Bessel functions are related to the Bessel function of the first kind by  . They occur in many physical problems, such as the expansion of a plane wave into spherical partial waves,

(3.21)

Figure 3.2 shows what the first few _j l_ look like, and Table 3.1 gives some explicit values. For the first two _l_ values, explicit forms are

(3.22)

(3.23)

### 3.2.1 Numerical Recursion (Method)

The classic way to calculate _j l_( _x_ ) would be by summing its power series for small values of _x_ / _l_ and summing its asymptotic expansion for large _x_ / _l_ values. The approach we adopt is based on the _recursion relations_

(3.24)

**Figure 3.2** The first four spherical Bessel functions _j l_( _x_ ) as functions of _x_. Notice that for small _x_ , the values for increasing _l_ become progressively smaller.

(3.25)

Equations (3.24) and (3.25) are the same relation, one written for upward recurrence from small to large _l_ values, and the other for downward recurrence from large _l_ to small _l_. With just a few additions and multiplications, recurrence relations permit rapid, simple computation of the entire set of _j l_ values for fixed _x_ and all _l_.

To recur upward in _l_ for fixed _x_ , we start with the known forms for _j_ 0 and _j_ 1 (3.22) and use (3.24). As you will prove for yourself, this upward recurrence usually seems to work at first but then fails. The reason for the failure can be seen from the plots of _j l_( _x_ ) and _n l_( _x_ ) vs. _x_ (Figure 3.2). If we start at   and _l_ = 0, we will see that as we recur _j l_ up to larger _l_ values with (3.24), we are essentially taking the difference of two "large" functions to produce a "small" value for _j l_. This process suffers from subtractive cancelation and always reduces the precision. As we continue recurring, we take the difference of two small functions with large errors and produce a smaller function with yet a larger error. After a while, we are left with only the round-off error (garbage).

To be more specific, let us call   the numerical value we compute as an approximation for _j l_( _x_ ). Even if we start with pure _j l_, after a short while the computer's lack of precision effectively mixes in a bit of _n l_( _x_ ):

(3.26)

This is inevitable because both _j l_ and _n l_ satisfy the same differential equation and, on that account, the same recurrence relation. The admixture of _n l_ becomes a problem when the numerical value of _n l_( _x_ ) is much larger than that of _j l_( _x_ ) because even a minuscule amount of a very large number may be large.

The simple solution to this problem ( _Miller's device_ ) is to use (3.25) for downward recursion of the _j l_ values starting at a large value _l_ = _L_. This avoids subtractive cancelation by taking small values of _j_ _l_ +1( _x_ ) and _j l_( _x_ ) and producing a larger _j_ _l_ −1( _x_ ) by addition. While the error may still behave like a Neumann function, the actual magnitude of the error will _decrease_ quickly as we move downward to smaller _l_ values. In fact, if we start iterating downward with arbitrary values for   and  , after a short while we will arrive at the correct _l_ dependence for this value of _x_. Although the precise value of   so obtained will not be correct because it depends upon the arbitrary values assumed for   and  , the relative values will be accurate. The absolute values are fixed from the known value (3.22),  . Because the recurrence relation is a linear relation between the _j l_ values, we only need normalize all the computed values via

(3.27)

Accordingly, after you have finished the downward recurrence, you obtain the final answer by normalizing all  values based on the known value for _j 0_.

### 3.2.2 Implementation and Assessment: Recursion Relations

A program implementing recurrence relations is most easily written using subscripts. If you need to polish up on your skills with subscripts, you may want to study our program `**Bessel.py**` in Listing 3.1 before writing your own.

  1. Write a program that uses both upward and downward recursion to calculate _j l_( _x_ ) for the first 25 _l_ values for _x_ = 0.1, 1, and 10.
  2. Tune your program so that at least one method gives "good" values (meaning a relative error  ). See Table 3.1 for some sample values.
  3. Show the convergence and stability of your results.
  4. Compare the upward and downward recursion methods, printing out  ,  , and the relative difference  .
  5. The errors in computation depend on _x_ , and for certain values of _x_ , both up and down recursions give similar answers. Explain the reason for this.

**Listing 3.1** **Bessel.py** determines spherical Bessel functions by downward recursion (you should modify this to also work by upward recursion).

## 3.3 Experimental Error Investigation

Numerical algorithms play a vital role in computational physics. Your _problem_ is to take a general algorithm and decide

  1. Does it converge?
  2. How precise are the converged results?
  3. How expensive (time-consuming) is it?

On first thought you may think, "What a dumb problem! All algorithms converge if enough terms are used, and if you want more precision, then use more terms." Well, some algorithms may be asymptotic expansions that just approximate a function in certain regions of parameter space and converge only up to a point. Yet even if a uniformly convergent power series is used as the algorithm, including more terms will decrease the algorithmic error but will also increase the round-off errors. And because round-off errors eventually diverge to infinity, the best we can hope for is a "best" approximation. Good algorithms are good not only because they are fast but also because they require fewer steps and thus incur less round-off error.

Let us assume that an algorithm takes _N_ steps to find a good answer. As a rule of thumb, the approximation (algorithmic) error decreases rapidly, often as the inverse power of the number of terms used:

(3.28)

Here _α_ and _β_ are empirical constants that change for different algorithms and may be only approximately constant, and even then only as _N_ → ∞. The fact that the error must fall off for large _N_ is just a statement that the algorithm converges.

In contrast to algorithmic error, round-off error grows slowly and somewhat randomly with _N_. If the round-off errors in each step of the algorithm are not correlated, then we know from the previous discussion that we can model the accumulation of error as a random walk with step size equal to the machine precision  :

(3.29)

This is the slow growth with _N_ that we expect from the round-off error. The total error in a computation is the sum of the two types of errors:

(3.30)

(3.31)

For small _N_ , we expect the first term to be the larger of the two, but as _N_ grows it will be overcome by the growing round-off error.

As an example, in Figure 3.3 we present a log–log plot of the relative error in numerical integration using the Simpson integration rule (Chapter 5). We use the log10 of the relative error because its negative tells us the number of decimal places of precision obtained.1) Let us assume   is the exact answer and _A_ ( _N_ ) the computed answer. If

(3.32)

We see in Figure 3.3 that the error does show a rapid decrease for small _N_ , consistent with an inverse power law (3.28). In this region, the algorithm is converging. As _N_ keeps increasing, the error starts to look somewhat erratic, with a slow increase on average. In accordance with (3.30), in this region the round-off error has grown larger than the approximation error and will continue to grow for increasing _N_. Clearly then, the smallest total error will be obtained if we can stop the calculation at the minimum near 10−14, that is, when  .

**Figure 3.3** A log–log plot of relative error vs. the number of points used for a numerical integration. The ordinate value of ∼ 10−14 at the minimum indicates that ∼ 14 decimal places of precision are obtained before round-off error begins to build up. Notice that while the round-off error does fluctuate indicating a statistical aspect of error accumulation, on average it is increasing but more slowly than did the algorithm's error decrease.

In realistic calculations, you would not know the exact answer; after all, if you did, then why would you bother with the computation? However, you may know the exact answer for a similar calculation, and you can use that similar calculation to perfect your numerical technique. Alternatively, now that you understand how the total error in a computation behaves, you should be able to look at a table or, better yet, a graph like Figure 3.3, of your answer and deduce the manner in which your algorithm is converging. Specifically, at some point you should see that the mantissa of the answer changes only in the less significant digits, with that place moving further to the right of the decimal point as the calculation executes more steps. Eventually, however, as the number of steps becomes even larger, roundoff error leads to a fluctuation in the less significant digits, with a gradual move towards the left on average. It is best to quit the calculation before this occurs.

Based upon this understanding, an approach to obtaining the best approximation is to deduce when your answer behaves like (3.30). To do that, we call   the exact answer and _A_ ( _N_ ) the computed answer after _N_ steps. We assume that for large enough values of _N_ , the approximation converges as

(3.33)

that is, that the round-off error term in (3.30) is still small. We then run our computer program with 2 _N_ steps, which should give a better answer, and use that answer to eliminate the unknown  :

(3.34)

To see if these assumptions are correct and determine what level of precision is possible for the best choice of _N_ , plot log10  vs. log10 _N_ , similar to what we have performed in Figure 3.3. If you obtain a rapid straight-line drop off, then you know you are in the region of convergence and can deduce a value for _β_ from the slope. As _N_ gets larger, you should see the graph change from a straight-line decrease to a slow increase as the round-off error begins to dominate. A good place to quit is before this. In any case, now you understand the error in your computation and therefore have a chance to control it.

As an example of how different kinds of errors enter into a computation, we assume that we know the analytic form for the approximation and round-off errors:

(3.35)

(3.36)

(3.37)

(3.38)

The total error is then a minimum when

(3.39)

(3.40)

For a double-precision calculation  , the minimum total error occurs when

(3.41)

In this case, most of the error is as a result of round-off and is not the approximation error.

Seeing that the total error is mainly the round-off error  , an obvious way to decrease the error is to use a smaller number of steps _N_. Let us assume, we do this by finding another algorithm that converges more rapidly with _N_ , for example, one with approximation error behaving like

(3.42)

The total error is now

(3.43)

The number of points for minimum error is found as before:

(3.44)

The error is now smaller by a factor of 4, with only 1/16 as many steps needed. Subtle are the ways of the computer. In this case, the better algorithm is quicker and, by using fewer steps, produces less round-off error.

**Exercise** Estimate the error now for a double-precision calculation.

### 3.3.1 Error Assessment

In Section 2.5, we have already discussed the Taylor expansion of sin _x_ :

(3.45)

We now extend that discussion with errors in mind. The series (3.45) converges in the mathematical sense for all values of _x_. Accordingly, a reasonable algorithm to compute the sin( _x_ ) might be

(3.46)

  1. Write a program that calculates sin( _x_ ) as the finite sum (3.46). (If you already did this in Chapter 2, then you may reuse that program and its results here. But remember, you should not be using factorials in the algorithm.)
  2. Calculate your series for   and compare it to the built-in function `**Math.sin(x)**` (you may assume that the built-in function is exact). Stop your summation at an _N_ value for which the next term in the series will be no more than 10−7 of the sum up to that point,

(3.47)

  3. Examine the terms in the series for   and observe the significant subtractive cancelations that occur when large terms add together to give small answers. (Do not use the identity   to reduce the value of _x_ in the series.) In particular, print out the near-perfect cancelation around  .
  4. See if better precision is obtained by using trigonometric identities to keep  .
  5. By progressively increasing _x_ from 1 to 10, and then from 10 to 100, use your program to determine experimentally when the series starts to lose accuracy and when it no longer converges.
  6. Make a series of graphs of the error vs. _N_ for different values of _x_. You should get curves similar to those in Figure 3.4.

**Figure 3.4** The error in the summation of the series for e− _x_ vs. _N_ for various _x_ values. The values of _x_ increase vertically for each curve. Note that a negative initial slope corresponds to decreasing error with _N_ , and that the dip corresponds to a rapid convergence followed by a rapid increase in error. (Courtesy of J. Wiren.)

Because this series summation is such a simple, correlated process, the round-off error does not accumulate randomly as it might for a more complicated computation, and we do not obtain the error behavior (3.33). We will see the predicted error behavior when we examine integration rules in Chapter 5.

1) Most computer languages use  . Yet because  , we have  .

# 4
Monte Carlo: Randomness, Walks, and Decays

* * *

This chapter starts with a discussion of how computers generate numbers that appear random, but really are not, and how we can test that. After discussing how computers generate pseudorandom numbers, we explore how these numbers are used to incorporate the element of chance into simulations. We do this first by simulating a random walk and then by simulating the spontaneous decay of an atom or nucleus. In Chapter 5, we show how to use these random numbers to evaluate integrals,and in Chapter 17, we investigate the use of random numbers to simulate thermal processes and the fluctuations of quantum systems.

* * *

## 4.1 Deterministic Randomness

Some people are attracted to computing because of its deterministic nature; it is nice to have a place in one's life where nothing is left to chance. Barring machine errors or undefined variables, you get the same output every time you feed your program the same input. Nevertheless, many computer cycles are used for _Monte Carlo_ calculations that at their very core include elements of chance. These are calculations in which random-like numbers generated by the computer are used to _simulate_ naturally random processes, such as thermal motion or radioactive decay, or to solve equations on the average. Indeed, much of computational physics' recognition has come about from the ability of computers to solve previously intractable problems using Monte Carlo techniques.

## 4.2 Random Sequences (Theory)

We define a sequence _r_ 1, _r_ 2,... as _random_ if there are no correlations among the numbers. Yet being random does not mean that all the numbers in the sequence are equally likely to occur. If all the numbers in a sequence are equally likely to occur, then the sequence is called _uniform_ , which does not say anything about being random. To illustrate, 1, 2, 3, 4,... is uniform but probably not random. Furthermore, it is possible to have a sequence of numbers that, in some sense, are random but have very short-range correlations among themselves, for example,

(4.1)

have short-range but not long-range correlations.

Mathematically, the likelihood of a number occurring is described by a distribution function _P_ ( _r_ ), where _P_ ( _r_ ) d _r_ is the probability of finding r in the interval [ _r_ , _r_ \+ d _r_ ]. A _uniform_ distribution means that _P_ ( _r_ ) = a constant. The standard random-number generator on computers generates uniform distributions between 0 and 1. In other words, the standard random-number generator outputs numbers in this interval, each with an equal probability yet each independent of the previous number. As we shall see, numbers can also be more likely to occur in certain regions than other, yet still be random.

By their very nature, computers are deterministic devices and so cannot create a random sequence. Computed random number sequences must contain correlations and in this way cannot be truly random. Although it may be a bit of work, if we know a computed random number _r m_ and its preceding elements, then it is always possible to figure out _r_ _m_ +1. For this reason, computers are said to generate _pseudorandom numbers_ (yet with our incurable laziness we will not bother saying "pseudo" all the time). While more sophisticated generators do a better job at hiding the correlations, experience shows that if you look hard enough or use pseudorandom numbers long enough, you will notice correlations. A primitive alternative to generating random numbers is to read in a table of truly random numbers determined by naturally random processes such as radioactive decay, or to connect the computer to an experimental device that measures random events. These alternatives are not good for production work, but have actually been used as a check in times of doubt.

### 4.2.1 Random-Number Generation (Algorithm)

The _linear congruent_ or _power residue_ method is the common way of generating a pseudorandom sequence of numbers   over the interval [0, _M_ – 1]. To obtain the next random number _r_ _i_ +1, you multiply the present random number _r_ _i_ by the constant _a_ , add another constant _c_ , take the _modulus_ by _M_ , and then keep just the fractional part (remainder)1):

(4.2)

(4.3)

The value for _r_ 1 (the _seed_ ) is frequently supplied by the user, and _mod_ is a built-in function on your computer for _remaindering_. In Python, the percent sign % is the modulus operator. This is essentially a bit-shift operation that ends up with the least significant part of the input number and thus counts on the randomness of round-off errors to generate a random sequence.

For example, if _c_ = 1, _a_ = 4, _M_ = 9, and you supply _r_ 1 = 3, then you obtain the sequence

(4.4)

(4.5)

(4.6)

(4.7)

(4.8)

We get a sequence of length _M_ = 9, after which the entire sequence repeats. If we want numbers in the range [0, 1], we divide the _r_ 's by _M_ = 9 to obtain

(4.9)

This is still a sequence of length 9, but is no longer a sequence of integers. If random numbers in the range [ _A, B_ ] are needed, you only need to _scale_ :

(4.10)

As a rule of thumb: _Before using a random-number generator in your programs, you should check its range and that it produces numbers that "look" random._

Although not a mathematical proof, you should always make a graphical display of your random numbers. Your visual cortex is quite refined at recognizing patterns and will tell you immediately if there is one in your random numbers. For instance, Figure 4.1 shows generated sequences from "good" and "bad" generators, and it is clear which is not random. (Although if you look hard enough at the random points, your mind may well pick out patterns there too.)

The linear congruent method (4.2) produces integers in the range [0, _M_ − 1] and therefore becomes completely correlated if a particular integer comes up a second time (the whole cycle then repeats). In order to obtain a longer sequence, _a_ and _M_ should be large numbers but not so large that _ar_ _i_ −1 overflows. On a computer using 48-bit integer arithmetic, the built-in random-number generator may use _M_ values as large as 248   3 × 1014. A 32-bit generator may use _M_ = 231   2 × 109. If your program uses approximately this many random numbers, you may need to reseed (start the sequence over again with a different initial value) during intermediate steps to avoid the cycle repeating.

**Figure 4.1** (a) A plot of successive random numbers ( _x_ , _y_ ) = ( _r i_, _r_ _i_ +1) generated with a deliberately "bad" generator, (b) A plot generated with the built in random number generator. While the plot (b) is not the proof that the distribution is random, the plot (a) is a proof enough that the distribution is not random.

Your computer probably has random-number generators that are better than the one you will compute with the power residue method. In Python, we use `**random.random()**`, the Mersenne Twister generator. We recommend that you use the best one you can find rather than write your own. To initialize a random sequence, you need to plant a seed in it. In Python, the statement `**random.seed(None)**` seeds the generator with the system time (see `**Walk.py**` in Listing 4.1). Our old standard, `**drand48**`, uses:

(4.11)

(4.12)

### 4.2.2 Implementation: Random Sequences

For scientific work, we recommend using an industrial-strength random-number generator. To see why, here we assess how _bad_ a careless application of the power residue method can be.

  1. Write a simple program to generate random numbers using the linear congruent method (4.2).
  2. For pedagogical purposes, try the unwise choice: ( _a, c, M, r_ 1) = (57, 1, 256, 10). Determine the _period_ , that is, how many numbers are generated before the sequence repeats.
  3. Take the pedagogical sequence of random numbers and look for correlations by observing clustering on a plot of successive pairs ( _x i, yi_) = ( _r_ 2 _i_ −1, _r_ 2 _i_ ), _i_ = 1, 2,... (Do _not_ connect the points with lines.) You may "see" correlations (Figure 4.1), which means that you should not use this sequence for serious work.
  4. Make your own version of Figure 4.2; that is, plot _r i_ vs. _i_.
  5. Test the built-in random-number generator on your computer for correlations by plotting the same pairs as above. (This should be good for serious work.)
  6. Test the linear congruent method again with reasonable constants like those in (4.11) and (4.12). Compare the scatterplot you obtain with that of the built-in random-number generator. (This, too, should be good for serious work.)

### 4.2.3 Assessing Randomness and Uniformity

Because the computer's random numbers are generated according to a definite rule, the numbers in the sequence must be correlated with each other. This can affect a simulation that assumes random events. Therefore, it is wise for you to test a random-number generator to obtain a numerical measure of its uniformity and randomness before you stake your scientific reputation on it. In fact, some tests are simple enough for you to make it a habit to run them simultaneously with your simulation. In the examples to follow, we test for randomness and uniformity.

  1. Probably the most obvious, but often neglected, test for randomness and uniformity is just to look at the numbers generated. For example, Table 4.1 presents some output from Python's `random` method. If you just look at these numbers you will know immediately that they all lie between 0 and 1, that they appear to differ from each other, and that there is no obvious pattern (like 0.3333).
  2. As we have seen, a quick visual test (Figure 4.2) involves taking this same list and plotting it with _r_ _i_ as ordinate and _i_ as abscissa. Observe how there appears to be a uniform distribution between 0 and 1 and no particular correlation between points (although your eye and brain will try to recognize some kind of pattern if you look long enough).

**Figure 4.2** A plot of a uniform pseudorandom sequence _r i_, vs. _i_. The points are connected to make it easier to follow the order. While this does not prove that a distribution is random, it at least shows the range of values and that there is fluctuation.

**Table 4.1** A table of a uniform, pseudo-random sequence _r i_ generated by Python's random method.

0.046 895 024 385 081 75 | 0.204 587 796 750 397 95 | 0.557 190 747 079 725 5 | 0.056 343 366 735 930 88
---|---|---|---
0.936 066 864 589 746 7 | 0.739 939 913 919 486 7 | 0.650 415 302 989 955 3 | 0.809 633 370 418 305 7
0.325 121 746 254 331 9 | 0.494 470 371 018 847 17 | 0.143 077 126 131 411 28 | 0.328 581 276 441 882 06
0.535 100 168 558 861 6 | 0.988 035 439 569 102 3 | 0.951 809 795 307 395 3 | 0.368 100 779 256 594 23
0.657 244 381 503 891 1 | 0.709 076 851 545 567 1 | 0.563 678 747 459 288 4 | 0.358 627 737 800 664 9
0.383 369 106 540 338 07 | 0.740 022 375 602 264 9 | 0.416 208 338 118 453 5 | 0.365 803 155 303 808 7
0.748 479 890 046 811 1 | 0.522 694 331 447 043 | 0.148 656 282 926 639 13 | 0.174 188 153 952 713 6
0.418 726 310 120 201 23 | 0.941 002 689 012 048 8 | 0.116 704 492 627 128 9 | 0.875 900 901 278 647 2
0.596 253 540 903 370 3 | 0.438 238 541 497 494 1 | 0.166 837 081 276 193 | 0.275 729 402 460 343 05
0.832 243 048 236 776 | 0.457 572 427 917 908 75 | 0.752 028 149 254 081 5 | 0.886 188 103 177 451 3
0.040 408 674 172 845 55 | 0.146 901 492 948 813 34 | 0.286 962 760 984 402 3 | 0.279 150 544 915 889 53
0.785 441 984 838 243 6 | 0.502 978 394 047 627 | 0.688 866 810 791 863 | 0.085 104 148 559 493 22
0.484 376 438 252 853 26 | 0.194 793 600 337 003 66 | 0.379 123 023 471 464 2 | 0.986 737 138 946 582 1
  3. A simple test of uniformity evaluates the _k_ th moment of a distribution:

(4.13)

If the numbers are distributed _uniformly_ , then (4.13) is approximately the moment of the distribution function _P_ ( _x_ ):

(4.14)

If (4.14) holds for your generator, then you know that the distribution is uniform. If the deviation from (4.14) varies as   then you _also_ know that the distribution is random because the   result derives from assuming randomness.

  4. Another simple test determines the near-neighbor correlation in your random sequence by taking sums of products for small _k:_

(4.15)

If your random numbers _x i_ and _x i+k_ are distributed with the joint probability distribution _P_ ( _x i, xi+k_) = 1 and are independent and uniform, then (4.15) can be approximated as an integral:

(4.16)

If (4.16) holds for your random numbers, then you know that they are uniform and independent. If the deviation from (4.16) varies as  , then you _also_ know that the distribution is random.

  5. As we have seen, an effective test for randomness is performed by making a scatterplot of ( _x t_ = _r_ 2 _i_ , _y_ _i_ = _r_ 2 _i_ +1) for many _i_ values. If your points have noticeable regularity, the sequence is not random. If the points are random, they should uniformly fill a square with no discernible pattern (a cloud) (as in Figure 4.1b).
  6. Test your random-number generator with (4.14) for _k_ = 1, 3, 7 and _N_ = 100, 10 000, 100 000. In each case print out

(4.17)

to check that it is of the order 1.

## 4.3 Random Walks (Problem)

Consider a perfume molecule released in the front of a classroom. It collides randomly with other molecules in the air and eventually reaches your nose despite the fact that you are hidden in the last row. Your _problem_ is to determine how many collisions, on the average, a perfume molecule makes in traveling a distance _R._ You are given the fact that a molecule travels an average ( _root-mean-square_ ) distance _r rms_ between collisions.

**Listing 4.1** **Walk.py** calls the random-number generator from the random package. Note that a different seed is needed to obtain a different sequence.

### 4.3.1 Random-Walk Simulation

There are a number of ways to simulate a random walk with (surprise, surprise) different assumptions yielding different physics. We will present the simplest approach for a 2D walk, with a minimum of theory, and end up with a model for _normal diffusion_. The research literature is full of discussions of various versions of this problem. For example, Brownian motion corresponds to the limit in which the individual step lengths approach zero, and with no time delay between steps. Additional refinements include collisions within a moving medium ( _abnormal diffusion_ ), including the velocities of the particles, or even pausing between steps. Models such as these are discussed in Chapter 16, and demonstrated by some of the corresponding applets given online.

In our random-walk simulation (Figure 4.3) an artificial _walker_ takes sequential steps with the _direction_ of each step _independent_ of the direction of the previous step. For our model, we start at the origin and take _N_ steps in the _xy_ plane of _lengths_ (not coordinates)

(4.18)

Although each step may be in a different direction, the distances along each Cartesian axis just add algebraically. Accordingly, the radial distance _R_ from the starting point after _N_ steps is

(4.19)

If the walk is random, the particle is equally likely to travel in any direction at each step. If we take the average of a large number of such random steps, all the cross terms in (4.19) will vanish and we will be left with

(4.20)

where   is the _root-mean-square_ step size.

To summarize, if the walk is random, then we expect that after a large number of steps the average _vector_ distance from the origin will vanish:

(4.21)

Yet   does not vanish. Equation 4.20 indicates that the average _scalar_ distance from the origin is   where each step is of average length _r rms_. In other words, the vector endpoint will be distributed uniformly in all quadrants, and so the displacement vector averages to zero, but the average length of that vector does not. For large _N_ values,   (the value if all steps were in one direction on a straight line), but does not vanish. In our experience, practical simulations agree with this theory, but rarely perfectly, with the level of agreement depending upon the details of how the averages are taken and how the randomness is built into each step.

**Figure 4.3** (a) A schematic of the _N_ steps in a random walk simulation that end up a distance _R_ from the origin. Note how the Δ _x_ 's for each step add vectorially. (b) A simulated walk in 3D from _Walk3D.py._

### 4.3.2 Implementation: Random Walk

The program `**Walk.py**` in Listing 4.1 is a sample random-walk simulation. It is key element is random values for the _x_ and _y_ components of each step,

Here we omit the scaling factor that normalizes each step to length 1. When using your computer to simulate a random walk, you should expect to obtain (4.20) only as the average displacement averaged over many trials, not necessarily as the answer for each trial. You may get different answers depending on just how you take your random steps (Figure 4.4b).

Start at the origin and take a 2D random walk with your computer.

  1. To increase the amount of randomness, independently choose random values for Δ _x'_ and Δ _y'_ in the range [−1, 1]. Then normalize them so that each step is of unit length

(4.22)

  2. Use a plotting program to draw maps of several independent 2D random walks, each of 1000 steps. Using evidence from your simulations, comment on whether these look like what you would expect of a random walk.

**Figure 4.4** (a) The steps taken in seven 2D random walk simulations. (b) The distance covered in two walks of _N_ steps using different schemes for including randomness. The theoretical prediction (4.20) is the straight line.

  3. If you have your walker taking _N_ steps in a single trial, then conduct a total number   of trials. Each trial should have _N_ steps and start with a different seed.
  4. Calculate the mean square distance _R_ 2 for each trial and then take the average of _R_ 2 for all your _K_ trials:

(4.23)

  5. Check the validity of the assumptions made in deriving the theoretical result (4.20) by checking how well

(4.24)

Do your checking for both a single (long) run and for the average over trials.

  6. Plot the root-mean-square distance   as a function of  -Values of _N_ should start with a small number, where   is not expected to be accurate, and end at a quite large value, where two or three places of accuracy should be expected on the average.
  7. Repeat the preceding and following analysis for a 3D walk as well.

## 4.4 Extension: Protein Folding and Self-Avoiding Random Walks

A protein is a large biological molecule made up of molecular chains (the residues of amino acids). These chains are formed from _monomers_ , that is, molecules that bind chemically with other molecules. More specifically, these chains consist of nonpolar hydrophobic (H) monomers that are repelled by water, and polar (P) monomers that are attracted by water. The actual structure of a protein results from a _folding process_ in which random coils of chains rearrange themselves into a configuration of minimum energy. We want to model that process on the computer.

Although molecular dynamics (Chapter 18) may be used to simulate protein folding, it is much slower than Monte Carlo techniques, and even then, it is hard to find the lowest energy states. Here we create a simple Monte Carlo simulation in which you to take a random walk in a 2D square lattice (Yue _et al._ , 2004). At the end of each step, you randomly choose an H or a P monomer and drop it on the lattice, with your choice weighted such that H monomers are more likely than P ones. The walk is restricted such that the only positions available after each step are the three neighboring sites, with the already-occupied sites excluded (this is why this technique is known as a _self-avoiding random walk_ ).

The goal of the simulation is to find the lowest energy state of an HP sequence of various lengths. These then may be compared to those in nature. Just how best to find such a state is an active research topic (Yue _et al._ , 2004). The energy of a chain is defined as

(4.25)

where _ _is a positive constant and _f_ is the number of H–H neighbor _not_ connected directly (P–P and H–P bonds do not count at lowering the energy). So if the neighbor next to an H is another H, it lowers the energy, but if it is a P it does not lower the energy. We show a typical simulation result in Figure 4.5, where a light dot is placed half way between two H (dark-dot) neighbors. Accordingly, for a given length of chain, we expect the natural state(s) of an H–P sequence to be those with the largest possible number _f_ of H–H contacts. That is what we are looking for.

  1. Modify the random walk program we have already developed so that it simulates a self-avoiding random walk. The key here is that the walk stops at a corner, or when there are no empty neighboring sites available.
  2. Make a random choice as to whether the monomer is an H or a P, with a weighting such that there are more H's than P's.
  3. Produce a visualization that shows the positions occupied by the monomers, with the H and P monomers indicated by different color dots. (Our visualization, shown in Figure 4.5, is produced by the program `ProteinFold.py`, available on the Instructor's site.)
  4. After the walk ends, record the energy and length of the chain.

**Figure 4.5** Two self-avoiding random walks that simulate protein chains with hydrophobic (H) monomers in light gray, and polar (P) monomers in black. The dark dots indicate regions where two H monomers are not directly connected.

  5. Run many folding simulations and save the outputs, categorized by length and energy.
  6. Examine the state(s) of the lowest energy for various chain lengths and compare the results to those from molecular dynamic simulations and actual protein structures (available on the Web).
  7. Do you think that this simple model has some merit?
  8.   Extend the folding to 3D.

## 4.5 Spontaneous Decay (Problem)

Your _problem_ is to simulate how a small number _N_ of radioactive particles decay.2) In particular, you are to determine when radioactive decay looks like exponential decay and when it looks _stochastic_ (containing elements of chance). Because the exponential decay law is a large-number approximation to a natural process that always leads to only a small number of nuclei remaining, our simulation should be closer to nature than is the exponential decay law (Figure 4.6). In fact, if you "listen" to the output of the decay simulation code, what you will hear sounds very much like a Geiger counter, an intuitively convincing demonstration of the realism of the simulation.

Spontaneous decay is a natural process in which a particle, with no external stimulation, decays into other particles. Although the probability of decay of any one particle in any time interval is constant, just when it decays is a random event. Because the exact moment when any one particle decays is always random, and because one nucleus does not influence another nucleus, the probability of decay is not influenced by how long the particle has been around or whether some other particles have decayed. In other words, the probability _P_ of any one particle decaying per unit time interval is a constant, yet when that particle decays it is gone forever. Of course, as the total number _N_ of particles decreases with time, so will the number that decay per unit time, but the probability of any one particle decaying in some time interval remains the same as long as that particle exists.

**Figure 4.6** _Circle_ : A sample containing _N_ nuclei, each of which has the same probability of decaying per unit time, _Graphs_ : Semilog plots of the number of nuclei vs. time for five simulations with differing initial numbers of nuclei. Exponential decay would be a straight line with bumps, similar to the initial behavior for _N_ = 100 000.

### 4.5.1 Discrete Decay (Model)

Imagine having a sample containing _N_ ( _t_ ) radioactive nuclei at time _t_ (Figure 4.6 circle). Let Δ _N_ be the number of particles that decay in some small time interval Δ _t_. We convert the statement "the probability _P_ of any one particle decaying per unit time is a constant" into the equation

(4.26)

(4.27)

where the constant _λ_ is called the _decay rate_ and the minus sign indicates a decreasing number. Because _N_ ( _t_ ) decreases in time, the _activity_ Δ _N_ ( _t_ )/Δ _t_ (sometimes called the decay rate) also decreases with time. In addition, because the total activity is proportional to the total number of particles present, it is too stochastic with an exponential-like decay in time. (Actually, because the number of decays Δ _N_ ( _t_ ) is proportional to the difference in random numbers, its tends to show even larger statistical fluctuations than does _N_ ( _t_ ).)

Equation 4.27 is a _finite-difference equation_ relating the experimentally quantities _N_ ( _t_ ), Δ _N_ ( _t_ ), and Δ _t._ Although a difference equation cannot be integrated the way a differential equation can, it can be simulated numerically. Because the process is random, we cannot predict a single value for Δ _N_ ( _t_ ), although we can predict the average number of decays when observations are made of many identical systems of _N_ decaying particles.

### 4.5.2 Continuous Decay (Model)

When the number of particles _N_ → ∞ and the observation time interval Δ _t_ → 0, our difference equation becomes a differential equation, and we obtain the familiar exponential decay law (4.27):

(4.28)

This can be integrated to obtain the time dependence of the total number of particles and of the total activity:

(4.29)

(4.30)

In this limit, we can identify the decay rate _λ_ with the inverse lifetime:

(4.31)

We see from its derivation that the exponential decay is a good description of nature for a large number of particles where Δ _N/N_   0. However, in nature _N_ ( _t_ ) can be a small number, and in that case we have a statistical and not a continuous process. The basic law of nature (4.26) is always valid, but as we will see in the simulation, the exponential decay (4.30) becomes less and less accurate as the number of particles gets smaller and smaller.

### 4.5.3 Decay Simulation with Geiger Counter Sound

A program for simulating radioactive decay is surprisingly simple but not without its subtleties. We increase time in discrete steps of Δ _t_ , and for each time interval we count the number of nuclei that have decayed during that Δ _t._ The simulation quits when there are no nuclei left to decay. Such being the case, we have an outer loop over the time steps Δ _t_ and an inner loop over the remaining nuclei for each time step. The pseudocode is simple (as is the code):

When we pick a value for the decay rate _λ_ = 1/ _τ_ to use in our simulation, we are setting the scale for times. If the actual decay rate is _λ_ = 0.3 × 106 s−1 and if we decide to measure times in units of 10−6 s, then we will choose random numbers 0 ≤ _r i_ 1, which leads to _λ_ values lying someplace near the middle of the range (e.g., _λ_   0.3). Alternatively, we can use a value of _λ_ = 0.3 × 106 s−1 in our simulation and then scale the random numbers to the range 0 ≤ _r i_ ≤ 106. However, unless you plan to compare your simulation to experimental data, you do not have to worry about the scale for time but instead should focus on the physics behind the slopes and relative magnitudes of the graphs.

**Listing 4.2** **DecaySound.py** simulates spontaneous decay in which a decay occurs if a random number is smaller than the decay parameter. The _winsound_ package lets us play a beep each time there is a decay, and this leads to the sound of a Geiger counter.

**Decay.py** is our sample simulation of the spontaneous decay. An extension of this program, **DecaySound.py,** in Listing 4.2, adds a beep each time an atom decays (unfortunately this works only with Windows). When we listen to the simulation, it sounds like a Geiger counter, with its randomness and its slowing down with time. This provides some rather convincing evidence of the realism of the simulation.

## 4.6 Decay Implementation and Visualization

Write a program to simulate the radioactive decay using the simple program in Listing 4.2 as a guide. You should obtain results like those in Figure 4.6.

  1. Plot the logarithm of the number left In _N_ ( _t_ ) and the logarithm of the decay rate Δ _N_ ( _t_ )/Δ _t_ (= 1) vs. time. Note that the simulation measures time in steps of Δ _t_ (generation number).
  2. Check that you obtain what looks like the exponential decay when you start with large values for _N_ ( _0_ ), but that the decay displays its stochastic nature for small _N_ ( _0_ ). (Large _N_ ( _0_ ) values are also stochastic; they just do not look like it.)
  3. Create two plots, one showing that the slopes of _N_ ( _t_ ) vs. _t_ are _independent_ _N_ ( _0_ ) and another showing that the slopes are proportional to the value chosen for _λ_.
  4. Create a plot showing that within the expected statistical variations, In _N_ ( _t_ ) and In Δ _N_ ( _t_ ) are proportional.
  5. Explain in your own words how a process that is spontaneous and random at its very heart can lead to the exponential decay.
  6. How does your simulation show that the decay is exponential-like and not a power law such as _N_ = _βt −α?_

1) You may obtain the same result for the modulus operation by subtracting _M_ until any further subtractions would leave a negative number; what remains is the _remainder._ 2) Spontaneous decay is also discussed in Chapter 7, where we fit an exponential function to a decay spectrum.

# 5
Differentiation and Integration

* * *

We start this chapter with a short discussion of numerical differentiation, an important but rather simple topic. We derive the forward-difference, central-difference, and extrapolated-difference methods for differentiation. They will be used throughout the book. The majority of this chapter deals with numerical integration, a basic tool of scientific computation. We derive Simpson's rule, the trapezoid rule, and the Gaussian quadrature rule. We discuss Gaussian quadrature (our personal workhorse) in its various forms, and indicate how to map the standard Gauss points to a wide range of intervals. We end the chapter with a discussion of Monte Carlo integration techniques, which are fundamentally different from all the other rules.

* * *

## 5.1 Differentiation

Problem Figure 5.1 shows the trajectory of a projectile with air resistance. The dots indicate the times _t_ at which measurements were made and tabulated. Your _problem_ is to determine the velocity _dy/dt_ as a function of time. Note that because there is realistic air resistance present, there is no analytic function to differentiate, only this graph or a table of numbers read from it.

You probably did rather well in your first calculus course and feel competent at taking derivatives. However, you may never have taken derivatives of a table of numbers using the elementary definition

(5.1)

In fact, even a computer runs into errors with this kind of limit because it is wrought with subtractive cancelation; as _h_ is made smaller, the computer's finite word length causes the numerator to fluctuate between 0 and the machine precision _ _m , and as the denominator approaches zero, overflow occurs.

**Figure 5.1** A trajectory of a projectile experiencing air resistance. Forward-difference approximation (slanted dashed line) and central-difference approximation (horizontal line) for the numerical first derivative at time _t_. (A tangent to the curve at _t_ yields the correct derivative.) The central difference is seen to be more accurate.

## 5.2 Forward Difference (Algorithm)

The most direct method for numerical differentiation starts by expanding a function in a Taylor series to obtain its value at a small step _h_ away:

(5.2)

(5.3)

If we ignore the _h_ 2 terms in (5.3), we obtain the _forward-difference_ derivative algorithm for the derivative (5.2) for _y_ ′( _t_ ):

(5.4)

An estimate of the error follows from substituting the Taylor series:

(5.5)

You can think of this approximation as using two points to represent the function by a straight line in the interval from _x_ to _x_ \+ _h_ (Figure 5.1a). The approximation (5.4) has an error proportional to _h_ (unless the heavens look down upon you kindly and make _y_ ″ vanish). We can make the approximation error smaller by making _h_ smaller, yet precision will be lost through the subtractive cancelation on the left-hand side (LHS) of (5.4) for too small an _h._

To see how the forward-difference algorithm works, let  . The exact derivative is _y_ ′ = 2 _bt_ , while the computed derivative is

(5.6)

This clearly becomes a good approximation only for small  .

## 5.3 Central Difference (Algorithm)

An improved approximation to the derivative starts with the basic definition (5.1) or geometrically as shown in Figure 5.1b. Now, rather than making a single step of _h_ forward, we form a _central difference_ by stepping forward half a step and backward half a step:

(5.7)

We estimate the error in the central-difference algorithm by substituting the Taylor series for   and   into (5.7):

(5.8)

The important difference between this central-difference algorithm and the forward difference one is that when   is subtracted from  , all terms containing an even power of _h_ in the two Taylor series cancel. This make the central-difference algorithm accurate to order _h_ 2 ( _h_ 3 before division by _h_ ), while the forward difference is accurate only to order _h._ If the _y_ ( _t_ ) is smooth, that is, if  , then you can expect the central-difference error to be smaller than the one with the central-difference algorithm.

If we now return to our parabola example (5.6), we will see that the central difference gives the exact derivative independent of _h_ :

(5.9)

This is to be expected because the higher derivatives equal zero for a second-order polynomial.

## 5.4 Extrapolated Difference (Algorithm)

Because a differentiation rule based on keeping a certain number of terms in a Taylor series also provides an expression for the error (the terms not included), we can reduce the theoretical error further by forming a combination of approximations whose summed errors extrapolate to zero. One such algorithm is the central-difference algorithm (5.7) using a half-step back and a half-step forward. A second algorithm is another central-difference approximation, but this time using quarter-steps:

(5.10)

A combination of the two, called the _extended difference algorithm,_ eliminates both the quadratic and linear terms:

(5.11)

(5.12)

Here (5.11) is the extended-difference algorithm and (5.12) gives its error, with _D_ cd representing the central-difference algorithm. If _h_ = 0.4 and   then there will be only one place of the round-off error and the truncation error will be approximately machine precision _ _m ; this is the best you can hope for.

When working with these and similar higher order methods, it is important to remember that while they may work as designed for well-behaved functions, they may fail badly for functions containing noise, as do data from computations or measurements. If noise is evident, it may be better to first smooth the data or fit them with some analytic function using the techniques of Chapter 7 and then differentiate.

## 5.5 Error Assessment

The approximation errors in numerical differentiation decrease with decreasing step size _h._ In turn, round-off errors increase with decreasing step size because you have to take more steps and do more calculations. Remember from our discussion in Chapter 3 that the best approximation occurs for an _h_ that makes the total error   a minimum, and that as a rough guide this occurs when

We have already estimated the approximation error in numerical differentiation rules by making a Taylor series expansion of   The approximation error with the forward-difference algorithm (5.4) is  , while that with the central-difference algorithm (5.8) is  :

(5.13)

To obtain a rough estimate of the round-off error, we observe that differentiation essentially subtracts the value of a function at argument _x_ from that of the same function at argument _x_ \+ _h_ and then divide by   As _h_ is made continually smaller, we eventually reach the round-off error limit where   and _y_ ( _t_ ) differ by just machine precision _ _m :

(5.14)

Consequently, round-off and approximation errors become equal when

(5.15)

(5.16)

(5.17)

We take   (which may be crude in general, although not bad for _e t_ or cos _t_ ) and assume double precision,

(5.18)

(5.19)

This may seem backward because the better algorithm leads to a larger _h_ value. It is not. The ability to use a larger _h_ means that the error in the central-difference method is about 1000 times smaller than the error in the forward-difference method.

The programming for numerical differentiation is simple:

  1. Use forward-, central-, and extrapolated-difference algorithms to differentiate the functions cos _t_ and e _t_ at _t_ = 0.1, 1., and 100.
    1. a) Print out the derivative and its relative error _ _as functions of _h._ Reduce the step size _h_ until it equals machine precision
    2. b) Plot   vs. log10 _h_ and check whether the number of decimal places obtained agrees with the estimates in the text.
    3. c) See if you can identify regions where algorithmic (series truncation) error dominates at large _h_ and the round-off error at small _h_ in your plot. Do the slopes agree with our model's predictions?

## 5.6 Second Derivatives (Problem)

Let us say that you have measured the position _y_ ( _t_ ) vs. time for a particle (Figure 5.1). Your _problem_ now is to determine the force on the particle. Newton's second law tells us that force and acceleration are linearly related:

(5.20)

where _F_ is the force, _m_ is the particle's mass, and _a_ is the acceleration. So by determining the derivative   from the _y_ ( _t_ ) values, we determine the force.

The concerns we expressed about errors in first derivatives are even more valid for second derivatives where additional subtractions may lead to additional cancelations. Let us look again at the central-difference method:

(5.21)

This algorithm gives the derivative at _t_ by moving forward and backward from _t_ by _h_ /2. We take the second derivative   to be the central difference of the first derivative:

(5.22)

(5.23)

As we did for the first derivatives, we determine the second derivative at _t_ by evaluating the function in the region surrounding _t._ Although the form (5.23) is more compact and requires fewer steps than (5.22), it may increase subtractive cancelation by first storing the "large" number   and then subtracting another large number 2 _y_ ( _t_ ) from it. We ask you to explore this difference as an exercise.

### 5.6.1 Second-Derivative Assessment

Write a program to calculate the second derivative of cos _t_ using the central-difference algorithms (5.22) and (5.23). Test it over four cycles. Start with   and keep reducing _h_ until you reach machine precision. Is there any noticeable differences between (5.22) and (5.23)?

## 5.7 Integration

**Problem: Integrating a Spectrum** An experiment has measured   the number of particles entering a counter per unit time. Your _problem_ is to integrate this spectrum to obtain the number of particles _N_ (1) that entered the counter in the first second:

(5.24)

## 5.8 Quadrature as Box Counting (Math)

The integration of a function may require some cleverness to do analytically, but is relatively straightforward on a computer. A traditional way to perform numerical integration by hand is to take a piece of a graph paper and count the number of boxes or _quadrilaterals_ lying below a curve of the integrand. For this reason, numerical integration is also called _numerical quadrature_ even when it becomes more sophisticated than simple box counting.

The Riemann definition of an integral is the limit of the sum over boxes as the width _h_ of the box approaches zero (Figure 5.2):

(5.25)

The numerical integral of a function _f_ ( _x_ ) is approximated as the equivalent of a finite sum over boxes of height _f_ ( _x_ ) and width _w_ _i_ :

(5.26)

**Figure 5.2** The integral   d _x_ is the area under the graph of _f_ ( _x_ ) from _a_ to _b._ Here we break up the area into four regions of equal widths _h_ and five integration points.

which is similar to the Riemann definition (5.25) except that there is no limit to an infinitesimal box size. Equation 5.26 is the standard form for all integration algorithms; the function _f_ ( _x_ ) is evaluated at _N_ points in the interval [ _a,b_ ], and the function values   are summed with each term in the sum weighted by _w i_. While, in general, the sum in (5.26) gives the exact integral only when _N_ → ∞, it may be exact for finite _N_ if the integrand is a polynomial. The different integration algorithms amount to different ways of choosing the points _x_ _i_ and weights _w i_. Generally, the precision increases as _N_ gets larger, with round-off error eventually limiting the increase. Because the "best" integration rule depends on the specific behavior of _f_ ( _x_ ), there is no universally best approximation. In fact, some of the automated integration schemes found in subroutine libraries switch from one method to another, as well as change the methods for different intervals until they find ones that work well for each interval.

In general, you should not attempt a numerical integration of an integrand that contains a singularity without first removing the singularity by hand.1) You may be able to do this very simply by breaking the interval down into several subintervals so the singularity is at an endpoint where an integration point is not placed or by a change of variable:

(5.27)

(5.28)

(5.29)

Likewise, if your integrand has a very slow variation in some region, you can speed up the integration by changing to a variable that compresses that region and places few points there, or divides up the interval and performs several integrations. Conversely, if your integrand has a very rapid variation in some region, you may want to change to variables that expand that region to ensure that no oscillations are missed.

**Listing 5.1** **TrapMethods.py** integrates a function   with the trapezoid rule. Note that the step size _h_ depends upon the size of interval here and that the weights at the ends and middle of the intervals differ.

## 5.9 Algorithm: Trapezoid Rule

Trapezoid and Simpson integration rules use evenly spaced values of _x_ (Figure 5.3). They use _N_ points   evenly spaced at a distance _h_ apart throughout the integration region [ _a, b_ ] and _include the endpoints_ in the integration region. This means that there are ( _N_ − 1) intervals of length _h_ :

(5.30)

where we start our counting at _i_ = 1. The trapezoid rule takes each integration interval _i_ and constructs a trapezoid of width _h_ in it (Figure 5.3). This approximates _f_ ( _x_ ) by a straight line in each interval _i_ and uses the average height   as the value for _f_. The area of each such trapezoid is

(5.31)

**Figure 5.3** Different shapes used to approximate the areas under the curve. (a) Straight-line sections used for the trapezoid rule. (b) Two parabolas used in Simpson's rule.

**Table 5.1** Elementary weights for uniform-step integration rules.

Name | Degree | Elementary weights
---|---|---
Trapezoid | 1 |
Simpson's | 2 |
  | 3 |
Milne | 4 |

In terms of our standard integration formula (5.26), the "rule" in (5.31) is for   points with weights   (Table 5.1).

In order to apply the trapezoid rule to the entire region [ _a, b_ ], we add the contributions from each subinterval:

(5.32)

You will notice that because the internal points are counted twice (as the end of one interval and as the beginning of the next), they have weights of  , whereas the endpoints are counted just once, and on that account have weights of only _h_ /2. In terms of our standard integration rule (5.57), we have

(5.33)

In Listing 5.1, we provide a simple implementation of the trapezoid rule.

## 5.10 Algorithm: Simpson's Rule

Simpson's rule approximates the integrand _f_ ( _x_ ) by a parabola for each interval (Figure 5.3b):

(5.34)

Again, all intervals equally spaced. The area under the parabola for each interval is

(5.35)

In order to relate the parameters _α, β,_ and _γ_ to the function, we consider an interval from −1 to +1, in which case

(5.36)

But we notice that

(5.37)

(5.38)

In this way, we can express the integral as the weighted sum over the values of the function at three points:

(5.39)

Because three values of the function are needed, we generalize this result to our problem by evaluating the integral over two adjacent intervals, in which case we evaluate the function at the two endpoints and in the middle (Table 5.1):

(5.40)

Simpson's rule requires the elementary integration to be over _pairs_ of intervals, which in turn requires that the _total number of intervals be even or that the number of points N be odd._ In order to apply Simpson's rule to the entire interval, we add up the contributions from each pair of subintervals, counting all but the first and last endpoints twice:

(5.41)

In terms of our standard integration rule (5.26), we have

(5.42)

The sum of these weights provides a useful check on your integration:

(5.43)

_Remember,_ the number of points _N_ must be odd for Simpson's rule.

## 5.11 Integration Error (Assessment)

In general, you should choose an integration rule that gives an accurate answer using the least number of integration points. We obtain a crude estimate of the _approximation_ or _algorithmic error ε_ for the equal-spacing rules and their relative error _ _by expanding _f_ ( _x_ ) in a Taylor series around the midpoint of the integration interval. We then multiply that error by the number of intervals _N_ to estimate the error for the entire region [ _a, b_ ]. For the trapezoid and Simpson rules this yields

(5.44)

where _ _is a measure of the relative error. We see that the third-derivative term in Simpson's rule cancels (much like the central-difference method does in differentiation).Equations 5.44 are illuminating in showing how increasing the sophistication of an integration rule leads to an error that decreases with a higher inverse power of _N_ , yet it is also proportional to higher derivatives of _f_. Consequently, for small intervals and functions _f_ ( _x_ ) with well-behaved derivatives, Simpson's rule should converge more rapidly than the trapezoid rule.

To model the round-off error in integration, we assume that after _N_ steps the _relative_ round-off error is random and of the form

(5.45)

where _ _m is the machine precision,   for single precision and   for double precision (the standard for science). Because most scientific computations are performed with doubles, we will assume double precision. We want to determine an _N_ that minimizes the total error, that is, the sum of the approximation and round-off errors:

(5.46)

This occurs, approximately, when the two errors are of equal magnitude, which we approximate even further by assuming that the two errors are equal:

(5.47)

To continue the search for optimum _N_ for a general function _f_ , we set the scale of function size and the lengths by assuming

(5.48)

The estimate (5.47), when applied to the _trapezoid rule,_ yields

(5.49)

(5.50)

(5.51)

The estimate (5.47), when applied to _Simpson's rule,_ yields

(5.52)

(5.53)

(5.54)

These results are illuminating in that they show how

  * Simpson's rule requires fewer point and has less error than the trapezoid rule.
  * It is possible to obtain an error close to machine precision with Simpson's rule (and with other higher order integration algorithms).
  * Obtaining the _best_ numerical approximation to an integral is not achieved by letting _N_ → ∞ but with a relatively small _N_ ≤ 1000. Larger _N_ only makes the round-off error dominate.

## 5.12 Algorithm: Gaussian Quadrature

It is often useful to rewrite the basic integration formula (5.26) with a weighting function _W_ ( _x_ ) separate from the integrand:

(5.55)

In the Gaussian quadrature approach to integration, the _N_ points and weights in (5.55) are chosen to make the integration exact if _g_ ( _x_ ) were a (2 _N_ − 1)-degree polynomial. To obtain this incredible optimization, the points _x_ _i_ end up having a specific distribution over [ _a, b_ ]. In general, if _g_ ( _x_ ) is smooth or can be made smooth by factoring out some _W_ ( _x_ ) (Table 5.2), Gaussian quadrature will produce higher accuracy than the trapezoid and Simpson rules for the same number of points. Sometimes the integrand may not be smooth because it has different behaviors in different regions. In these cases, it makes sense to integrate each region separately and then add the answers together. In fact, some "smart" integration subroutines decide for themselves how many intervals to use and which rule to use in each.

All the rules indicated in Table 5.2 are Gaussian with the general form (5.55). We can see that in one case the weighting function is an exponential, in another a Gaussian, and in several an integrable singularity. In contrast to the equally spaced rules, there is never an integration point at the extremes of the intervals, yet the values of the points and weights change as the number of points _N_ changes, and the points are not spaced equally.

**Table 5.2** Types of Gaussian integration rules.

Integral | Name | Integral | Name
---|---|---|---
  | Gauss |   | Gauss–Chebyshev
  | Gauss–Hermite |   | Gauss–Laguerre
  | Associated Gauss–Laguerre |  |

**Table 5.3** Points and weights for 4-point Gaussian quadrature (for checking computation).

± _y_ _i_ | _w_ _i_
---|---
0.339 981 043 584 856 | 0.652 145 154 862 546
0.861 136 311 594 053 | 0.347 854 845 137 454

The derivation of the Gaussian points will be outlined below, but we point out here that for ordinary Gaussian (Gauss–Legendre) integration, the points _y_ _i_ turn out to be the _N_ zeros of the Legendre polynomials, with the weights related to the derivatives

(5.56)

Programs to generate these points and weights are standard in mathematical function libraries, are found in tables such as those in (Abramowitz and Stegun, 1972), or can be computed. The _gauss_ program we provide also scales the points to span specified regions. As a check that the program's points are correct, you may want to compare them to the four-point set given in Table 5.3.

### 5.12.1 Mapping Integration Points

Our standard convention (5.26) for the general interval [ _a, b_ ] is

(5.57)

With Gaussian points and weights, the _y_ interval   must be _mapped_ onto the _x_ interval  . Here are some mappings we have found useful in our work. In all cases,   are the elementary Gaussian points and weights for the interval [−1, 1], and we want to scale to _x_ with various ranges.

  1.

(5.58)

(5.59)

  2.

(5.60)

  3.  , _scale set by a:_

(5.61)

  4.

(5.62)

  5.

(5.63)

As you can see, even if your integration range extends out to infinity, there will be points at large but not infinite _x._ As you keep increasing the number of grid points _N,_ the last _x i_ gets larger but always remains finite.

### 5.12.2 Gaussian Points Derivation

We want to perform a numerical integration with _N_ integration points:

(5.64)

where _f_ ( _x_ ) is a polynomial of degree 2 _N_ − 1 or less. The unique property of Gaussian quadrature is that (5.64) will be exact, as long as we ignore the effect of round-off error. Determining the _x_ _i_ 's and _w_ _i_ 's requires some knowledge of special functions and some cleverness (Hildebrand, 1956). The knowledge needed is the two properties of Legendre polynomials _P_ _N_ ( _x_ ) of order _N_ :

  1. _P_ _N_ ( _x_ ) is orthogonal to every polynomial of order less than _N._
  2. _P_ _N_ ( _x_ ) has _N_ real roots in the interval  .

We define now a new polynomial of degree equal to or less than _N_ obtained by dividing the integrand _f_ ( _x_ ) by the Legendre polynomial _P_ _N_ ( _x_ ):

(5.65)

(5.66)

Here the remainder _r_ ( _x_ ) is an (unknown) polynomial of degree _N_ or less, which we will not need to determine. If we now substitute (5.66) into (5.64), and use the fact that _P N_ is orthogonal to every polynomial of degree less than or equal to _N,_ only the second, _r_ ( _x_ ), term remains

(5.67)

Yet because _r_ ( _x_ ) is a polynomial of degree _N_ or less, we can use a standard _N_ point rule to evaluate the integral exactly (the type of quadrature we did with the Simpson rule).

Now that we know it is possible to integrate a 2 _N_ − 1 or less degree polynomial with just _N_ points, we extert some cleverness to determine just what those points will be. We substitute (5.66) into (5.64) and note that

(5.68)

The cleverness is realizing that if we choose the _N_ integration points to be the zeros or roots of the Legendre polynomial _P_ _N_ ( _x_ ), then the first term on the RHS of (5.68) will vanish because   for each _x i_:

(5.69)

This is our derivation that the _N_ integration points over the interval (−1, 1) are the _N_ zeros of the Legendre polynomial _P_ _N_ ( _x_ ). As indicated in (5.56), the weights are related to the derivative of the Legendre polynomials evaluated at the roots of the polynomial. The actual derivation of the weights we leave to Hildebrand (1956).

### 5.12.3 Integration Error Assessment

  1. Write a double-precision program to integrate an arbitrary function numerically using the trapezoid rule, the Simpson rule, and Gaussian quadrature. For our assumed _problem,_ there is an analytic answer with which to compare:

(5.70)

  2. Compute the relative error _ _= |(numerical-exact)/exact| in each case. Present your data in the tabular form _N_ | _ T_ | _ S_ | _ G_
---|---|---|---
2 |... |... |...
10 |... |... |...

with spaces or tabs separating the fields. Try _N_ values of 2, 10, 20, 40, 80, 160,... ( _Hint_ : Even numbers may not be the assumption of every rule.)

  3. Make a log–log plot of relative error vs. _N_ (Figure 5.4). You should observe that

(5.71)

This means that a power-law dependence appears as a straight line on a log–log plot, and that if you use log10, then the ordinate on your log–log plot will be the negative of the number of decimal places of precision in your calculation.
  4. Use your plot or table to estimate the power-law dependence of the error _ _on the number of points _N,_ and to determine the number of decimal places of precision in your calculation. Do this for both the trapezoid and Simpson rules and for both the algorithmic and round-off error regimes. (Note that it may be hard to make _N_ large enough to reach the round-off error regime for the trapezoid rule because the approximation error is so large.)

**Figure 5.4** Log–log plots of the error in the integration of exponential decay using the trapezoid rule, Simpson's rule, and Gaussian quadrature vs. the number of integration points _N_. Approximately 15 decimal places of precision are attainable with double precision (a), and seven places with single precision (b). The algorithms are seen to stop converging when round-off error (the fluctuating and increasing part near the bottom) starts to dominate.

In Listing 5.2, we give a sample program that performs an integration with Gaussian points. The method `**gauss**` generates the points and weights and may be useful in other applications as well.

**Listing 5.2** `**IntegGauss.py**` integrates the function _f_ ( _x_ ) via Gaussian quadrature. The points and weights are generated in the method gauss, which will be the same for other applications as well. Note that the level of desired precision is set by the parameter eps, which should be set by the user, as should the value for job, which controls the mapping of the points onto arbitrary intervals (they are generated in (−1, 1)).

## 5.13 Higher Order Rules (Algorithm)

As in numerical differentiation, we can use the known functional dependence of the error on interval size _h_ to reduce the integration error. For simple rules like the trapezoid and Simpson rules, we have the analytic estimates (5.47), while for others you may have to experiment to determine the _h_ dependence. To illustrate, if _A_ ( _h_ ) and _A_ ( _h_ /2) are the values of the integral determined for intervals _h_ and _h_ /2, respectively, we know that the integrals have expansions with a leading error term proportional to _h_ 2,

(5.72)

(5.73)

Consequently, we make the _h_ 2 term vanish by computing the combination

(5.74)

Clearly, this particular trick (Romberg's extrapolation) works only if the _h_ 2 term dominates the error and then only if the derivatives of the function are well behaved. An analogous extrapolation can also be made for other algorithms.

In Table 5.1, we gave the weights for several equal interval rules. Whereas the Simpson rule used two intervals, the three-eighths rule uses three intervals, and the Milne2) rule uses four intervals. (These are single-interval rules and must be strung together to obtain a rule _extended_ over the entire integration range. This means that the points that end one interval and begin the next are weighted twice.) You can easily determine the number of elementary intervals integrated over, and check whether you and we have written the weights right, by summing the weights for any rule. The sum is the integral of _f_ ( _x_ ) = 1 and must equal _h_ times the number of intervals (which in turn equals _b_ – _a_ ):

(5.75)

## 5.14 Monte Carlo Integration by Stone Throwing (Problem)

Imagine yourself as a farmer walking to your furthermost field to add algae-eating fish to a pond having an algae explosion. You get there only to read the instructions and discover that you need to know the area of the pond in order to determine the correct number of the fish to add. Your _problem_ is to measure the area of this irregularly shaped pond with just the materials at hand (Gould _et al.,_ 2006).

It is hard to believe that Monte Carlo techniques can be used to evaluate integrals. After all, we do not want to gamble on the values! While it is true that other methods are preferable for single and double integrals, it turns out that Monte Carlo techniques are best when the dimensionality of integrations gets large! For our pond problem, we will use a _sampling_ technique (Figure 5.5):

  1. Walk off a box that completely encloses the pond and remove any pebbles lying on the ground within the box.
  2. Measure the lengths of the sides in natural units like _feet_. This tells you the area of the enclosing box _A_ box.
  3. Grab a bunch of pebbles, count their number, and then throw them up in the air in random directions.
  4. Count the number of splashes in the pond _N_ pond and the number of pebbles lying on the ground within your box _N_ box.
  5. Assuming that you threw the pebbles uniformly and randomly, the number of pebbles falling into the pond should be proportional to the area of the pond _A_ pond. You determine that area from the simple ratio

(5.76)

### 5.14.1 Stone Throwing Implementation

Use sampling (Figure 5.5) to perform a 2D integration and thereby determine π:

  1. Imagine a circular pond enclosed in a square of side
  2. We know the analytic answer that the area of a circle
  3. Generate a sequence of random numbers
  4. For   to _N,_ pick
  5. If  , let  ; otherwise let
  6. Use (5.76) to calculate the area, and in this way π.
  7. Increase _N_ until you get π to three significant figures (we don't ask much – that's only slide-rule accuracy).

**Figure 5.5** (a) Stones into a pond as a technique for measuring its area. The ratio of "hits" to total number of stones thrown equals the ratio of the area of the pond to that of the box. (b) The evaluation of an integral via a Monte Carlo (stone throwing) technique of the ratio of areas.

## 5.15 Mean Value Integration (Theory and Math)

The standard Monte Carlo technique for integration is based on the _mean value theorem_ (presumably familiar from elementary calculus):

(5.77)

The theorem states the obvious if you think of integrals as areas. The value of the integral of some function _f_ ( _x_ ) between _a_ and _b_ equals the length of the interval   times the mean value of the function over that interval   (Figure 5.6). The Monte Carlo integration algorithm uses random points to evaluate the mean in (5.77). With a sequence   of _N_ uniform random numbers, we want to determine the _sample mean_ by _sampling_ the function _f_ ( _x_ ) at these points:

(5.78)

This gives us the very simple integration rule:

(5.79)

**Figure 5.6** The area under the curve _f_ ( _x_ ) is the same as that under the horizontal line whose height  .

Equation 5.79 looks much like our standard algorithm for integration (5.26) with the points _x_ _i_ chosen randomly and with uniform weights   Because no attempt has been made to obtain an optimal answer for a given value of _N,_ this does not seem like it would be an efficient means to evaluate integrals; but you must admit that it is simple. If we let the number of samples of _f_ ( _x_ ) approach infinity _N_ → ∞ or if we keep the number of samples finite and take the average of infinitely many runs, the laws of statistics ensure us that (5.79) will approach the correct answer, at least if there were no round-off errors.

For readers who are familiar with statistics, we remind you that the uncertainty in the value obtained for the integral _I_ after _N_ samples of _f_ ( _x_ ) is measured by the standard deviation _σ_ _I_ If _σ_ _f_ is the standard deviation of the integrand _f_ in the sampling, then for normal distributions we have

(5.80)

So for large _N,_ the error in the value obtained for the integral decreases as  .

## 5.16 Integration Exercises

  1. Here are two integrals that quadrature may find challenging:

(5.81)

    1. a) Evaluate these integrals using two different integration rules and compare the answers.
    2. b) Explain why the computer may have trouble with these integrals.
  2. The next three problems are examples of how _elliptic integrals_ enter into realistic physics problems. It is straightforward to evaluate any integral numerically using the techniques of this chapter, but it may be difficult for you to know if the answers you obtain are correct. One way to hone your integral-evaluating skills is to compare your answers from quadrature to power series expressions, or to a polynomial approximations of know precision. To help you in this regard, we present here a polynomial approximation for an elliptic integral (Abramowitz and Stegun, 1972):

(5.82)

  3. Compute _K_ ( _m_ ) by evaluating the integral in (5.82) numerically. Tune your integral evaluation until you obtain agreement at the   level with the polynomial approximation.
  4. In Section 15.1.2, we will derive an expression for the period _T_ of a realistic pendulum for which the maximum angle of displacement _θ_ m is not necessarily small:

(5.83)

(5.84)

where _T_ 0 is the period for small-angle oscillations. The integral in (5.83) can be expressed in terms of an elliptic integral of the first kind. If you think of an elliptic integral as a generalized trigonometric function, then this is a closed-form solution; otherwise, it is an integral needing numerical evaluation.
    1. a) Use numerical quadrature to determine the ratio _T_ / _T_ 0 for five values of _θ_ m between 0 and π. Show that you have attained at least four places of accuracy by progressively increasing the number of integration points until changes occur only in the fifth place, or beyond.
    2. b) Use the power series (5.84) to determine the ratio _T_ / _T_ 0. Continue summing terms until changes in the sum occur only in the fifth place, or beyond.
    3. c) Plot the values you obtain for _T_ / _T_ 0 vs. _θ_ m for both the integral and power series solution. Note that any departure from 1 indicates breakdown of the familiar small-angle approximation for the pendulum.
  5. In the classic E&M text (Jackson, 1988), there is the problem of an infinite, grounded, thin, plane sheet of conducting material with a hole of radius _a_ cut in it. The hole contains a conducting disk of slightly smaller radius kept at potential _V_ and separated from the sheet by a thin ring of insulating material. solves for the potential a perpendicular distance _z_ above the _edge_ of the disk in terms of an elliptic integral:

(5.85)

where  . Use numerical integration to calculate and then plot the potential for   and values of _z_ in the interval (0.05, 10). Compare to a 1/ _r_ fall off.
  6. Figure 5.7 shows a current loop of radius _a_ carrying a current _I._ The point _P_ is a distance _r_ from the center of the loop with spherical coordinates ( _r, θ, φ_ ). Jackson (1988) solves for the _φ_ component of the vector potential at point _P_ in terms of elliptic integrals:

(5.86)

(5.87)

(5.88)

Here _K_ ( _k_ ) is a complete elliptic integral of the first kind and _E_ ( _k_ ) is a complete elliptic integral of the second kind. For  , and  , compute and plot
    1. a)
    2. b)

## 5.17 Multidimensional Monte Carlo Integration (Problem)

Let us say that we want to calculate some properties of a small atom such as magnesium with 12 electrons. To do that, we need to integrate atomic wave functions over the three coordinates of each of 12 electrons. This amounts to a   integral. If we use 64 points for each integration, this requires about   evaluations of the integrand. If the computer were fast and could evaluate the integrand a million times per second, this would take about 1059 s, which is significantly longer than the age of the universe  .

**Figure 5.7** A ring of radius _a_ carries a current _l_. Find vector potential at point P.

Your _problem_ is to find a way to perform multidimensional integrations so that you are still alive to savor the results. Specifically, evaluate the 10D integral

(5.89)

Check your numerical answer against the analytic one, 155/6.

It is easy to generalize mean value integration to many dimensions by picking random points in a multidimensional space. For example, in 2D:

(5.90)

### 5.17.1 Multi Dimension Integration Error Assessment

When we perform a multidimensional integration, the relative error in the Monte Carlo technique, being statistical, decreases as  . This is valid even if the _N_ points are distributed over _D_ dimensions. In contrast, when we use these same _N_ points to perform a _D_ -dimensional integration as _D_ separate 1D integrals using a rule such as Simpson's, we use _N/D_ points for each integration. For fixed _N,_ this means that the number of points used for each integration decreases as the number of dimensions _D_ increases, and so the error in each integration _increases_ with _D._ Furthermore, the total error will be approximately _N_ times the error in each integral. If you put these trends together and do the analysis for a particular integration rule, you will find that at a value of   the error in Monte Carlo integration is approximately equal to that of conventional schemes. For larger values of _D,_ the Monte Carlo method is always more accurate!

### 5.17.2 Implementation: 10D Monte Carlo Integration

Use a built-in random-number generator to perform the 10D Monte Carlo integration in (5.89).

  1. Conduct 16 trials and take the average as your answer.
  2. Try sample sizes of  .
  3. Plot the relative error vs.   and see if linear behavior occurs.
  4. What is your estimate for the accuracy of the integration?
  5. Show that for a dimension  , the error in multidimensional Monte Carlo integration is approximately equal to that of conventional schemes, and that for larger values of _D,_ the Monte Carlo method is more accurate.

## 5.18 Integrating Rapidly Varying Functions (Problem)

It is common in many physical applications to integrate a function with an approximately Gaussian dependence on _x._ The rapid falloff of the integrand means that our Monte Carlo integration technique would require an incredibly large number of points to have sufficient points where the integrand is large. Your _problem_ is to make Monte Carlo integration more efficient for rapidly varying integrands.

## 5.19 Variance Reduction (Method)

If the function being integrated never differs much from its average value, then the standard Monte Carlo mean value method (5.79) should work well with a large, but manageable, number of points. Yet for a function with a large _variance_ (i.e., one that is not "flat"), many of the evaluations of the function may occur for _x_ values at which the function is very small, and thus makes an insignificant contribution to the integral; this is, basically, a waste of time. The method can be improved by mapping the function _f_ into a different function _g_ that has a smaller variance over the interval. We indicate two methods here and refer you to Press _et al._ (1994) and Koonin (1986) for more details.

The first method is a _variance reduction_ or _subtraction technique_ in which we devise a flatter function on which to apply the Monte Carlo technique. Suppose we construct a function _g_ ( _x_ ) with the following properties on [ _a, b_ ]:

(5.91)

We now evaluate the integral of the difference   and add the result to _J_ to obtain the required integral

(5.92)

If we are clever enough to find a simple _g_ ( _x_ ) that makes the variance of   less than that of _f_ ( _x_ ), and that we can integrate analytically, we can obtain even more accurate answers in less time.

## 5.20 Importance Sampling (Method)

A second method for improving Monte Carlo integration is called _importance sampling_ because samples the integrand in the most important regions. It derives from the identity

(5.93)

If we now use a _probability distribution_ for our random numbers that includes _x_ ( _x_ ), the integral can be approximated as

(5.94)

The improvement arising from (5.94) is that with a judicious choice of weighting function  , we can make _f_ ( _x_ )/ _w_ ( _x_ ) more constant and thus easier to integrate accurately.

## 5.21 von Neumann Rejection (Method)

A simple and ingenious method for generating random points with a probability distribution _w_ ( _x_ ) was deduced by von Neumann and is implemented in Listing 5.3. This method is essentially the same as the rejection or sampling method used to guess the area of a pond, only now the pond has been replaced by the weighting function _w_ ( _x_ ), and the arbitrary box around the lake by the arbitrary constant _W_ 0. Imagine a graph of _w_ ( _x_ ) vs. _x_ (Figure 5.8). Walk off your box by placing the line   on the graph, with the only condition being   We next "throw stones" at this graph and count only those splashes that fall into the _w_ ( _x_ ) pond. That is, we generate uniform distributions in _x_ and   with the maximum _y_ value equal to the width of the box  :

**Figure 5.8** The von Neumann rejection technique for generating random points with weight _W_ ( _x_ ). A random point is accepted if it lies below the curve of _W_ ( _x_ ) and rejected if it lies above. This generates a random distribution weighted by whatever _W_ ( _x_ ) function is plotted.

(5.95)

We then reject all _x_ _i_ that do not fall into the pond:

(5.96)

The _x_ _i_ values so accepted will have the weighting _w_ ( _x_ ) (Figure 5.8). The largest acceptance occurs where _w_ ( _x_ ) is large, in this case for midrange _x._ In Chapter 17, we apply a variation of the rejection technique known as the _Metropolis algorithm._ This algorithm has now become the cornerstone of computation thermodynamics.

**Listing 5.3** `**vonNeuman.py**` uses rejection to generate a weighted random distribution.

### 5.21.1 Simple Random Gaussian Distribution

The central limit theorem can be used to deduce a Gaussian distribution via a simple summation. The theorem states, under rather general conditions, that if { _r i_} is a sequence of mutually independent random numbers, then the sum

(5.97)

is distributed normally. This means that the generated _x_ values have the distribution

(5.98)

## 5.22 Nonuniform Assessment

Use the von Neumann rejection technique to generate a normal distribution of standard deviation 1 and compare it to the simple Gaussian method.

### 5.22.1 Implementation

In order for _w_ ( _x_ ) to be the weighting function for random numbers over [ _a, b_ ], we want it to have the properties

(5.99)

where   is the probability of obtaining an _x_ in the range  . For the uniform distribution over  .

**Inverse transform/change of variable method ** Let us consider a change of variables that takes our original integral _I_ (5.93) to the form

(5.100)

Our aim is to make this transformation such that there are equal contributions from all parts of the range in _W_ ; that is, we want to use a uniform sequence of random numbers for _W_. To determine the new variable, we start with _u_ ( _r_ ), the uniform distribution over [0,1],

(5.101)

We want to find a mapping _r_ ↔ _x_ or probability function _w_ ( _x_ ) for which probability is conserved:

(5.102)

This means that even though _x_ and _r_ are related by some (possibly) complicated mapping, _x_ is also random with the probability of _x_ lying in   equal to that of r lying in  .

To find the mapping between _x_ and _r_ (the tricky part), we change variables to _W_ ( _x_ ) defined by the integral

(5.103)

We recognize _W_ ( _x_ ) as the (incomplete) integral of the probability density _u_ ( _r_ ) up to some point _x._ It is another type of distribution function, the integrated probability of finding a random number less than the value _x._ The function _W_ ( _x_ ) is on that account called a _cumulative distribution function_ and can also be thought of as the area to the left of _r_ = _x_ on the plot of _u_ ( _r_ ) vs. _r_. It follows immediately from the definition (5.103) that _W_ ( _x_ ) has the properties

(5.104)

(5.105)

Consequently,   is a uniform sequence of random numbers, and we just need to invert (5.103) to obtain _x_ values distributed with probability _w_ ( _x_ ).

The crux of this technique is being able to invert (5.103) to obtain  . Let us look at some analytic examples to get a feel for these steps (numerical inversion is possible and frequent in realistic cases).

**Uniform weight function _w_** We start with the familiar uniform distribution

(5.106)

After following the rules, this leads to

(5.107)

(5.108)

where _W_ ( _x_ ) is always taken as uniform. In this way, we generate uniform random   and uniform random  .

**Exponential weight** We want random points with an exponential distribution:

(5.109)

(5.110)

In this way, we generate uniform random _r_ : [0,1] and obtain _x_ = −γ ln(1 − _r_ ) distributed with an exponential probability distribution for _x_ > 0. Notice that our prescription (5.93) and (5.94) tells us to use _w_ ( _x_ ) = e−x/λ/ _λ_ to remove the exponential-like behavior from an integrand and place it in the weights and scaled points   Because the resulting integrand will vary less, it may be approximated better as a polynomial:

(5.111)

**Gaussian (normal) distribution** We want to generate points with a normal distribution:

(5.112)

This by itself is rather hard but is made easier by generating uniform distributions in angles and then using trigonometric relations to convert them to a Gaussian distribution. But before doing that, we keep things simple by realizing that we can obtain (5.112) with mean   and standard deviation _σ_ by scaling and a translation of a simpler _w_ ( _x_ ):

(5.113)

We start by generalizing the statement of probability conservation for two different distributions (5.102) to two dimensions (Press _et al.,_ 1994):

(5.114)

(5.115)

We recognize the term in vertical bars as the Jacobian determinant:

(5.116)

To specialize to a Gaussian distribution, we consider 2π _r_ as angles obtained from a uniform random distribution _r_ , and _x_ and _y_ as Cartesian coordinates that will have a Gaussian distribution. The two are related by

(5.117)

The inversion of this mapping produces the Gaussian distribution

(5.118)

The solution to our problem is at hand. We use (5.117) with _r_ 1 and _r_ 2 uniform random distributions, and _x_ and _y_ are then Gaussian random distributions centered around _x_ = 0.

1) In Chapter 26, we show how to remove such a singularity even when the integrand is unknown. 2) There is, not coincidentally, a Milne Computer Center at Oregon State University, although there is no longer a central computer there.

# 6
Matrix Computing

This chapter examines various aspects of computing with matrices, and in particular applications of the Python packages. Because these packages are optimized and robust, we strongly recommend their use even in all your small programs (small programs often grow big). The two-mass-on-a-string problem is formulated as a matrix problem, and extends the Newton–Raphson search technique to be discussed in Chapter 7. Although there is some logic in having this chapter after Chapter 7, it is placed here to provide the matrix tools needed for visualizations (which is why Problem 3 preceds 1).

## 6.1 Problem 3: N–D Newton–Raphson; Two Masses on a String

**Problem** Two weights ( _W_ 1, _W_ 2) = (10, 20) are hung from three pieces of string with lengths ( _L_ 1, _L_ 2, _L_ 3) = (3, 4, 4) and a horizontal bar of length _L_ = 8 (Figure 6.1). Find the angles assumed by the strings and the tensions exerted by the strings.

In spite of the fact that this is a simple problem requiring no more than first-year physics to formulate, the coupled transcendental equations that result are just about inhumanely painful to solve analytically.1) However, we will show you how the computer can solve this problem, but even then only by a trial-and-error technique with no guarantee of success. Your _problem_ is to test this solution for a variety of weights and lengths and then to extend it to the three-weight problem (not as easy as it may seem). In either case check the physical reasonableness of your solution; the deduced tensions should be positive and similar in magnitude to the weights of the spheres, and the deduced angles should correspond to a physically realizable geometry, as confirmed with a sketch such as Figure 6.2. Some of the exploration you should do is to see at what point your initial guess gets so bad that the computer is unable to find a physical solution.

**Figure 6.1** Two weights connected by three pieces of string and suspended from a horizontal bar of length _L_. The lengths are all known, but the angles and the tensions in the strings are to be determined.

**Figure 6.2** A free body diagram for one weight in equilibrium. Balancing the forces in the _x_ and _y_ directions for all weights leads to the equations of static equilibrium.

### 6.1.1 Theory: Statics

We start with the geometric constraints that the horizontal length of the structure is _L_ and that the strings begin and end at the same height (Figure 6.1):

(6.1)

(6.2)

(6.3)

(6.4)

(6.5)

Observe that the last three equations include trigonometric identities as independent equations because we are treating sin _θ_ and cos _θ_ as independent variables; this makes the search procedure easier to implement. The basics physics says that because there are no accelerations, the sum of the forces in the horizontal and vertical directions must equal zero (Figure 6.2):

(6.6)

(6.7)

(6.8)

(6.9)

Here _W_ _i_ is the weight of mass _i_ and _T_ _i_ is the tension in string _i_. Note that because we do not have a rigid structure, we cannot assume the equilibrium of torques.

### 6.1.2 Algorithm: Multidimensional Searching

Equations (6.1)–(6.9) are nine simultaneous nonlinear equations. While linear equations can be solved directly, nonlinear equations cannot (Press _et al_., 1994). You can use the computer to _search_ for a solution by guessing, but there is no guarantee of finding one.

Unfortunately, not everything in life is logical, as we need to use a search technique now that will not be covered until Chapter 7. While what we do next is self-explanatory, you may want to look at Chapter 7 now if you are not at all familiar with searching.

We apply to our set the same Newton–Raphson algorithm as used to solve a single equation by renaming the nine unknown angles and tensions as the subscripted variable _y_ _i_ and placing the variables together as a vector:

(6.10)

The nine equations to be solved are written in a general form with zeros on the right-hand sides and placed in a vector:

(6.11)

(6.12)

The solution to these equations requires a set of nine _x i_ values that make all nine _f_ _i_ 's vanish simultaneously. Although these equations are not very complicated (the physics after all is elementary), the terms quadratic in _x_ make them nonlinear, and this makes it hard or impossible to find an analytic solution. The search algorithm guesses a solution, expands the nonlinear equations into linear form, solves the resulting linear equations, and continues to improve the guesses based on how close the previous one was to making _**f**_ = 0. (We discuss search algorithms using this procedure in Chapter 7.)

Explicitly, let the approximate solution at any one stage be the set _x i_ and let us assume that there is an (unknown) set of corrections Δ _x_ _i_ for which

(6.13)

We solve for the approximate Δ _x_ _i_ 's by assuming that our previous solution is close enough to the actual one for two terms in the Taylor series to be accurate:

(6.14)

We now have a solvable set of nine linear equations in the nine unknowns Δ _x i_, which we express as a single matrix equation

(6.15)

Note now that the derivatives and the _f_ 's are all evaluated at known values of the _x_ _i_ 's, so that only the vector of the Δ _x_ _i_ values is unknown. We write this equation in matrix notation as

(6.16)

Here we use bold italic letters to emphasize the vector nature of the columns of _f_ i and Δ _x_ _i_ values, and denote the matrix of the derivatives _F_ ′ (it is also sometimes denoted by _J_ because it is the _Jacobian_ matrix).

The equation _F_ ′ **Δ _x_ = − _f_** is in the standard form for the solution of a linear equation (often written as _A **x**_ = _**b**_ ), where **Δ _x_** is the vector of unknowns and _**b**_ = − _**f**_. Matrix equations are solved using the techniques of linear algebra, and in the sections to follow we shall show how to do that. In a formal sense, the solution of (6.16) is obtained by multiplying both sides of the equation by the inverse of the _F_ ′ matrix:

(6.17)

where the inverse must exist if there is to be a unique solution. Although we are dealing with matrices now, this solution is identical in form to that of the 1D problem, Δ _x_ = −(1/ _f_ ′) _f_. In fact, one of the reasons, we use formal or abstract notation for matrices is to reveal the simplicity that lies within.

As we indicate for the single-equation Newton–Raphson method in Section 7.3, even in a case such as this where we can deduce analytic expressions for the derivatives ∂ _f i_/∂ _x_ _j_ , there are 9 × 9 = 81 such derivatives for this (small) problem, and entering them all would be both time-consuming and error-prone. In contrast, especially for more complicated problems, it is straightforward to program a forward-difference approximation for the derivatives,

(6.18)

where each individual _x_ _j_ is varied independently because these are partial derivatives and _δx_ _j_ , are some arbitrary changes you input. While a central-difference approximation for the derivative would be more accurate, it would also require more evaluations of the _f_ 's, and once we find a solution it does not matter how accurate our algorithm for the derivative was.

As also discussed for the 1D Newton–Raphson method (Section 7.3.1), the method can fail if the initial guess is not close enough to the zero of _f_ (here all _N_ of them) for the _f_ 's to be approximated as linear. The _backtracking_ technique may be applied here as well, in the present case, progressively decreasing the corrections Δ _x_ _i_ until   decreases.

## 6.2 Why Matrix Computing?

Physical systems are often modeled by systems of simultaneous equations written in matrix form. As the models are made more realistic, the matrices correspondingly become larger, and it becomes more important to use a good linear algebra library. Computers are unusually good with matrix manipulations because those manipulations typically involve the continued repetition of a small number of simple instructions, and algorithms exist to do this quite efficiently. Further speedup may be achieved by _tuning_ the codes to the computer's architecture, as discussed in Chapter 11.

Industrial-strength subroutines for matrix computing are found in well-established scientific libraries. These subroutines are usually an order of magnitude or more faster than the elementary methods found in linear algebra texts,2) are usually designed to minimize the round-off error, and are often "robust," that is, have a high chance of being successful for a broad class of problems. For these reasons, we recommend that you _do not write your own matrix methods_ but instead get them from a library. An additional value of library routines is that you can often run the same program either on a desktop machine or on a parallel supercomputer, with matrix routines automatically adapting to the local architecture. The thoughtful reader may be wondering when a matrix is "large" enough to require the use of a library routine. One rule of thumb is "if you have to wait for the answer," and another rule is if the matrices you are using take up a good fraction of your computer's random-access memory (RAM).

## 6.3 Classes of Matrix Problems (Math)

It helps to remember that the rules of mathematics apply even to the world's most powerful computers. For example, you _should_ encounter problems solving equations if you have more unknowns than equations, or if your equations are not linearly independent. But do not fret. While you cannot obtain a unique solution when there are not enough equations, you may still be able to map out a space of allowable solutions. At the other extreme, if you have more equations than unknowns, you have an _overdetermined_ problem, which may not have a unique solution. An overdetermined problem is sometimes treated using data fitting techniques in which a solution to a sufficient set of equations is found, tested on the unused equations, and then improved if needed. Not surprisingly, this technique is known as the _linear least-squares method_ (as in Chapter 7) because the technique minimizes the disagreement with the equations. The most basic matrix problem is a system of linear equations:

(6.19)

where **A** is a known _N_ × _N_ matrix, _**x**_ is an unknown vector of length _N_ , and _**b**_ is a known vector of length _N_. The obvious way to solve this equation is to determine the inverse of **A** and then form the solution by multiplying both sides of (6.19) by **A** −1:

(6.20)

Both the direct solution of (6.19) and the determination of a matrix's inverse are standards in a matrix subroutine library. A more efficient way to solve (6.19) is by Gaussian elimination or lower–upper (LU) decomposition. This yields the vector _**x**_ without explicitly calculating **A** −1. However, sometime you may want the inverse for other purposes, in which case (6.20) is preferred.

If you have to solve the matrix equation

(6.21)

with _**x**_ an unknown vector and _λ_ an unknown parameter, then the direct solution (6.20) will not be of much help because the matrix _b_ = _λx_ contains the unknowns _λ_ and _x_. Equation 6.21 is the _eigenvalue problem_. It is harder to solve than (6.19) because solutions exist for only certain, if any, values of λ. To find a solution, we use the identity matrix to rewrite (6.21) as

(6.22)

We see that multiplication of (6.22) by [ **A** − _λI_ ]−1 yields the _trivial solution_

(6.23)

While the trivial solution is a bona fide solution, it is nonetheless trivial. A more interesting solution requires the existence of a condition that forbids us from multiplying both sides of (6.22) by [ **A** − _λI_ ]−1. That condition is the nonexistence of the inverse, and if you recall that Cramer's rule for the inverse requires division by det[ **A** − _λI_ ], it is clear that the inverse fails to exist (and in this way eigenvalues _do_ exist) when

(6.24)

The _λ_ values that satisfy this _secular equation_ are the eigenvalues of (6.21). If you are interested in only the eigenvalues for (6.21), you should look for a matrix routine that solves (6.24). To do that, you need a subroutine to calculate the determinant of a matrix, and then a search routine to find the zero of (6.24). Such routines are available in libraries.

The traditional way to solve the eigenvalue problem (6.21) for both eigenvalues and eigenvectors is by _diagonalization_. This is equivalent to successive changes of basis vectors, each change leaving the eigenvalues unchanged while continually decreasing the values of the off-diagonal elements of _**A**_. The sequence of transformations is equivalent to continually operating on the original equation with the transformation matrix **U** until one is found for which **UAU** −1 is diagonal:

(6.25)

(6.26)

(6.27)

The diagonal values of **_UAU_** −1 are the eigenvalues with eigenvectors

(6.28)

that is, the eigenvectors are the columns of the matrix **U** −1. A number of routines of this type are found in subroutine libraries.

### 6.3.1 Practical Matrix Computing

Many scientific programming bugs arise from the improper use of arrays.3) This may be as a result of the extensive use of matrices in scientific computing or to the complexity of keeping track of indices and dimensions. In any case, here are some rules of thumb to observe.

**Computers are finite** Unless you are careful, your matrices may so much memory that your computation will slow down significantly, especially if it starts to use virtual memory. As a case in point, let us say that you store data in a 4D array with each index having a _physical dimension_ of 100: `**A[100] [100] [100] [100]**`. This array of (100)4 64-byte words occupies  :1 GB of memory.

**Processing time** Matrix operations such as inversion require on the order of _N_ 3 steps for a square matrix of dimension _N_. Therefore, doubling the dimensions of a 2D square matrix (as happens when the number of integration steps is doubled) leads to an _eightfold_ increase in the processing time.

**Paging** Many operating systems have _virtual memory_ in which disk space is used when a program runs out of RAM (see Chapter 10) for a discussion of how computers arrange memory). This is a slow process that requires writing a full _page_ of words to the disk. If your program is near the memory limit at which paging occurs, even a slight increase in a matrix's dimension may lead to an order of magnitude increase in the execution time.

**Figure 6.3** (a) Row-major order used for matrix storage in Python, C and Java. (b) Column-major order used for matrix storage in Fortran. The table at the bottom shows how successive matrix elements are actually stored in a linear fashion in memory.

**Matrix storage** While we think of matrices as multidimensional blocks of stored numbers, the computer stores them as linear strings. For instance, a matrix `**a[3,3]**` in Python, is stored in _row-major order_ (Figure 6.3a):

(6.29)

while in Fortran, starting subscripts at 0, it is stored in _column-major order_ (Figure 6.3b):

(6.30)

It is important to keep this linear storage scheme in mind in order to write proper code and to permit the mixing of Python and Fortran programs.

When dealing with matrices, you have to balance the clarity of the operations being performed against the efficiency with which the computer performs them. For example, having one matrix with many indices such as `**V[L,Nre,Nspin,k,kp,Z,A]**` may be neat packaging, but it may require the computer to jump through large blocks of memory to get to the particular values needed (large _strides_ ) as you vary `**k, kp, and Nre**`. The solution would be to have several matrices such as `**V1[Nre,Nspin,k,kp,Z,A]**`, `**V2[Nre,Nspin,k,kp,Z,A]**`, and `**V3[Nre,Nspin,k,kp,Z,A]**`.

**Subscript 0** It is standard in Python, C, and Java to have array indices begin with the value 0. While this is now permitted in Fortran, the standard in Fortran and in most mathematical equations has been to start indices at 1. On that account, in addition to the different locations in memory as a result of row-major and column-major ordering, the same matrix element may be referenced differently in the different languages:

**Location** | **Python/C element** | **Fortran element**
---|---|---
Lowest | `**a [0,0]**` | `**a (1,1)**`
| `**a [0,1]**` | `**a (2,1)**`
| `**a [1,0]**` | `**a (3,1)**`
| `**a [1,1]**` | `**a (1,2)**`
| `**a [2,0]**` | `**a (2,2)**`
Highest | `**a [2,1]**` | `**a (3,2)**`

**Tests** _Always_ test a library routine on a small problem whose answer you know (such as the exercises in Section 6.6). Then you will know if you are supplying it with the right arguments and if you have all the links working.

## 6.4 Python Lists as Arrays

A _list_ is Python's built-in sequence of numbers or objects. Although called a "list" it is similar to what other computer languages call an "array." It may be easier for you to think of a Python list as a container that holds a bunch of items in a definite order. (Soon we will describe the higher level `**array**` data types available with the _NumPy_ package.) In this section we review some of Python's native _list_ features. Python interprets a sequence of ordered items, _L_ = _l_ 0, _l_ 1,... , _l_ _N_ −1, as a _list_ and represents it with a single symbol `**L**`:

We observe that square brackets with comma separators [1, 2, 3] are used for lists, and that a square bracket is also used to indicate the index for a list item, as in line 2 `**(L[0])**`. Lists contain sequences of arbitrary objects that are _mutable_ or changeable. As we see in line 7 in the `**L**` command, an entire list can be referenced as a single object, in this case to obtain its printout.

Python also has a built-in type of list known as a _tuple_ whose elements are not mutable. Tuples are indicated by round parenthesis (.., .., .), with individual elements still referenced by square brackets:

Note that the error message arises when we try to change an element of a tuple.

Most languages require you to specify the size of an array before you can start storing objects in it. In contrast, Python lists are _dynamic_ , which means that their sizes adjust as needed. In addition, while a list is essentially one dimensional because it is a sequence, a compound list can be created in Python by having the individual elements themselves as lists:

Python can perform a large number of operations on lists, for example,

**Operation** | **Effect** | **Operation** | **Effect**
---|---|---|---
`**L = [1, 2, 3, 4]**` | Form list | `**L1 + L2**` | Concatenate lists
`**L[i]**` | _i_ th element | `**len(L)**` | Length of list _L_
`**i in L**` | True if _i_ in _L_ | `**L[i:j]**` | Slice from _i_ to _j_
`**for i in L**` | Iteration index | `**L.append(x)**` | Append _x_ to end of _L_
`**L.count(x)**` | Number of _x_ 's in _L_ | `**L.index(x)**` | Location of 1st _x_ in _L_
`**L.remove(x)**` | Remove 1st _x_ in _L_ | `**L.reverse()**` | Reverse elements in _L_
`**L.sort()**` | Order elements in _L_ |  |

## 6.5 Numerical Python (NumPy) Arrays

Although basic Python does have an array data type, it is rather limited and we suggest using NumPy `**arrays**`, which will convert Python lists into arrays. Because _we often useNumPy's array command_ to form computer representations of vectors and matrices, you will have to import _NumPy_ in your programs (or Visual, which includes NumPy) to use those examples. For instance, here we show the results of running our program Matrix.py from a shell:

We see here that we have initialized an array object, added two 1D array objects together, and printed out the result. Likewise we see that multiplying an array by a constant does in fact multiply each element by that constant (line 8). We then construct a "matrix" as a 1D array of two 1D arrays, and when we print it out, we note that it does indeed look like a matrix. However, when we multiply this matrix by itself, the result is not   that one normally expects from matrix multiplication. So if you need actual mathematical matrices, then you need to use NumPy!

Now we give some examples of the use of NumPy, but do refer the reader to the NumPy Tutorial (NumPyTut, 2012) and to the articles in _Computing in Science and Engineering_ (Perez _et al_., 2010) for more information. To start, we note that a NumPy array can hold up to 32 dimensions (32 indices), but each element must be of the same type (a _uniform_ array). The elements are not restricted to just floatingpoint numbers or integers, but can be any object, as long as all elements are of this same type. (Compound objects may be useful, for example, for storing parts of data sets.) There are various ways to create arrays, with square brackets [...] used for indexing in all cases. First we start with a Python list (tuples work as well) and create an array from it:

Notice that it is essential to have the square brackets within the round brackets because the square brackets produce the list object while the round brackets indicate a function argument. Note also that because the data in our original list were all integers, the created array is a 32-bit integer data types, which we can check by affixing the `**dtype**` method:

If we had started with floating-point numbers, or a mix of floats and ints, we would have ended up with floating-point arrays:

When describing NumPy arrays, the number of "dimensions," `**ndim**`, means the number of indices, which, as we said, can be as high as 32. What might be called the "size" or "dimensions" of a matrix in mathematics is called the _shape_ of a NumPy array. Furthermore, NumPy does have a `**size**` method that returns the total number of elements. Because Python's lists and tuples are all one dimensional, if we want an array of a particular shape, we can attain that by affixing the `**reshape**` method when we create the array. Where Python has a `**range**` function to generate a sequence of numbers, NumPy has an `**arange**` function that creates an array, rather than a list. Here we use it and then reshape the 1D array into a 3 × 4 array:

Note that here we imported NumPy as the object np, and then affixed the `**arange**` and `**reshape**` methods to this object. We then checked the shape of `**a**`, and found it to have three rows and four columns of long integers (Python 3 may just say `**ints**`). Note too, as we see on line 9, NumPy uses parentheses () to indicate the shape of an array, and so `**(3L,4L)**` indicates an array with three rows and four columns of long ints.

Now that we have shapes in our minds, we should note that NumPy offers a number of ways to change shapes. For example, we can transpose an array with the `**.T**` method, or `**reshape**` into a vector:

And again, `**(1,12)**` indicates an array with 1 row and 12 columns. Yet another handy way to take a matrix and extract just what you want from it is to use Python's _slice_ operator `**start:stop:step:**` to take a slice out of an array:

We note here that Python indices start counting from 0, and so 1:3 means indices 0, 1, 2 (without 3). As we discuss in Chapter 11, slicing can be very useful in speeding up programs by picking out and placing in memory just the specific data elements from a large data set that need to be processed. This avoids the time-consuming jumping through large segments of memory as well as excessive reading from disk.

Finally, we remind you that while all elements in a NumPy array must be of the same data type, that data type can be compound, for example, an array of arrays:

Furthermore, an array can be composed of complex numbers by specifying the complex data type as an option on the `**array**` command. NumPy then uses the _j_ symbol for the imaginary number _i_ :

In the next section, we discuss using true mathematical matrices with NumPy, which is one use of an array object. Here we note that if you wanted the familiar matrix product from two arrays, you would use the dot function, whereas ***** is used for an element-by-element product:

NumPy is actually optimized to work well with arrays, and in part this is because arrays are handled and processed much as if they were simple, scalar variables.4) For example, here is another example of _slicing_ , a technique that is also used in ordinary Python with lists and tuples, in which two indices separated by a colon indicate a range:

Here we start by creating the NumPy array `**stuff**` of floats, all of whose 10 elements are initialized to zero. Then we create the array `**t**` containing the four elements [0, 1, 2, 3] by assigning four variables uniformly in the range 0–4 (the "a" in `**arange**` creates floating-point variables, `**range**` creates integers). Next we use a slice to assign `**[sqrt(0+1), sqrt(1+1), sqrt(2+1), sqrt(3+1)] = [1, 1.414, 1.732, 2]**` to the middle elements of the `**stuff**` array. Note that the NumPy version of the `**sqrt**` function, one of many universal functions _(ufunctions)_ supported by NumPy5), has the amazing property of automatically outputting an array whose length is that of its argument, in this case, the array `**t**`. In general, major power in NumPy comes from its _broadcasting_ operation, an operation in which values are assigned to multiple elements via a single assignment statement. Broadcasting permits Python to _vectorize_ array operations, which means that the same operation can be performed on different array elements in parallel (or nearly so). Broadcasting also speeds up processing because array operations occur in C instead of Python, and with a minimum of array copies being made. Here is a simple sample of broadcasting:

The first line creates the NumPy array `**w**`, and the second line "broadcasts" the value 23.7 to all elements in the array. There are many possible array operations in NumPy and various rules pertaining to them; we recommend that the serious user explore the extensive NumPy documentation for additional information.

### 6.5.1 NumPy's linalg Package

The array objects of NumPy and Visual are not the same as mathematical matrices, although an array can be used to represent a matrix. Fortunately, there is the `**LinearAlgebra**` package that treats 2D arrays (a 1D array of 1D arrays) as mathematical matrices, and also provides a simple interface to the powerful LAPACK linear algebra library (Anderson _et al_., 2013). As we keep saying, there is much to be gained in speed and reliability from using these libraries rather than writing your own matrix routines.

Our first example from linear algebra is the standard matrix equation

(6.31)

where we have used an uppercase bold character to represent a matrix and a lowercase bold italic character to represent a 1D matrix (a vector). Equation 6.31 describes a set of linear equations with _**x**_ an unknown vector and **A** a known matrix. Now we take _y y **A**_ to be 3 × 3, _**b**_ to be 3 × 1, and let the program figure out that _**x**_ must be a 3 × 1 vector.6) We start by importing all the packages, by inputting a matrix and a vector, and then by printing out **A** and _**x**_ :

Because we have the matrices **A** and _**b**_ , we can go ahead and solve **A _x_** = _**b**_ using NumPy's `**solve**` command, and then test how close **A _x_** − _**b**_ is to a zero vector:

This is really quite impressive. We have solved the entire set of linear equations (by elimination) with just the single command `**solve**`, performed a matrix multiplication with the single command `**dot**`, did a matrix subtraction with the usual operator, and are left with a residual essentially equal to machine precision.

Although there are more efficient numerical approaches, a direct way to solve

(6.32)

is to calculate the inverse **A** −1, and then multiply both sides of the equation by the inverse, yielding

(6.33)

Here we first tested that `**inv(A)**` is in fact the inverse of `**A**` by seeing if `**A**` times `**inv(A)**` equals the identity matrix. Then we used the inverse to solve the matrix equation directly, and got the same answer as before (some error at the level of machine precision is just fine).

Our second example occurs in the solution for the principal-axes system of a cube, and requires us to find a coordinate system in which the inertia tensor is diagonal. This entails solving the eigenvalue problem

(6.34)

where **I** is the inertia matrix (tensor), _**ω**_ is an unknown eigenvector, and _λ_ is an unknown eigenvalue. The program `**Eigen.py**` solves for the eigenvalues and vectors, and shows how easy it is to deal with matrices. Here it is in an abbreviated interpretive mode:

**Table 6.1** The operators of NumPy and their effects.

Operator| Effect| Operator| Effect
---|---|---|---
`**dot(a, b[,out])**`|  Dot product arrays| ` **vdot(a, b)**`|  Dot product
`**inner(a, b)**`|  Inner product arrays| ` **outer(a, b)**`|  Outer product
`**tensordot(a, b)**`|  Tensor dot product| ` **einsum()**`|  Einstein sum
`**linalg.matrix_power(M, n)**`|  Matrix to power n| ` **kron(a, b)**`|  Kronecker product
`**linalg.cholesky(a)**`|  Cholesky decomp| ` **linalg.qr(a)**`|  QR factorization
`**linalg.svd(a)**`|  Singular val decomp| ` **linalg.eig(a)**`|  Eigenproblem
`**linalg.eigh(a)**`|  Hermitian eigen| ` **linalg.eigvals(a)**`|  General eigen
`**linalg.eigvalsh(a)**`|  Hermitian eigenvals| ` **linalg.norm(x)**`|  Matrix norm
`**linalg.cond(x)**`|  Condition number| ` **linalg.det(a)**`|  Determinant
`**linalg.slogdet(a)**`|  Sign and log(det)| ` **trace(a)**`|  Diagnol sum
`**linalg.solve(a, b)**`|  Solve equation| ` **linalg.tensorsolve(a, b)**`|  Solve _ax = b_
`**linalg.lstsq(a, b)**`|  Least-squares solve| ` **linalg.inv(a)**`|  Inverse
`**linalg.pinv(a)**`|  Penrose inverse| ` **linalg.tensorinv(a)**`|  Inverse N–D array

We see here how, after we set up the array `**I**` on line 3, we then solve for its eigenvalues and eigenvectors with the single statement `**Es**`, `**evectors = eig(I)**` on line 8. We then extract the first eigenvector on line 14 and use it, along with the first eigenvalue, to check that (6.34) is in fact satisfied to machine precision.

Well, we think by now you have some idea of the use of NumPy. In Table 6.1 we indicate some more of what is available.

## 6.6 Exercise: Testing Matrix Programs

Before you direct the computer to go off crunching numbers on a million elements of some matrix, it is a good idea to try out your procedures on a small matrix, especially one for which you know the right answer. In this way, it will take you only a short time to realize how hard it is to get the calling procedure perfect. Here are some exercises.

  1. Find the numerical inverse of  .
    1. a) As a general check, applicable even if you do not know the analytic answer, check your inverse in both directions; that is, check that **AA** −1 = **A** −1 **A** = _I_ , and note the number of decimal places to which this is true. This also gives you some idea of the precision of your calculation.
    2. b) Determine the number of decimal places of agreement there is between your numerical inverse and the analytic result:
  Is this similar to the error in **AA** −1?
  2. Consider the same matrix **A** as before, here being used to describe three simultaneous linear equations, **A _x_** = _b_ , or explicitly,

(6.35)

Now the vector _**b**_ on the RHS is assumed known, and the problem is to solve for the vector _**x**_. Use an appropriate subroutine to solve these equations for the three different **_x_** vectors appropriate to these three different _**b**_ values on the RHS:

The solutions should be

(6.36)

  3. Consider the matrix  , where you are free to use any values you want for _α_ and _β_. Use a numerical eigenproblem solver to show that the eigenvalues and eigenvectors are the complex conjugates:

(6.37)

  4. Use your eigenproblem solver to find the eigenvalues of the matrix

(6.38)

    1. a) Verify that you obtain the eigenvalues _λ_ 1 = 5, _λ_ 2 = _λ_ 3 = −3. Notice that double roots can cause problems. In particular, there is a uniqueness issue with their eigenvectors because any combination of these eigenvectors is also an eigenvector.
    2. b) Verify that the eigenvector for _λ_ 1 = 5 is proportional to

(6.39)

    3. c) The eigenvalue −3 corresponds to a double root. This means that the corresponding eigenvectors are degenerate, which in turn means that they are not unique. Two linearly independent ones are

(6.40)

In this case, it is not clear what your eigenproblem solver will give for the eigenvectors. Try to find a relationship between your computed eigenvectors with the eigenvalue −3 and these two linearly independent ones.
  5. Imagine that your model of some physical system results in _N_ = 100 coupled linear equations in _N_ unknowns:

In many cases, the _a_ and _b_ values are known, so your exercise is to solve for all the _x_ values, taking _**a**_ as the _Hilbert_ matrix and _**b**_ as its first column:

Compare to the analytic solution

(6.41)

  6. _Dirac Gamma Matrices:_ The Dirac equation extends quantum mechanics to include relativity and spin 1/2. The extension of the Hamiltonian operator for an electron requires it to contain matrices, and those matrices are expressed in terms of 4 × 4 _**γ**_ matrices that can be represented in terms of the familiar 2 × 2 Pauli matrices _**σ**_ i:

(6.42)

(6.43)

Confirm the following properties of the _γ_ matrices:

(6.44)

(6.45)

### 6.6.1 Matrix Solution of the String Problem

In Section 6.1, we set up the solution to our problem of two masses on a string. Now we have the matrix tools needed to solve it. Your _problem_ is to check out the physical reasonableness of the solution for a variety of weights and lengths. You should check that the deduced tensions are positive and the deduced angles correspond to a physical geometry (e.g., with a sketch). Because this is a physics-based problem, we know the sine and cosine functions must be less than 1 in magnitude and that the tensions should be similar in magnitude to the weights of the spheres. Our solution is given in `**NewtonNDanimate.py**` (Listing 6.1), which shows graphically the step-by-step search for a solution.

**Listing 6.1** The code **NewtonNDanimate.py** that shows the step-by-step search for solution of the two-mass-on-a-string problem via a Newton–Raphson search.

### 6.6.2 Explorations

  1. See at what point your initial guess for the angles of the strings gets so bad that the computer is unable to find a physical solution.
  2. A possible problem with the formalism we have just laid out is that by incorporating the identity sin2 _θ_ _i_ \+ cos2 _θ_ _i_ = 1 into the equations, we may be discarding some information about the sign of sin _θ_ or cos _θ_. If you look at Figure 6.1, you can observe that for some values of the weights and lengths, _θ_ 2 may turn out to be negative, yet cos _θ_ should remain positive. We can build this condition into our equations by replacing _f_ 7− _f_ 9 with _f_ 's based on the form

(6.46)

See if this makes any difference in the solutions obtained.
  3.   Solve the similar three-mass problem. The approach is the same, but the number of equations is larger.

1) Almost impossible anyway, as L. Molnar has supplied me with an analytic solution. 2) Although we prize the book, _Numerical Recipes_ by Press _et al_. (1994), and what it has accomplished, we cannot recommend taking subroutines from it. They are neither optimized nor documented for easy, stand-alone use, whereas the subroutine libraries recommended in this chapter are. 3) Even a vector _V_ ( _N_ ) is called an array, albeit a 1D one. 4) We thank Bruce Sherwood for helpful comments on these points. 5) A ufunction is a function that operates on N–D arrays in an element-by-element fashion, supporting array broadcasting, type casting, and several other standard features. In other words, a ufunc is a vectorized wrapper for a function that takes a fixed number of scalar inputs and produces a fixed number of scalar outputs. 6) Do not be bothered by the fact that although we think these vectors as 3 × 1, they sometimes get printed out as 1 × 3; think of all the trees that get saved!

# 7
Trial-and-Error Searching and Data Fitting

* * *

In this chapter, we add more tools to our computational toolbox. First, we devise ways to find solutions to equations by a trial-and-error search, sometimes using our new-found numerical differentiation tools. Although trial-and-error searching may not sound very precise, it is in fact widely used to solve problems where analytic solutions do not exist or are not practical. We have already looked at one such example in Chapter 6, where we saw how the two-weights-on-a-string problem led to matrix equations. In Chapter 8, we combine trial-and-error searching with the solution of ordinary differential equations to solve the general quantum eigenvalue problem. The second part of this chapter introduces some aspects of data fitting. We examine how to interpolate within a table of numbers and how to do a least-squares fit of a function to data, the latter often requiring a search.

* * *

## 7.1 Problem 1: A Search for Quantum States in a Box

Many computer techniques are well-defined sets of procedures leading to definite outcomes. In contrast, some computational techniques are trial-and-error algorithms in which decisions on what path to follow are made based on the current values of variables, and the program quits only when it thinks it has solved the problem. (We already did some of this when we summed a power series until the terms became small.) Writing this type of program is usually interesting because we must foresee how to have the computer act intelligently in all possible situations, and running them is very much like an experiment in which it is hard to predict what the computer will come up with.

**Problem** Probably the most standard problem in quantum mechanics1) is to solve for the energies of a particle of mass _m_ bound within a 1D square well of radius _a_ :

(7.1)

As shown in quantum mechanics texts (Gottfried, 1966), the energies of the bound states   within this well are solutions of the transcendental equations

(7.2)

(7.3)

where even and odd refer to the symmetry of the wave function. Here we have chosen units such that  . Your _problem_ is to

  1. Find several bound-state energies _E_ B for even wave functions, that is, the solution of (7.2).
  2. See if making the potential deeper, say, by changing the 10 to a 20 or a 30, produces a larger number of, or deeper, bound states.

## 7.2 Algorithm: Trial-and-Error Roots via Bisection

Trial-and-error root finding looks for a value of _x_ for which

(7.4)

where the 0 on the right-hand side is conventional (an equation such as   can easily be written as  . The search procedure starts with a guessed value for _x_ , substitutes that guess into _f_ ( _x_ ) (the "trial"), and then sees how far the LHS is from zero (the "error"). The program then changes _x_ based on the error and tries out the new guess in _f_ ( _x_ ). The procedure continues until   to some desired level of precision, or until the changes in _x_ are insignificant, or if the search seems endless.

The most elementary trial-and-error technique is the _bisection algorithm._ It is reliable but slow. If you know some interval in which _f_ ( _x_ ) changes sign, then the bisection algorithm will always converge to the root by finding progressively smaller and smaller intervals within which the zero lies. Other techniques, such as the Newton–Raphson method we describe next, may converge more quickly, but if the initial guess is not close, it may become unstable and fail completely.

**Figure 7.1** A graphical representation of the steps involved in solving for a zero of _f_ ( _x_ ) using the bisection algorithm. The bisection algorithm takes the midpoint of the interval as the new guess for _x_ , and so each step reduces the interval size by one-half. Four steps are shown for the algorithm.

The basis of the bisection algorithm is shown in Figure 7.1. We start with two values of _x_ between which we know that a zero occurs. (You can determine these by making a graph or by stepping through different _x_ values and looking for a sign change.) To be specific, let us say that _f_ ( _x_ ) is negative at _x_ − and positive at _x_ +:

(7.5)

(Note that it may well be that _x_ − > _x_ + if the function changes from positive to negative as _x_ increases.) Thus, we start with the interval _x_ + ≤ _x_ ≤ _x_ − within which we know a zero occurs. The algorithm (implemented as in Listing 7.1) then picks the new _x_ as the bisection of the interval and selects as its new interval the half in which the sign change occurs:

This process continues until the value of _f_ ( _x_ ) is less than a predefined level of precision or until a predefined (large) number of subdivisions occurs.

**Listing 7.1** **The Bisection.py** is a simple implementation of the bisection algorithm for finding a zero of a function, in this case 2 cos _x – x_.

The example in Figure 7.1 shows the first interval extending from _x_ − = _x_ +1 to _x_ + = _x_ −1. We bisect that interval at _x_ , and because _f_ ( _x_ ) < 0 at the midpoint, we set _x_ − ≡ _x_ −2 = _x_ and label it _x_ −2 to indicate the second step. We then use _x_ +2 ≡ _x_ +1 and _x_ −2 as the next interval and continue the process. We see that only _x_ − changes for the first three steps in this example, but for the fourth step _x_ + finally changes. The changes then become too small for us to show.

### 7.2.1 Implementation: Bisection Algorithm

  1. The first step in implementing any search algorithm is to get an idea of what your function looks like. For the present problem, you do this by making a plot of   Note from your plot some approximate values at which  . Your program should be able to find more exact values for these zeros.
  2. Write a program that implements the bisection algorithm and use it to find some solutions of (7.2).
  3. _Warning:_ Because the tan function has singularities, you have to be careful. In fact, your graphics program (or Maple) may not function accurately near these singularities. One cure is to use a different but equivalent form of the equation. Show that an equivalent form of (7.2) is

(7.6)

  4. Make a second plot of (7.6), which also has singularities but at different places. Choose some approximate locations for zeros from this plot.
  5. Evaluate _f_ ( _E_ B) and thus determine directly the precision of your solution.
  6. Compare the roots you find with those given by Maple or Mathematica.

## 7.3 Improved Algorithm: Newton–Raphson Searching

The Newton–Raphson algorithm finds approximate roots of the equation

(7.7)

more quickly than the bisection method. As we see graphically in Figure 7.2, this algorithm is the equivalent of drawing a straight line   tangent to the curve at an _x_ value for which   and then using the intercept of the line with the _x_ -axis at   as an improved guess for the root. If the "curve" was a straight line, the answer would be exact; otherwise, it is a good approximation if the guess is close enough to the root for _f_ ( _x_ ) to be nearly linear. The process continues until some set level of precision is reached. If a guess is in a region where _f_ ( _x_ ) is nearly linear (Figure 7.2), then the convergence is much more rapid than for the bisection algorithm.

The analytic formulation of the Newton–Raphson algorithm starts with an old guess _x_ 0 and expresses a new guess _x_ as a correction _Δx_ to the old guess:

(7.8)

(7.9)

**Figure 7.2** A graphical representation of the steps involved in solving for a zero of _f_ ( _x_ ) using the Newton–Raphson method. The Newton–Raphson method takes the new guess as the zero of the line tangent to _f_ ( _x_ ) at the old guess. Two guesses are shown.

We next expand the known function _f_ ( _x_ ) in a Taylor series around _x_ 0 and keep only the linear terms:

(7.10)

We then determine the correction Δ _x_ by calculating the point at which this linear approximation to _f_ ( _x_ ) crosses the _x_ -axis:

(7.11)

(7.12)

The procedure is repeated starting at the improved _x_ until some set level of precision is obtained.

The Newton–Raphson algorithm (7.12) requires evaluation of the derivative d _f_ / d _x_ at each value of _x_ 0. In many cases, you may have an analytic expression for the derivative and can build it into the algorithm. However, especially for more complicated problems, it is simpler and less error-prone to use a numerical forward-difference approximation to the derivative2):

(7.13)

where _δx_ is some small change in _x_ that you just chose (different from the Δ used for searching in (7.12)). While a central-difference approximation for the derivative would be more accurate, it would require additional evaluations of the _f_ 's, and once you find a zero, it does not matter how you got there. In Listing 7.2, we give a program `**NewtonCD.py**` that implement the search with the central difference derivative.

**Listing 7.2** **NewtonCD.py** uses the Newton–Raphson method to search for a zero of the function _f_ ( _x_ ). A central-difference approximation is used to determine d _f_ /d _x_.

### 7.3.1 Newton–Raphson with Backtracking

Two examples of possible problems with the Newton–Raphson algorithm are shown in Figure 7.3. In Figure 7.3a, we see a case where the search takes us to an _x_ value where the function has a local minimum or maximum, that is, where d _f_ / d _x_ = 0. Because Δ _x_ = – _f_ / _f_ ′, this leads to a horizontal tangent (division by zero), and so the next guess is _x_ = ∞, from where it is hard to return. When this happens, you need to start your search with a different guess and pray that you do not fall into this trap again. In cases where the correction is very large but maybe not infinite, you may want to try backtracking (described below) and hope that by taking a smaller step you will not get into as much trouble.

In Figure 7.3b, we see a case where a search falls into an infinite loop surrounding the zero without ever getting there. A solution to this problem is called _backtracking._ As the name implies, in cases where the new guess _x_ 0 \+ Δ _x_ leads to an increase in the magnitude of the function,  , you can backtrack somewhat and try a smaller guess, say, _x_ 0 \+ Δ _x_ /2. If the magnitude of _f_ still increases, then you just need to backtrack some more, say, by trying _x_ 0 \+ Δ _x_ /4 as your next guess, and so forth. Because you know that the tangent line leads to a local decrease in  _f_  , eventually an acceptable small enough step should be found.

The problem in both these cases is that the initial guesses were not close enough to the regions where _f_ ( _x_ ) is approximately linear. So again, a good plot may help produce a good first guess. Alternatively, you may want to start your search with the bisection algorithm and then switch to the faster Newton–Raphson algorithm when you get closer to the zero.

**Figure 7.3** Two examples of how the Newton–Raphson algorithm may fail if the initial guess is not in the region where _f_ ( _x_ ) can be approximated by a straight line. (a) A guess lands at a local minimum/maximum, that is, a place where the derivative vanishes, and so the next guess ends up at _x_ = ∞. (b)The search has fallen into an infinite loop. The technique know as "backtracking" could eliminate this problem.

### 7.3.2 Implementation: Newton–Raphson Algorithm

  1. Use the Newton–Raphson algorithm to find some energies _E_ B that are solutions of (7.2). Compare these solutions with the ones found with the bisection algorithm.
  2. Again, notice that the 10 in (7.2) is proportional to the strength of the potential that causes the binding. See if making the potential deeper, say, by changing the 10 to a 20 or a 30, produces more or deeper bound states. (Note that in contrast to the bisection algorithm, your initial guess must be closer to the answer for the Newton–Raphson algorithm to work.)
  3. Modify your algorithm to include backtracking and then try it out on some difficult cases.
  4. Evaluate _f_ ( _E_ B) and thus determine directly the precision of your solution.

## 7.4 Problem 2: Temperature Dependence of Magnetization

**Problem** Determine _M_ ( _T_ ) the magnetization as a function of temperature for simple magnetic materials.

A collection of _N_ spin-1/2 particles each with the magnetic moment _µ_ is at temperature _T_. The collection has an external magnetic field _B_ applied to it and comes to equilibrium with _N_ L particles in the lower energy state (spins aligned with the magnetic field), and with _N_ U particles in the upper energy state (spins opposed to the magnetic field). The Boltzmann distribution law tells us that the relative probability of a state with energy _E_ is proportional to  , where _k_ B is Boltzmann's constant. For a dipole with moment _µ_ , its energy in a magnetic field is given by the dot product  . Accordingly, spin-up particle have lower energy in a magnetic field than spin-down particles, and thus are more probable.

Applying the Boltzmann distribution to our spin problem, we have that the number of particles in the lower energy level (spin up) is

(7.14)

while the number of particles in the upper energy level (spin down) is

(7.15)

As discussed by (Kittel, 2005), we now assume that the molecular magnetic field _B = λ M_ is much larger than the applied magnetic field and so replace _B_ by the molecular field. This permits us to eliminate _B_ from the preceding equations. The _magnetization M_ ( _T_ ) is given by the individual magnetic moment _µ_ times the net number of particles pointing in the direction of the magnetic field:

(7.16)

(7.17)

Note that this expression appears to make sense because as the temperature approaches zero, all spins will be aligned along the direction of _B_ and so _M_ ( _T_ = 0) = _Nµ_.

**Solution via Searching** Equation 7.17 relates the magnetization and the temperature. However, it is not really a solution to our problem because _M_ appears on the LHS of the equation as well as within the hyperbolic function on the RHS. Generally, a _transcendental equation_ of this sort does not have an analytic solution that would give _M_ as simply a function of the temperature _T._ But by working backward, we can find a numerical solution. To do that we first express (7.17) in terms of the reduced magnetization _m_ , the reduced temperature _t,_ and the Curie temperature _T c_:

(7.18)

(7.19)

While it is no easier to find an analytic solution to (7.18) than it was to (7.17), the simpler form of (7.18) makes the programming easier as we search for values of _t_ and _m_ that satisfy (7.18).

**Figure 7.4** A function of the reduced magnetism _m_ at three reduced temperatures _t_. A zero of this function determines the value of the magnetism at a particular value of _t_.

One approach to a trial-and-error solution is to define a function

(7.20)

and then, for a variety of fixed _t_ = _t i_ values, search for those _m_ values at which _f_ ( _m, t i_) = 0. (One could just as well fix the value of _m_ to _m j_ and search for the value of _t_ for which _f_ ( _m j, t_) = 0; once you have a solution, you have a solution.) Each zero so found gives us a single value of _m_ ( _t i_). A plot or a table of these values for a range of _t i_ values then provides the best we can do as the desired solution _m_ ( _t_ ).

Figure 7.4 shows three plots of _f_ ( _m,t_ ) as a function of the reduced magnetization _m_ , each plot for a different value of the reduced temperature. As you can see, other than the uninteresting solution at _m_ = 0, there is only one solution (a zero) near _m_ = 1 for _t_ = 0.5, and no solution at other temperatures.

### 7.4.1 Searching Exercise

  1. Find the root of (7.20) to six significant figures for _t_ = 0.5 using the bisection algorithm.
  2. Find the root of (7.20) to six significant figures for _t_ = 0.5 using the Newton–Raphson algorithm.
  3. Compare the time it takes to find the solutions for the bisection and Newton–Raphson algorithms.
  4. Construct a plot of the reduced magnetization _m_ ( _t_ ) as a function of the reduced temperature _t_.

## 7.5 Problem 3: Fitting An Experimental Spectrum

* * *

Data fitting is an art worthy of serious study by all scientists (Bevington and Robinson, 2002). In the sections to follow, we just scratch the surface by examining how to interpolate within a table of numbers and how to do a least-squares fit to data. We also show how to go about making a least-squares fit to nonlinear functions using some of the search techniques and subroutine libraries we have already discussed.

* * *

**Problem** The cross sections measured for the resonant scattering of neutrons from a nucleus are given in Table 7.1. Your _problem_ is to determine values for the cross sections at energy values lying between those in the table.

You can solve this _problem_ in a number of ways. The simplest is to numerically _interpolate_ between the values of the experimental _f_ ( _E i_) given in Table 7.1. This is direct and easy but does not account for there being experimental noise in the data. A more appropriate solution (discussed in Section 7.7) is to find the _best fit_ of a theoretical function to the data. We start with what we believe to be the "correct" theoretical description of the data,

**Table 7.1** Experimental values for a scattering cross section ( _f_ ( _E_ ) in the theory), each with absolute error _±σ i_, as a function of energy ( _x i_ in the theory).

_i_ = | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
---|---|---|---|---|---|---|---|---|---
_E i_ (MeV) | 0 | 25 | 50 | 75 | 100 | 125 | 150 | 175 | 200
_g(E i)_ (mb) | 10.6 | 16.0 | 45.0 | 83.5 | 52.8 | 19.9 | 10.8 | 8.25 | 4.7
Error (mb) | 9.34 | 17.9 | 41.5 | 85.5 | 51.5 | 21.5 | 10.8 | 6.29 | 4.14

(7.21)

where _f r, Er_, and _Γ_ are unknown parameters. We then adjust the parameters to obtain the best fit. This is a best fit in a statistical sense but in fact may not pass through all (or any) of the data points. For an easy, yet effective, introduction to statistical data analysis, we recommend (Bevington and Robinson, 2002).

These two techniques of interpolation and least-squares fitting are powerful tools that let you treat tables of numbers as if they were analytic functions and sometimes let you deduce statistically meaningful constants or conclusions from measurements. In general, you can view data fitting as _global_ or _local_. In global fits, a single function in _x_ is used to represent the entire set of numbers in a table like Table 7.1. While it may be spiritually satisfying to find a single function that passes through all the data points, if that function is not the correct function for describing the data, the fit may show nonphysical behavior (such as large oscillations) between the data points. The rule of thumb is that if you must interpolate, keep it local and view global interpolations with a critical eye.

Consider Table 7.1 as ordered data that we wish to interpolate. We call the independent variable _x_ and its tabulated values _x i_( _i_ = 1, 2,...), and assume that the dependent variable is the function _g_ ( _x_ ), with the tabulated values _g i_ = _g_ ( _x i_). We assume that _g_ ( _x_ ) can be approximated as an ( _n_ – 1)-degree polynomial in each interval _i_ :

(7.22)

Because our fit is local, we do not assume that one _g_ ( _x_ ) can fit all the data in the table but instead use a different polynomial, that is, a different set of _a i_ values, for each interval. While each polynomial is of low degree, multiple polynomials are needed to span the entire table. If some care is taken, the set of polynomials so obtained will behave well enough to be used in further calculations without introducing much unwanted noise or discontinuities.

The classic interpolation formula was created by Lagrange. He figured out a closed-form expression that directly fits the ( _n_ – 1)-order polynomial (7.22) to _n_ values of the function _g_ ( _x_ ) evaluated at the points _x i_ . The formula for each interval is written as the sum of polynomials:

(7.23)

(7.24)

For three points, (7.23) provides a second-degree polynomial, while for eight points it gives a seventh-degree polynomial. For example, assume that we are given the points and function values

(7.25)

With four points, the Lagrange formula determines a third-order polynomial that reproduces each of the tabulated values:

(7.26)

As a check, we see that

(7.27)

If the data contain little noise, this polynomial can be used with some confidence within the range of the data, but with risk beyond the range of the data.

Notice that Lagrange interpolation has no restriction that the points _x i_ be evenly spaced. Usually, the Lagrange fit is made to only a small region of the table with a small value of _n_ , despite the fact that the formula works perfectly well for fitting a high-degree polynomial to the entire table. The difference between the value of the polynomial evaluated at some _x_ and that of the actual function can be shown to be the _remainder_

(7.28)

where ζ lies somewhere in the interpolation interval. What significant here is that we see that if significant high derivatives exist in _g_ ( _x_ ), then it cannot be approximated well by a polynomial. For example, a table of noisy data would have significant high derivatives.

### 7.5.1 Lagrange Implementation, Assessment

Consider the experimental neutron scattering data in Table 7.1. The expected theoretical functional form that describes these data is (7.21), and our empirical fits to these data are shown in Figure 7.5.

  1. Write a subroutine to perform an _n_ -point Lagrange interpolation using (7.23). Treat _n_ as an arbitrary input parameter. (You may also perform this exercise with the spline fits discussed in Section 7.5.2.)
  2. Use the Lagrange interpolation formula to fit the entire experimental spectrum with one polynomial. (This means that you must fit all nine data points with an eight-degree polynomial.) Then use this fit to plot the cross section in steps of 5MeV.
  3. Use your graph to deduce the resonance energy _E_ r (your peak position) and _Γ_ (the full-width at half-maximum). Compare your results with those predicted by a theorist friend, ( _E_ r, _Γ_ ) = (78, 55) MeV.
  4. A more realistic use of Lagrange interpolation is for local interpolation with a small number of points, such as three. Interpolate the preceding cross-sectional data in 5-MeV steps using three-point Lagrange interpolation for each interval. (Note that the end intervals may be special cases.)
  5. We deliberately have not discussed _extrapolation_ of data because it can lead to serious _systematic_ errors; the answer you get may well depend more on the function you assume than on the data you input. Add some adventure to your life and use the programs you have written to extrapolate to values outside Table 7.1. Compare your results to the theoretical Breit–Wigner shape (7.21).

This example shows how easy it is to go wrong with a high-degree-polynomial fit to data with errors. Although the polynomial is guaranteed to pass through all the data points, the representation of the function away from these points can be quite unrealistic. Using a low-order interpolation formula, say, _n_ = 2 or 3, in each interval usually eliminates the wild oscillations, but may not have any theoretical justification. If these local fits are matched together, as we discuss in the next section, a rather continuous curve results. Nonetheless, you must recall that if the data contain errors, a curve that actually passes through them may lead you astray. We discuss how to do this properly with least-squares fitting in Section 7.7.

### 7.5.2 Cubic Spline Interpolation (Method)

If you tried to interpolate the resonant cross section with Lagrange interpolation, then you saw that fitting parabolas (three-point interpolation) within a table may avoid the erroneous and possibly catastrophic deviations of a high-order formula. (A two-point interpolation, which connects the points with straight lines, may not lead you far astray, but it is rarely pleasing to the eye or precise.) A sophisticated variation of an _n_ = 4 interpolation, known as _cubic splines_ , often leads to surprisingly eye-pleasing fits. In this approach (Figure 7.5), cubic polynomials are fit to the function in each interval, with the additional constraint that the first and second derivatives of the polynomials be continuous from one interval to the next. This continuity of slope and curvature is what makes the spline fit particularly eye-pleasing. It is analogous to what happens when you use the flexible spline drafting tool (a lead wire within a rubber sheath) from which the method draws its name.

**Figure 7.5** Three fits to data. _Dashed_ : Lagrange interpolation using an eight-degree polynomial; _Short dashes_ : cubic splines fit ; _Long dashed_ : Least-squares parabola fit.

The series of cubic polynomials obtained by spline-fitting a table of data can be integrated and differentiated and is guaranteed to have well-behaved derivatives. The existence of meaningful derivatives is an important consideration. As a case in point, if the interpolated function is a potential, you can take the derivative to obtain the force. The complexity of simultaneously matching polynomials and their derivatives over all the interpolation points leads to many simultaneous linear equations to be solved. This makes splines unattractive for hand calculation, yet easy for computers and, not surprisingly, popular in both calculations and computer drawing programs. To illustrate, the smooth solid curve in Figure 7.5 is a spline fit.

The basic approximation of splines is the representation of the function _g_ ( _x_ ) in the subinterval [ _x i_, _x_ _i_ +1] with a cubic polynomial:

(7.29)

(7.30)

This representation makes it clear that the coefficients in the polynomial equal the values of _g_ ( _x_ ) and its first, second, and third derivatives at the tabulated points _x i_. Derivatives beyond the third vanish for a cubic. The computational chore is to determine these derivatives in terms of the _N_ tabulated _g i_ values. The matching of _g i_ at the _nodes_ that connect one interval to the next provides the equations

(7.31)

The matching of the first _and_ second derivatives at each interval's boundaries provides the equations

(7.32)

The additional equations needed to determine all constants are obtained by matching the third derivatives at adjacent nodes. Values for the third derivatives are found by approximating them in terms of the second derivatives:

(7.33)

As discussed in Chapter 5, a _central-difference approximation_ would be more accurate than a forward-difference approximation, yet (7.33) keeps the equations simpler.

It is straightforward yet complicated to solve for all the parameters in (7.30). We leave that to the references (Thompson, 1992; Press _et al.,_ 1994). We can see, however, that matching at the boundaries of the intervals results in only ( _N_ – 2) linear equations for _N_ unknowns. Further input is required. It usually is taken to be the boundary conditions at the endpoints _a = x_ 1 and _b = x N_, specifically, the second derivatives _g″(a)_ and _g″(b)_. There are several ways to determine these second derivatives:

**Natural spline:** Set _g″(a)_ = _g″(b)_ = 0; that is, permit the function to have a slope at the endpoints but no curvature. This is "natural" because the derivative vanishes for the flexible spline drafting tool (its ends being free).

**Input values for _g′_ at the boundaries:** The computer uses _g′(a)_ to approximate _g″(a)._ If you do not know the first derivatives, you can calculate them numerically from the table of _g i_ values.

**Input values for _g″_ at the boundaries:** Knowing values is of course better than approximating values, but it requires the user to input information. If the values of _g″_ are not known, they can be approximated by applying a forward-difference approximation to the tabulated values:

(7.34)

#### 7.5.2.1 Cubic Spline Quadrature (Exploration)

A powerful integration scheme is to fit an integrand with splines and then integrate the cubic polynomials analytically. If the integrand _g_ ( _x_ ) is known only at its tabulated values, then this is about as good an integration scheme as is possible; if you have the ability to calculate the function directly for arbitrary _x_ , Gaussian quadrature may be preferable. We know that the spline fit to _g_ in each interval is the cubic (7.30)

(7.35)

It is easy to integrate this to obtain the integral of _g_ for this interval and then to sum over all intervals:

(7.36)

(7.37)

Making the intervals smaller does not necessarily increase precision, as subtractive cancelations in (7.36) may get large.

**Spline Fit of Cross Section (Implementation)** Fitting a series of cubics to data is a little complicated to program yourself, so we recommend using a library routine. While we have found quite a few Java-based spline applications available on the Internet, none seemed appropriate for interpreting a simple set of numbers. That being the case, we have adapted the `**splint.c**` and the `**spline.c**` functions from (Press _et al.,_ 1994) to produce the `**SplineInteract.py**` program shown in Listing 7.3 (there is also an applet). Your _problem_ for this section is to carry out the assessment in Section 7.5.1 using cubic spline interpolation rather than Lagrange interpolation.

## 7.6 Problem 4: Fitting Exponential Decay

**Problem** Figure 7.6 presents actual experimental data on the number of decays Δ _N_ of the π meson as a function of time (Stetz _et al.,_ 1973). Notice that the time has been "binned" into Δ _t_ = 10 ns intervals and that the smooth curve is the theoretical exponential decay expected for very large numbers of pions (which there is not). Your _problem_ is to deduce the lifetime _τ_ of the π meson from these data (the tabulated lifetime of the pion is 2.6 × 10–8 s).

**Theory** Assume that we start with _N_ 0 particles at time _t_ = 0 that can decay to other particles.3) If we wait a short time Δ _t_ , then a small number Δ _N_ of the particles will decay _spontaneously,_ that is, with no external influences. This decay is a stochastic process, which means that there is an element of chance involved in just when a decay will occur, and so no two experiments are expected to give exactly the same results. The basic law of nature for spontaneous decay is that the number of decays Δ _N_ in a time interval Δ _t_ is proportional to the number of particles _N_ ( _t_ ) present at that time and to the time interval

(7.38)

**Listing 7.3** **SplineInteract.py** performs a cubic spline fit to data and permits interactive control. The arrays _x_ [] and _y_ [] are the data to fit, and the values of the fit at Nfit points are output.

**Figure 7.6** A reproduction of the experimental measurement by Stetz _et al_. (1973) giving the number of decays of a π meson as a function of time since its creation. Measurements were made during time intervals (box sizes) of 10-ns width. The dashed curve is the result of a linear least-squares fit to the log _N_ ( _t_ ).

Here _τ_ = 1/ _λ_ is the _lifetime_ of the particle, with _λ_ the rate parameter. The actual decay _rate_ is given by the second equation in (7.38). If the number of decays Δ _N_ is very small compared to the number of particles _N_ , and if we look at vanishingly small time intervals, then the difference equation (7.38) becomes the differential equation

(7.39)

This differential equation has an exponential solution for the number as well as for the decay rate:

(7.40)

Equation 7.40 is the theoretical formula we wish to "fit" to the data in Figure 7.6. The output of such a fit is a "best value" for the lifetime _τ_.

## 7.7 Least-Squares Fitting (Theory)

Books have been written and careers have been spent discussing what is meant by a "good fit" to experimental data. We cannot do justice to the subject here and refer the reader to Bevington and Robinson (2002); Press _et al_. (1994); Thompson (1992). However, we will emphasize three points:

  1. If the data being fit contain errors, then the "best fit" in a statistical sense should not pass through all the data points.
  2. If the theory is not an appropriate one for the data (e.g., the parabola in Figure 7.5), then its best fit to the data may not be a good fit at all. This is good, for this is how we know that the theory is not right.
  3. Only for the simplest case of a linear least-squares fit, can we write down a closed-form solution to evaluate and obtain the fit. More realistic problems are usually solved by _trial-and-error_ search procedures, sometimes using sophisticated subroutine libraries. However, in Section 7.8.2 we show how to conduct such a nonlinear search using familiar tools.

Imagine that you have measured _N_ D data values of the independent variable _y_ as a function of the dependent variable _x:_

(7.41)

where ± _σ i_ is the experimental uncertainty in the _i_ th value of _y_. (For simplicity we assume that all the errors _σ i_ occur in the dependent variable, although this is hardly ever true (Thompson, 1992)). For our problem, _y_ is the number of decays as a function of time, and _x i_ are the times. Our goal is to determine how well a mathematical function _y = g_ ( _x_ ) (also called a _theory_ or a _model_ ) can describe these data. Additionally, if the theory contains some parameters or constants, our goal can be viewed as determining the best values for these parameters. We assume that the theory function _g_ ( _x_ ) contains, in addition to the functional dependence on _x_ ,an additional dependence upon _M_ P parameters { _a_ 1, _a_ 2,..., _a_ _M_ P}. Notice that the parameters { _a m_} are not variables, in the sense of numbers read from a meter, but rather are parts of the theoretical model, such as the size of a box, the mass of a particle, or the depth of a potential well. For the exponential decay function (7.40), the parameters are the lifetime _τ_ and the initial decay rate d _N_ (0)/ _dt._ We include the parameters as

(7.42)

where the _a i_'s are parameters and _x_ the independent variable. We use the chisquare ( _χ_ 2) measure (Bevington and Robinson, 2002) as a gauge of how well a theoretical function _g_ reproduces data:

(7.43)

where the sum is over the _N_ D experimental points ( _x i, yi ±σi_). The definition (7.43) is such that smaller values of _χ_ 2 are better fits, with _χ_ 2 = 0 occurring if the theoretical curve went through the center of every data point. Notice also  that the  weighting means that measurements with larger errors4) contribute less to _χ_ 2.

_Least-squares fitting_ refers to adjusting the parameters in the theory until a minimum in _χ_ 2 is found, that is, finding a curve that produces the least value for the summed squares of the deviations of the data from the function _g_ ( _x_ ). In general, this is the best fit possible and the best way to determine the parameters in a theory. The _M_ P parameters { _a m, m_ = 1, _M_ P} that make _χ_ 2 an extremum are found by solving the _M_ P equations:

(7.44)

Often, the function _g_ ( _x_ ; { _a m_}) has a sufficiently complicated dependence on the _a m_ values for (7.44) to produce _M_ P simultaneous nonlinear equations in the _a m_ values. In these cases, solutions are found by a trial-and-error search through the _M_ P-dimensional parameter space, as we do in Section 7.8.2. To be safe, when such a search is completed, you need to check that the minimum _χ_ 2 you found is _global_ and not _local_. One way to do that is to repeat the search for a whole grid of starting values, and if different minima are found, to pick the one with the lowest _χ_ 2.

### 7.7.1 Least-Squares Fitting: Theory and Implementation

When the deviations from theory are as a result of random errors and when these errors are described by a Gaussian distribution, there are some useful rules of thumb to remember (Bevington and Robinson, 2002). You know that your fit is good if the value of _χ_ 2 calculated via the definition (7.43) is approximately equal to the number of degrees of freedom _χ_ 2  _N_ D− _M_ p where _N D_ is the number of data points and _M_ P is the number of parameters in the theoretical function. If your _χ_ 2 is much less than _N_ D − _M_ P, it does not mean that you have a "great" theory or a really precise measurement; instead, you probably have too many parameters or have assigned errors ( _σ i_ values) that are too large. In fact, too small _χ_ 2 may indicate that you are fitting the random scatter in the data rather than missing approximately one-third of the error bars, as expected if the errors are random. If your _χ_ 2 is significantly greater than _N_ D − _M_ P, the theory may not be good, you may have significantly underestimated your errors, or you may have errors that are not random.

The _M_ P simultaneous equations (7.44) can be simplified considerably if the functions _g_ ( _x_ ; { _a m_}) depend _linearly_ on the parameter values _a i_, for example,

(7.45)

**Figure 7.7** A linear least-squares best fit of a straight line to data. The deviation of theory from experiment is greater than would be expected from statistics, which means that a straight line is not a good theory to describe these data.

In this case (also known as _linear regression or straight-line fit)_ ,as shown in Figure 7.7, there are _M_ p = 2 parameters, the slope _a_ 2, and the _y_ intercept _a_ 1. Notice that while there are only two parameters to determine, there still may be an arbitrary number _N_ D of data points to fit. Remember, a unique solution is not possible unless the number of data points is equal to or greater than the number of parameters. For this linear case, there are just two derivatives,

(7.46)

and after substitution, the _χ_ 2 minimization equations (7.44) can be solved (Press _et al.,_ 1994):

(7.47)

(7.48)

(7.49)

Statistics also gives you an expression for the _variance_ or uncertainty in the deduced parameters:

(7.50)

This is a measure of the uncertainties in the values of the fitted parameters arising from the uncertainties _σ i_ in the measured _y i_ values. A measure of the dependence  of the parameters on each other is given by the _correlation coefficient_ :

(7.51)

Here cov( _a_ 1, _a_ 2) is the _covariance_ of _a_ 1 and _a_ 2 and vanishes if _a_ 1 and _a_ 2 are independent. The correlation coefficient _ρ_ ( _a_ 1, _a_ 2)lies in the range −1 ≤ _ρ_ ≤ 1, with a positive _ρ_ indicating that the errors in _a_ 1 and _a_ 2 are likely to have the same sign, and a negative _ρ_ indicating opposite signs.

The preceding analytic solutions for the parameters are of the form found in statistics books but are not optimal for numerical calculations because subtractive cancelation can make the answers unstable. As discussed in Chapter 3, a rearrangement of the equations can decrease this type of error. For example, Thompson (1992) gives improved expressions that measure the data relative to their averages:

(7.52)

In `**Fit.py**` in Listing 7.4 we give a program that fits a parabola to some data. You can use it as a model for fitting a line to data, although you can use our closed-form expressions for a straight-line fit. In `**Fit.py**` on the instructor's site, we give a program for fitting to the decay data.

## 7.8 Exercises: Fitting Exponential Decay, Heat Flow and Hubble's Law

  1. Fit the exponential decay law (7.40) to the data in Figure 7.6. This means finding values for _τ_ and Δ _N_ (0)/Δ _t_ that provide a best fit to the data, and then judging how good the fit is.
    1. a) Construct a table of approximate values for(Δ _N_ /Δ _t i, ti_), for _i_ =1, _N_ D as read from Figure 7.6. Because time was measured in bins, _t i_ should correspond to the middle of a bin.
    2. b) Add an estimate of the error _σ i_ to obtain a table of the form (Δ _N_ /Δ _t i_ ± σ _i_ , _t i_). You can estimate the errors by eye, say, by estimating how much the histogram values appear to fluctuate about a smooth curve, or you can take   (This last approximation is reasonable for large numbers, which this is not.)
    3. c) In the limit of very large numbers, we would expect a plot of ln |d _N_ /d _t_ |vs. _t_ to be a straight line:

(7.53)

This means that if we treat ln |Δ _N_ ( _t_ )/Δ _t_ | as the dependent variable and time Δ _t_ as the independent variable, we can use our linear-fit results. Plot ln|Δ _N_ /Δ _t_ | vs. Δ _t_.

    4. d) Make a least-squares fit of a straight line to your data and use it to determine the lifetime _τ_ of the π meson. Compare your deduction to the tabulated lifetime of 2.6 × 10−8 s and comment on the difference.
    5. e) Plot your best fit on the same graph as the data and comment on the agreement.
    6. f) Deduce the goodness of fit of your straight line and the approximate error in your deduced lifetime. Do these agree with what your "eye" tells you?
    7. g) Now that you have a fit, look at the data again and estimate what a better value for the errors in the ordinates might be.
  2. Table 7.2 gives the temperature _T_ along a metal rod whose ends are kept at a fixed constant temperature. The temperature is a function of the distance _x_ along the rod.
    1. a) Plot the data in Table 7.3 to verify the appropriateness of a linear relation

(7.54)

    2. b) Because you are not given the errors for each measurement, assume that the least significant figure has been rounded off and so _σ_ ≥ 0.05.
    3. c) Use that to compute a least-squares straight-line fit to these data.
    4. d) Plot your best _a_ \+ _bx_ on the curve with the data.
    5. e) After fitting the data, compute the variance and compare it to the deviation of your fit from the data. Verify that about one-third of the points miss the σ error band (that is what is expected for a normal distribution of errors).
    6. f) Use your computed variance to determine the _χ_ 2 of the fit. Comment on the value obtained.
    7. g) Determine the variances _σ a_ and _σ b_ and check whether it makes sense to use them as the errors in the deduced values for _a_ and _b_.
  3. In 1929, Edwin Hubble examined the data relating the radial velocity _v_ of 24 extra galactic nebulae to their distance _r_ from our galaxy (Hubble, 1929). Although there was considerable scatter in the data, he fit them with a straight line:

(7.55)

where _H_ is now called the Hubble constant. Table 7.3 contains the distances and velocities used by Hubble.
    1. a) Plot the data to verify the appropriateness of a linear relation

(7.56)

    2. b) Because you are not given the errors for each measurement, you may assume that the least significant figure has been rounded off and so _σ_ ≥ 1. Or, you may assume that astronomical measurements are hard to make and that there are at least 10% errors in the data.

**Table 7.2** Temperature vs. distance as measured along a metal rod.

_x i_ (cm) | 1.0 | 2.0 | 3.0 | 4.0 | 5.0 | 6.0 | 7.0 | 8.0 | 9.0
---|---|---|---|---|---|---|---|---|---
_T i_ (C) | 14.6 | 18.5 | 36.6 | 30.8 | 59.2 | 60.1 | 62.2 | 79.4 | 99.9

**Table 7.3** Distance vs. radial velocity for 24 extragalactic nebulae.

Object | _r_ (Mpc) | _v_ (km/s) | Object | _r_ (Mpc) | _v_ (km/s)
---|---|---|---|---|---
| 0.032 | 170 | 3627 | 0.9 | 650
| 0.034 | 290 | 4826 | 0.9 | 150
6822 | 0.214 | –130 | 5236 | 0.9 | 500
598 | 0.263 | –70 | 1068 | 1.0 | 920
221 | 0.275 | –185 | 5055 | 1.1 | 450
224 | 0.275 | –220 | 7331 | 1.1 | 500
5457 | 0.45 | 200 | 4258 | 1.4 | 500
4736 | 0.5 | 290 | 4141 | 1.7 | 960
5194 | 0.5 | 270 | 4382 | 2.0 | 500
4449 | 0.63 | 200 | 4472 | 2.0 | 850
4214 | 0.8 | 300 | 4486 | 2.0 | 800
3031 | 0.9 | –30 | 4649 | 2.0 | 1090
    3. c) Compute a least-squares straight-line fit to these data.
    4. d) Plot your best _a_ \+ _Hr_ on the curve with the data.
    5. e) After fitting the data, compute the variance and compare it to the deviation of your fit from the data. Verify that about one-third of the points miss the _σ_ error band (that is what is expected for a normal distribution of errors).
    6. f) Use your computed variance to determine the _χ_ 2 of the fit. Comment on the value obtained.
    7. g) Determine the variances _σ a_ and _σ b_ and check whether it makes sense to use them as the errors in the deduced values for _a_ and _b_.
    8. h) Now that you have a fit, look at the data again and estimate what a better value for the errors in the ordinates might be.

### 7.8.1 Linear Quadratic Fit

As indicated earlier, as long as the function being fitted depends _linearly_ on the unknown parameters _a i_, the condition of minimum _χ_ 2 leads to a set of simultaneous linear equations for the _a_ 's that can be solved by hand or on the computer using matrix techniques. To illustrate, suppose we want to fit the quadratic polynomial

(7.57)

to the experimental measurements ( _x i, yi, i_= l, _N_ D)(Figure 7.8). Because this _g_ ( _x_ ) is linear in all the parameters _a i_, we can still make a linear fit although _x_ is raised to the second power. (However, if we tried to a fit a function of the form _g_ ( _x_ ) = ( _a 1 \+ a2x_) exp(− _a 3x_) to the data, then we would not be able to make a linear fit because there is not a linear dependence on _a 3_)

**Figure 7.8** A linear least-squares best fit of a parabola to data. Here we see that the fit misses approximately one-third of the points, as expected from the statistics for a good fit.

The best fit of this quadratic to the data is obtained by applying the minimum _χ_ 2 condition (7.44) for _M_ p = 3 parameters and _N_ D (still arbitrary) data points. A solution represents the maximum likelihood that the deduced parameters provide a correct description of the data for the theoretical function _g_ ( _x_ ). Equation 7.44 leads to the three simultaneous equations for _a 1_, _a 2_,and _a_ 3:

(7.58)

(7.59)

(7.60)

_Note_ : Because the derivatives are independent of the parameters (the _a_ 's), the a dependence arises only from the term in square brackets in the sums, and because that term has only a linear dependence on the _a_ 's, these equations are linear in the _a_ 's.

**Exercise** Show that after some rearrangement, (7.58)–(7.60) can be written as

(7.61)

Here the definitions of the _S_ 's are simple extensions of those used in (7.47)–(7.49) and are programmed in `**Fit.py**` shown in Listing 7.4. After placing the three unknown parameters into a vector _x_ and the known three RHS terms in (7.61) into a vector **_b_** , these equations assume the matrix form

(7.62)

The solution for the parameter vector **_a_** is obtained by solving the matrix equations. Although for 3 × 3 matrices, we can write out the solution in a closed form, for larger problems the numerical solution requires matrix methods.

**Listing 7.4** **Fit.py** performs a least-squares fit of a parabola to data using the NumPy linalg package to solve the set of linear equations _Sa = s_.

**Linear Quadratic Fit Assessment**

  1. Fit the quadratic (7.57) to the following data sets [given as ( _x 1, y1_), ( _x 2, y2_),...] In each case, indicate the values found for the _a_ 's the number of degrees of freedom, _and_ the value of _χ_ 2.
    1. a) (0,1)
    2. b) (0,1), (1,3)
    3. c) (0,1), (1,3), (2, 7)
    4. d) (0,1), (1,3), (2, 7), (3,15)
  2. Find a fit to the last set of data to the function

(7.63)

_Hint_ : A judicious change of variables will permit you to convert this to a linear fit. Does a minimum _χ_ 2 still have meaning here?

### 7.8.2 Problem 5: Nonlinear Fit to a Breit–Wigner

**Problem** Remember how earlier in this chapter we interpolated the values in Table 7.1 in order to obtain the experimental cross section _∑_ as a function of energy. Although we did not use it, we also gave the theory describing these data, namely, the Breit-Wigner resonance formula (7.21):

(7.64)

Your _problem_ is to determine what values for the parameters ( _E_ r, _f_ r, and _Γ_ in (7.64) provide the best fit to the data in Table 7.1.

Because (7.64) is not a linear function of the parameters ( _E r_, _Σ_ 0, _Γ_ ), the three equations that result from minimizing _χ_ 2 are not linear equations and so cannot be solved by the techniques of _linear_ algebra (matrix methods). However, in our study of the masses on a string problem, we showed how to use the Newton–Raphson algorithm to search for solutions of simultaneous nonlinear equations. That technique involved expansion of the equations about the previous guess to obtain a set of linear equations and then solving the linear equations with the matrix libraries. We now use this same combination of fitting, trial-and-error searching, and matrix algebra to conduct a nonlinear least-squares fit of (7.64) to the data in Table 7.1.

Recall that the condition for a best fit is to find values of the _M_ p parameters _a m_ in the theory _g_ ( _x, a m_) that minimize . This leads to the _M_ p equations (7.44) to solve

(7.65)

To find the form of these equations appropriate to our problem, we rewrite our theory function (7.64) in the notation of (7.65):

(7.66)

(7.67)

The three derivatives required in (7.65) are then

(7.68)

Substitution of these derivatives into the best-fit condition (7.65) yields three simultaneous equations in _a_ 1, _a_ 2, and _a_ 3 that we need to solve in order to fit the _N_ D = 9 data points ( _x i, yi_) in Table 7.1:

(7.69)

Even without the substitution of (7.64) for _g_ ( _x, a_ ), it is clear that these three equations depend on the _a_ 's in a nonlinear fashion. That is okay because in Section 6.1.2 we derived the _N_ -dimensional Newton–Raphson search for the roots of

(7.70)

where we have made the change of variable _y i_ → _a i_ for the present problem. We use that same formalism here for the _N_ = 3 Equation 7.69 by writing them as

(7.71)

(7.72)

(7.73)

Because _f r_ ≡ _a 1_ is the peak value of the cross section, _E_ R ≡ _a 2_ is the energy at which the peak occurs, and   is the full width of the peak at half-maximum, good guesses for the _a_ 's can be extracted from a graph of the data. To obtain the nine derivatives of the three _f_ 's with respect to the three unknown _a_ 's, we use two nested loops over _i_ and _j_ , along with the forward-difference approximation for the derivative

(7.74)

where Δ _a j_ corresponds to a small, say ≤1%, change in the parameter value.

**Nonlinear Fit Implementation** Use the Newton–Raphson algorithm as outlined in Section 7.8.2 to conduct a nonlinear search for the best-fit parameters of the Breit–Wigner theory (7.64) to the data in Table 7.1. Compare the deduced values of ( _f_ r, _E_ R, _Γ_ ) to that obtained by inspection of the graph.

1) We solve this same problem in Section 9.1 using an approach that is applicable to almost any potential and which also provides the wave functions. The approach of this section works only for the eigen energies of a square well. 2) We discuss numerical differentiation in Chapter 5. 3) Spontaneous decay is discussed further and simulated in Section 4.5. 4) If you are not given the errors, you can guess them on the basis of the apparent deviation of the data from a smooth curve, or you can weigh all points equally by setting _σ i≡1_ and continue with the fitting.

# 8
Solving Differential Equations: Nonlinear Oscillations

* * *

Part of the joy of having computational tools at your disposal is that it is easy to solve almost every differential equation. Consequently, while most traditional (read "analytic") treatments of oscillations are limited to the small displacements about equilibrium where the restoring forces are linear, we eliminate those restrictions here and explore some interesting _nonlinear_ physics. First, we look at oscillators that may be harmonic for certain parameter values, but then become anharmonic We start with simple systems that have analytic solutions, use them to test various differential-equation solvers, and then include time-dependent forces to investigate nonlinear resonances and beating.1)

## 8.1 Free Nonlinear Oscillations

**Problem** In Figure 8.1, we show a mass _m_ attached to a spring that exerts a restoring force toward the origin, as well as a hand that exerts a time-dependent external force on the mass. We are told that the restoring force exerted by the spring is non-harmonic, that is, not simply proportional to displacement from equilibrium, but we are not given details as to how this is nonharmonic. Your _problem_ is to solve for the motion of the mass as a function of time. You may assume that the motion is constrained to one dimension.

## 8.2 Nonlinear Oscillators (Models)

This is a problem in classical mechanics for which Newton's second law provides us with the equation of motion

[8.1]

**Figure 8.1** A mass _m_ (the block) attached to a spring with restoring force _F k_( _x_ ) driven by an external time-dependent driving force (the hand).

where _F k_( _x_ ) is the restoring force exerted by the spring and _F_ ext( _x, t_ ) is the external force. Equation 8.1 is the differential equation we must solve for arbitrary forces. Because we are not told just how the spring departs from being linear, we are free to try out some different models. As our first model, we wish to try a potential that is a harmonic oscillator for small displacements _x_ and also contains a perturbation that introduces a nonlinear term to the force for large _x_ values:

[8.2]

[8.3]

[8.4]

where we have omitted the time-dependent external force. Equation 8.4 is the second-order ODE we need to solve. If _ax_   1, we should have essentially harmonic motion, but as _x_ → 1/ _α_ the anharmonic effects would be large.

We can understand the basic physics of this model by looking at the curves as shown in Figure 8.2a. As long as _x_ < 1/ _α_ , there will be a _restoring force_ and the motion will be periodic (repeated exactly and indefinitely in time), even although it is harmonic only for small-amplitude oscillations. Yet, if the amplitude of oscillation is large, there will be an asymmetry in the motion to the right and left of the equilibrium position. And if _x_ > 1/ _α_ , the force will become repulsive and the mass will be pushed away from the origin.

As a second model of a nonlinear oscillator, we assume that the spring's potential function is proportional to some arbitrary _even_ power _p_ of the displacement _x_ from equilibrium:

[8.5]

We require an even _p_ to ensure that the force,

[8.6]

**Figure 8.2** (a) The potentials of an harmonic oscillator (solid curve) and of an anharmonic oscillator (dashed curve). If the amplitude becomes too large for the anharmonic oscillator, the motion becomes unbound. (b) The shapes of the potential energy function _V_ ( _x_ ) ∝ | _x_ | _p_ for _p_ = 2 and _p_ = 6. The "linear" and "nonlinear" labels refer to the restoring force derived from these potentials.

contains an odd power of _p,_ which guarantees that it is a _restoring_ force for positive and negative _x_ values. We display some characteristics of this potential in Figure 8.2b. We see that _p_ = 2 is the harmonic oscillator and that _p_ = 6 is nearly a square well with the mass moving almost freely until it hits the wall at  . Regardless of the _p_ value, the motion will be periodic, but it will be harmonic only for _p_ = 2. Newton's law (8.1) gives the second-order ODE we need to solve

[8.7]

## 8.3 Types of Differential Equations (Math)

* * *

The background material in this section is presented to avoid confusion about semantics. The well-versed reader may want to skim or skip it.

**Order** A general form for a _first-order_ differential equation is

[8.8]

where the "order" refers to the degree of the derivative on the LHS. The derivative or force function _f_ ( _t_ , _y_ ) on the RHS is arbitrary. For instance, even if _f_ ( _t_ , _y_ ) is a nasty function of _y_ and _t_ such as

[8.9]

this is still first order in the derivative. A general form for a _second-order_ differential equation is

[8.10]

The derivative function _f_ on the RHS is arbitrary and may involve any power of the first derivative as well. To illustrate,

[8.11]

is a second-order differential equation, as is Newton's law (8.1).

In the differential equations (8.8) and (8.10), the time _t_ is the _independent_ variable and the position _y_ is the _dependent_ variable. This means that we are free to vary the time at which we want a solution, but not the value of the position _y_ at that time. Note that we often use the symbol _y_ or _Y_ for the dependent variable but that this is just a symbol. In some applications, we use _y_ to describe a position that is an independent variable instead of _t._

**Ordinary and partial** Differential equations such as (8.1) and (8.8) are _ordinary_ differential equations because they contain only _one_ independent variable, in these cases _t_. In contrast, an equation such as the Schrödinger equation

[8.12]

contains several independent variables, and this makes it a _partial differential equation_ (PDE). The partial derivative symbol _∂_ is used to indicate that the dependent variable _ψ_ depends simultaneously on several independent variables. In the early parts of this book, we limit ourselves to ordinary differential equations. In Chapters 19–, we examine a variety of PDEs.

**Linearand nonlinear** Part of the liberation of computational science is that we are no longer limited to solving _linear equations._ A _linear_ equation is one in which only the first power of _y_ or of d _n y_ / d _n t_ appears; a _nonlinear_ equation may contain higher powers. For example,

[8.13]

An important property of linear equations is the _law of linear superposition_ that lets us add solutions together to form new ones. As a case in point, if _A_ ( _t_ ) and _B_ ( _t_ ) are solutions of the linear equation in (8.13), then

[8.14]

is also a solution for arbitrary values of the constants _α_ and _β_. In contrast, even if we were clever enough to guess that the solution of the nonlinear equation in (8.13) is

[8.15]

(which you can verify by substitution), things would be amiss if we tried to obtain a more general solution by adding together two such solutions:

[8.16]

(which you can verify by substitution).

**Initial and boundary conditions** The general solution of a first-order differential equation always contains one arbitrary constant. The general solution of a second-order differential equation contains two such constants, and so forth. For any specific problem, these constants are fixed by the _initial conditions._ For a first-order equation, the sole initial condition may be the position _y_ ( _t_ ) at some time. For a second-order equation, the two initial conditions may be the position and velocity at some time. Regardless of how powerful the hardware and software that you utilize, mathematics remains valid, and so you must know the initial conditions in order to solve the problem uniquely.

In addition to the initial conditions, it is possible to further restrict the solutions of differential equations. One such way is by _boundary conditions_ that constrain the solution to have fixed values at the boundaries of the solution space. Problems of this sort are called _eigenvalue problems,_ and they are so demanding that solutions do not always exist, and even when they do exist, a trial-and-error search may be required to find them. In Chapter 9, we discuss how to extend the techniques of the present unit to boundary-value problems.

## 8.4 Dynamic Form for ODEs (Theory)

A standard form for ODEs, which has found proven to be useful in both numerical analysis (Press _et al.,_ 1994) and classical dynamics (Scheck, 1994; Tabor, 1989; José and Salatan, 1998), is to express ODEs of _any order_ as _N_ simultaneous first-order ODEs in the _N_ unknowns  :

[8.17]

[8.18]

[8.19]

where a _y_ ( _i_ ) dependence in _f_ is allowed, but not any dependence on the derivatives  . These equations can be expressed more succinctly by use of the _N_ -dimensional vectors (indicated here in boldface italic) **_y_** and **_f_** :

[8.20]

The utility of such compact notation is that we can study the properties of the ODEs, as well as develop algorithms to solve them, by dealing with the single equation (8.20) without having to worry about the individual components. To see how this works, let us convert Newton's law

[8.21]

to this standard form. The rule is that the RHS may _not_ contain any explicit derivatives, although individual components of _y_ ( _i_ ) may represent derivatives. To pull this off, we define the position _x_ as the first dependent variable _y_ (0) and the velocity d _x_ /d _t_ as the second dependent variable _y_ (1):

[8.22]

The second-order ODE (8.21) now becomes two simultaneous first-order ODEs:

[8.23]

This expresses the acceleration (the second derivative in (8.21)) as the first derivative of the velocity [ _y_ (1)]. These equations are now in the standard form (8.20), with the derivative or force function _f_ having the two components

[8.24]

where _F_ may be an explicit function of time as well as of position and velocity.

To be even more specific, applying these definitions to our spring problem (8.7), we obtain the coupled first-order equations

[8.25]

where _y_ (0)( _t_ ) is the position of the mass at time _t_ and _y_ (1)( _t_ ) is its velocity. In the standard form, the components of the force function and the initial conditions are

[8.26]

## 8.5 ODE Algorithms

The classic way to solve an ODE is to start with the known initial value of the dependent variable,  , and then use the derivative function _f_ ( _t_ , _y_ ) to advance the initial value one small step _h_ forward in time to obtain  . Once you can do that, you can solve the ODE for all _t_ values by just continuing stepping to larger times, one small _h_ at a time (Figure 8.3).2) Error is always a concern when integrating differential equations because derivatives require small differences, and small differences are prone to subtractive cancelations and roundoff error accumulation. In addition, because our stepping procedure for solving the differential equation is a continuous extrapolation of the initial conditions, with each step building on a previous extrapolation, this is somewhat like a castle built on sand; in contrast to interpolation, there are no tabulated values on which to anchor your solution.

It is simplest if the time steps used throughout the integration remain constant in size, and that is mostly what we shall do. Industrial-strength algorithms, such as the one we discuss in Section 8.6, adapt the step size by making _h_ larger in regions where _y_ varies slowly (this speeds up the integration and cuts down on the round-off error) and making _h_ smaller in regions where _y_ varies rapidly.

### 8.5.1 Euler's Rule

Euler's rule (Figure 8.4) is a simple algorithm for integrating the differential equation (8.8) by one step and is just the forward-difference algorithm for the derivative:

[8.27]

[8.28]

where   is the value of _y_ at time _t n._ We know from our discussion of differentiation that the error in the forward-difference algorithm is  , and so then is the error in Euler's rule.

**Figure 8.3** A sequence of uniform steps of length _h_ taken in solving a differential equation. The solution starts at time _t_ = 0 and is integrated in steps of _h_ until _t_ = _T_.

**Figure 8.4** Euler's algorithm for integration of a differential equation one step forward in time. This linear extrapolation with the slope evaluated at the initial point is seen to lead to an error Δ.

To indicate the simplicity of this algorithm, we apply it to our oscillator problem (8.4) for the first time step:

[8.29]

Compare these to the projectile equations familiar from first-year physics,

[8.30]

and we see that with Euler's rule the acceleration does not contribute to the distance covered (no _h_ 2 term), yet it does contribute to the velocity (and so will contribute belatedly to the distance in the next time step). This is clearly a simple algorithm that requires very small _h_ values to obtain precision. Yet using small values for _h_ increases the number of steps and the accumulation of the roundoff error, which may lead to instability.3) Whereas we do not recommend Euler's algorithm for general use, it is commonly used to start off a more precise algorithm.

## 8.6 Runge–Kutta Rule

Although no single algorithm is good for solving all ODEs, the fourth-order Runge–Kutta algorithm, `**rk4**`, or its extension with adaptive step size, `**rk45**`, has proven to be robust and capable of industrial-strength work. In spite of `**rk4**` being our recommended standard method, we derive the simpler `**rk2**` here, and just state the result for `**rk4**`.

**Figure 8.5** The `**rk2**` algorithm for integration of a differential equation uses a slope (bold line segment) evaluated at the interval's midpoint, and is seen to lead to a smaller error than Euler's algorithm in Figure 8.4.

The Runge–Kutta algorithms for integrating a differential equation are based upon the formal (exact) integral of our differential equation:

[8.31]

[8.32]

To derive the second-order Runge–Kutta algorithm `**rk2**` (Figure 8.5 and `**rk2.py**`), we expand _f_ ( _t, y_ ) in a Taylor series about the _midpoint_ of the integration interval and retain two terms:

[8.33]

Because   raised to any odd power is equally positive and negative over the interval  , the integral of the   term in (8.32) vanishes and we obtain our algorithm:

[8.34]

[8.35]

We see that while `**rk2**` contains the same number of terms as Euler's rule, it obtains a higher level of precision by taking advantage of the cancelation of the  ( _h_ ) terms. Yet the price for improved precision is having to evaluate the derivative function and the solution _y_ at the middle of the time interval,  . And there is the rub, for we do not know the value of   and cannot use this algorithm to determine it. The way out of this quandary is to use Euler's algorithm to approximate  :

[8.36]

Putting the pieces all together gives the complete `**rk2**` algorithm:

[8.37]

[8.38]

where we use boldface italic to indicate the vector nature of _y_ and _f_. We see that the known derivative function **_f_** is evaluated at the ends and the midpoint of the interval, but that only the (known) initial value of the dependent variable **_y_** is required. This makes the algorithm self-starting.

As an example of the use of `**rk2**`, we apply it to our spring problem:

These equations say that the position _y_ (0) changes because of the initial velocity and force, while the velocity _y_ (1) changes because of the external force at _t_ = _h_ /2 and the internal force at two intermediate positions. We see that the position _y_ (0) now has an _h_ 2 time dependence, which at last brings us up to the level of first-year physics.

The fourth-order Runge–Kutta method `**rk4**` (Listing 8.1) obtains _ _( _h_ 4) precision by approximating _y_ as a Taylor series up to _h_ 2 (a parabola) at the midpoint of the interval, which again leads to cancellation of lower order error. `**rk4**` provides an excellent balance of power, precision, and programming simplicity. There are now four gradient ( _k_ ) terms to evaluate, with four subroutine calls. This provides an improved approximation to _f_ ( _t, y_ ) near the midpoint. Although `**rk4**` is computationally more expensive than the Euler method, its precision is much better, and some time is saved by using larger values for the step size _h._ Explicitly, `**rk4**` requires the evaluation of four intermediate slopes, and these are approximated with the Euler algorithm to (Press _et al.,_ 1994):

[8.39]

**Listing 8.1** **rk4.py** solves an ODE with the RHS given by the method _f_ ( ) using a fourth-order Runge–Kutta algorithm. Note that to avoid introducing bugs, the method _f_ ( ), which you will need to change for each problem, is kept separate from the algorithm.

A variation of `**rk4**`, known as the Runge–Kutta–Fehlberg method (Mathews, 2002), or `**rk45**`, varies the step size while doing the integration with the hope of obtaining better precision and maybe better speed. Our implementation, `**rk45.py**`, is given in Listing 8.2. It automatically doubles the step size and tests to see how an estimate of the error changes. If the error is still within acceptable bounds, the algorithm will continue to use the larger step size and thus speed up the computation; if the error is too large, the algorithm will decrease the step size until an acceptable error is found. As a consequence of the extra information obtained in the testing, the algorithm does obtains  ( _h_ 5) precision, but sometimes at the expense of extra computing time. Whether that extra time is recovered by being able to use a larger step size depends upon the application.

**Listing 8.2** **rk45.py** solves an ODE with the RHS given by the method _f_ ( ) using a fourth-order Runge–Kutta algorithm with adaptive step size.

## 8.7 Adams–Bashforth–Moulton Predictor–Corrector Rule

Another approach for obtaining high precision in an ODE algorithm uses the solution from previous steps _y_ _n_ –2 and _y_ _n_ –1 in addition to _y n,_ to predict _y_ _n_ +1. (The Euler and `**rk**` methods use just one previous step.) Many of these types of methods tend to be like a Newton's search method; we start with a guess or _prediction_ for the next step and then use an algorithm such as `**rk4**` to check on the prediction and thereby obtain a _correction._ As with `**rk45**`, one can use the correction as a measure of the error and then adjust the step size to obtain improved precision (Press _et al_., 1994). For those readers who may want to explore such methods, `**ABM.py**` in Listing 8.3 gives our implementation of the _Adams–Bashforth–Moulton_ predictor–corrector scheme.

**Listing 8.3** **ABM.py** solves an ODE with the RHS given by the method _f_ ( ) using the ABM predictor–corrector algorithm.

### 8.7.1 Assessment: rk2 vs. rk4 vs. rk45

While you are free to do as you please, unless you are very careful, we recommend that you _not_ write your own `**rk4**` or `**rk45**` methods. You will be using this code for some high-precision work, and unless you get every fraction and method call just right, your code may appear to work well but still not give all the precision that you should be obtaining, and therefore we give you `**rk4**`, and `**rk45**` codes to use. However, we do recommend that you write your own `**rk2**`, as doing so will make it clearer as to how the Runge–Kutta methods work, but without all the pain and danger.

  1. Write your own `**rk2**` method. Design your method for a general ODE; this means making the derivative function _f_ ( _t, x_ ) a separate method.
  2. Use your `**rk2**` solver in a program that solves the equation of motion (8.7) or (8.25). Plot both the position _x_ ( _t_ ) and velocity d _x_ /d _t_ as functions of time.
  3. Once your ODE solver is running, do a number of things to check that it is working well and that you know what _h_ values to use.
    1. a) Adjust the parameters in your potential so that it corresponds to a pure harmonic oscillator (set _p_ = 2 or _α_ = 0). For an oscillator initially at rest, we have an analytic result with which to compare:

[8.40]

    2. b) Pick values of _k_ and _m_ such that the period _T_ = 2π/ω is a nice number with which to work (something like _T_ = 1).
    3. c) Start with a step size _h_   _T_ /5 and make _h_ smaller until the solution looks smooth, has a period that remains constant for a large number of cycles, and agrees with the analytic result. As a general rule of thumb, we suggest that you start with _h_   _T_ /100, where _T_ is a characteristic time for the problem at hand. You should start with a large _h_ so that you can see a bad solution turn good.
    4. d) Make sure that you have exactly the same initial conditions for the analytic and numerical solutions (zero displacement, nonzero velocity) and then plot the two together. It is good if you cannot tell them apart, yet that only ensures that there are approximately two places of agreement.
    5. e) Try different initial velocities and verify that a _harmonic_ oscillator is _isochronous,_ that is, that its period does _not_ change as the amplitude varies.
  4. Now that you know you can get a good solution of an ODE with `**rk2**`, compare the solutions obtained with the `**rk2, rk4**`, and `**rk45**` solvers.
  5. Make a table of comparisons similar to Table 8.1. There we compare `**rk4**` and `**rk45**` for the two equations

[8.41]

[8.42]

with initial conditions ([ _y_ (0), _y_ '(0)] = (1,1). Although nonlinear, (8.41) does have the analytic solution _y_ ( _t_ ) = 1 + sin _t_4). Equation 8.42 corresponds to our standard potential (8.5), with _p_ = 6. Although we have not tuned `**rk45**`, Table 8.2 shows that by setting its tolerance parameter to a small enough number, `**rk45**` will obtain better precision than `**rk4**` (Figure 8.6), but that it requires ~ 10 times more floating-point operations and takes ~ 5 times longer. Yet for (8.41), we obtained increased precision in less time.

**Figure 8.6** The logarithm of the relative error in the solution of an ODE obtained with `**rk4**` using a differing number _N_ of time steps over a fixed time interval. The logarithm approximately equals the negative of the number of places of precision. Increasing the number of steps used for a fixed interval is seen to lead to smaller error.

**Table 8.1** Comparison of ODE solvers for different equations.

**Eq.No.** | **Method** | **Initial _h_** | **No. of flops** | **Time (ms)** | **Relative error**
---|---|---|---|---|---
(8.41) | `**rk4**` | 0.01 | 1000 | 5.2 | 2.2 × 10–8
| `**rk45**` | 1.00 | 72 | 1.5 | 1.8 × 10–8
(8.42) | `**rk4**` | 0.01 | 227 | 8.9 | 1.8 × 10–8
| `**rk45**` | 0.1 | 3143 | 36.7 | 5.7 × 10–11

## 8.8 Solution for Nonlinear Oscillations (Assessment)

Use your `**rk4**` program to study anharmonic oscillations by trying powers in the range _p_ = 2–12, or anharmonic strengths in the range 0 ≤ _ax_ ≤ 2. Do _not_ include any explicit time-dependent forces yet. Note that for large values of _p,_ the forces and accelerations get large near the turning points, and so you may need a smaller step size _h_ than that used for the harmonic oscillator.

  1. Check that the solution remains periodic with constant amplitude and period for a given initial condition regardless of how nonlinear you make the force. In particular, check that the maximum speed occurs at _x_ = 0 and that the velocity _v_ = 0 at maximum _x_ 's, the latter being a consequence of energy conservation.
  2. Verify that nonharmonic oscillators are _nonisochronous,_ that is, vibrations with different amplitudes have different periods (Figure 8.7).
  3. Explain why the shapes of the oscillations change for different _p_ 's or _α_ 's.
  4. Devise an algorithm to determine the period _T_ of the oscillation by recording times at which the mass passes through the origin. Note that because the motion may be asymmetric, you must record at least three times to deduce the period.

**Figure 8.7** The position vs. time for oscillations within the potential   for four different initial amplitudes. Each is seen to have a different period.

  5. Construct a graph of the deduced period as a function of initial amplitude.
  6. Verify that the motion is oscillatory, but not harmonic, as the energy approaches _k_ /6 _α_ 2 or for _p_ > 6.
  7. Verify that for the anharmonic oscillator with _E_ = _k_ /6 _α_ 2, the motion changes from oscillatory to translational. See how close you can get to the _separatrix_ where a single oscillation takes an infinite time. (There is no separatrix for the power-law potential.)

### 8.8.1 Precision Assessment: Energy Conservation

We have not explicitly built energy conservation into our ODE solvers. Nonetheless, unless you have explicitly included a frictional force, it follows mathematically from the form of the equations of motion that energy must be a constant for all values of _p_ or _α_. That being the case, the constancy of energy is a demanding test of the numerics.

  1. Plot the potential energy PE( _t_ ) = V[ _x_ ( _t_ )], the kinetic energy KE( _t_ ) = _mv_ 2( _t_ )/2, and the total energy _E_ ( _t_ ) = KE( _t_ ) + PE( _t_ ), for 50 periods. Comment on the correlation between PE( _t_ ) and KE( _t_ ) and how it depends on the potential's parameters.
  2. Check the long-term _stability_ of your solution by plotting

[8.43]

for a large number of periods (Figure 8.6). Because _E_ ( _t_ ) should be independent of time, the numerator is the absolute error in your solution, and when divided by _E_ (0), becomes the relative error (approximately 10–11). If you cannot achieve 11 or more places, then you need to decrease the value of _h_ or debug.
  3. Because a particle bound by a large- _p_ oscillator is essentially "free" most of the time, you should observe that the average of its kinetic energy over time exceeds its average potential energy. This is actually the physics behind the Virial theorem for a power-law potential (Marion and Thornton, 2003):

[8.44]

Verify that your solution satisfies the Virial theorem. (Those readers who have worked the perturbed oscillator problem can use this relation to deduce an effective _p_ value, which should be between 2 and 3.)

## 8.9 Extensions: Nonlinear Resonances, Beats, Friction

Problem So far our oscillations have been rather simple. We have ignored friction and have assumed that there are no external forces (hands) influencing the system's natural oscillations. Determine

  1. How the oscillations change when friction is included?
  2. How the resonances and beats of nonlinear oscillators differ from those of linear oscillators?
  3. How introducing friction affects resonances?

### 8.9.1 Friction (Model)

The world is full of friction, and not all of it is bad. For while friction makes it harder to pedal a bike through the wind, it also lets you walk on ice, and generally adds stability to dynamical systems. The simplest models for frictional force are called _static, kinetic,_ and _viscous_ friction:

[8.45]

Here _N_ is the _normal force_ on the object under consideration, _μ_ and _b_ are parameters, and _v_ is the velocity. This model for static friction is appropriate for objects at rest, while the model for kinetic friction is appropriate for an object sliding on a dry surface. If the surface is lubricated, or if the object is moving through a viscous medium, then a frictional force dependent on velocity is a better model.5)

  1. Extend your harmonic oscillator code to include the three types of friction in (8.45) and observe how the motion differs for each.
  2. _Hint:_ For the simulation with static plus kinetic friction, each time the oscillator has _v_ = 0 you need to check that the restoring force exceeds the static force of friction. If not, the oscillation must end at that instant. Check that your simulation terminates at nonzero _x_ values.
  3. For your simulations with viscous friction, investigate the qualitative changes that occur for increasing _b_ values:  Underdamped: | _b_ < 2 _mω_ 0 | Oscillate within decaying envelope
---|---|---
Critically: | _b_ = 2 _mω_ 0 | Nonoscillatory, finite decay time
Overdamped: | _b_ > 2 _mω_ 0 | Nonoscillatory, infinite decay time

### 8.9.2 Resonances and Beats: Model, Implementation

Stable physical systems will oscillate if displaced slightly from their rest positions. The frequency _ω 0_ with which such a system executes small oscillations about its rest positions is called its _natural frequency._ If an external sinusoidal force is applied to this system, and if the frequency of the external force equals the natural frequency _ω_ 0, then a _resonance_ may occur in which the oscillator absorbs energy from the external force and the amplitude of oscillation increases with time. If the oscillator and the driving force remain in phase over time, the amplitude of oscillation will increase continuously unless there is some mechanism, such as friction or nonlinearities, to limit the growth.

If the frequency of the driving force is close to, but not exactly equal to, the natural frequency of the oscillator, then a related phenomena, known as _beating,_ may occur. In this situation there is interference between the natural oscillation, which is independent of the driving force, and the oscillation resulting from the external force. If the frequency of the external driving force is very close to the natural frequency, then the resulting motion

[8.46]s

resembles the natural oscillation of the system at the average frequency ( _ω_ \+ _ω_ 0)/2, yet with an amplitude   that varies slowly with the _beat frequency_ ( _ω_ – _ω_ 0)/2.

## 8.10 Extension: Time-Dependent Forces

To extend our simulation to include an external force,

[8.47]

we need to include a time dependence in the force function _f_ ( _t_ , _y_ ) of our ODE solver.

  1. Add the sinusoidal time-dependent external force (8.47) to the space-dependent restoring force in your program (do not include friction yet).
  2. Start with a very large value for the magnitude of the driving force _F_ 0. This should lead to _mode locking_ (the 500-pound-gorilla effect), where the system is overwhelmed by the driving force and, after the transients die out, the system oscillates in phase with the driver regardless of the driver's frequency.
  3. Now lower _F_ 0 until it is close to the magnitude of the natural restoring force of the system. You need to have this near equality for beating to occur.
  4. Verify that for the harmonic oscillator, the beat frequency, that is, the number of variations in intensity per unit time, equals the frequency difference ( _ω_ – _ω_ 0)/2π in cycles per second, where _ω_   _ω_ 0.
  5. Once you have a value for _F_ 0 matched well with your system, make a series of runs in which you progressively increase the frequency of the driving force in the range  .
  6. Make of plot of the maximum amplitude of oscillation vs. the driver's _ω_.
  7. Explore what happens when you make a nonlinear system resonate. If the nonlinear system is close to being harmonic, you should get beating in place of the blowup that occurs for the linear system. Beating occurs because the natural frequency changes as the amplitude increases, and thus the natural and forced oscillations fall out of phase. Yet once out of phase, the external force stops feeding energy into the system, and so the amplitude decreases, and with the decrease in amplitude, the frequency of the oscillator returns to its natural frequency, the driver and oscillator get back in phase, and the entire cycle repeats.
  8. Investigate now how the inclusion of viscous friction modifies the curve of amplitude vs. driver frequency. You should find that friction broadens the curve.
  9. Explain how the character of the resonance changes as the exponent _p_ in the potential   is made larger and larger. At large _p_ , the mass effectively "hits" the wall and falls out of phase with the driver, and so the driver is less effective at pumping energy into the system.

1) In Chapter 15, we make a related study of the realistic pendulum and its chaotic behavior. Some special properties of nonlinear equations are also discussed in Chapter 24. 2) To avoid confusion, notice that _y_ ( _n_ ) is the _n_ th component of the _y_ vector, while _y n_ is the value of _y_ after _n_ time steps. Yes, there is a price to pay for elegance in notation. 3) Instability is often a problem when you integrate a _y_ ( _t_ ) that decreases as the integration proceeds, analogous to upward recursion of spherical Bessel functions. In this case, and if you have a linear ODE, you are best off integrating _inward_ from large times to small times and then scaling the answer to agree with the initial conditions. 4) Be warned, the `**rk**` procedures may be inaccurate for this equation if integrated exactly through the point _y_ ( _t_ ) = 0 because then terms in the equation proportional to _y_ vanish and this leaves _y_ '2 = 0, which is problematic. A different algorithm may be better there. 5) The effect of air resistance on projectile motion is studied in Section 9.6.

# 9
ODE Applications: Eigenvalues, Scattering, and Projectiles

* * *

Now that we know how to solve ODEs numerically, we use our newfound skills in some different ways. First, we combine our ODE solver with a search algorithm to solve the quantum eigenvalue problem. Then we solve some of the simultaneous ODEs that arise in the scattering problem, and explore classical chaotic scattering. Finally, we look upward to balls falling out of the sky and planets that do not.

* * *

## 9.1 Problem: Quantum Eigenvalues in Arbitrary Potential

Quantum mechanics describes phenomena that occur on atomic or subatomic scales (an elementary particle is subatomic). It is a statistical theory in which the probability that a particle is located in a region _dx_ around the point _x_ is  , where   is called the _wave function_. If a particle of definite energy _E_ moving in one dimension experiences a potential _V(x)_ , its wave function is determined by an ordinary differential equation (partial differential equation for more than one dimension) known as the time-independent Schrödinger equation1):

(9.1)

Although we say we are solving for the energy _E_ , in practice we solve for the wave vector _κ_ , where the two are related for bound states by

(9.2)

The Schrödinger equation now takes the form

(9.3)

When our problem tells us that the particle is bound, this means that it is confined to some finite region of space, which implies that _ψ_ ( _x_ ) is normalizeable. The only way for that to happen is if _ψ_ ( _x_ ) decay exponentially as   (where the potential vanishes):

(9.4)

In summary, although it is straightforward to solve the ODE (9.1) with the techniques we have learned so far, we must also require that the solution _ψ_ ( _x_ ) simultaneously satisfies the boundary conditions (9.4). This extra condition turns the ODE problem into an _eigenvalue problem_ that has solutions ( _eigenvalues_ ) for only certain values of the energy _E_ or equivalently _k_. The ground-state energy corresponds to the smallest (most negative) eigenvalue. The ground-state wave function (eigenfunction), which we must determine in order to find its energy, must be nodeless and even (symmetric) about _x = 0_. The excited states have higher (less negative) energies and wave functions that may be odd (antisymmetric).

### 9.1.1 Model: Nucleon in a Box

The numerical methods we describe are capable of handling the most realistic potential shapes. Yet to make a connection with the standard textbook case and to permit some analytic checking, we will use a simple model in which the potential _V_ ( _x_ ) in (9.1) is a finite square well (Figure 9.1):

(9.5)

where values of 83 MeV for the depth and 2 fm for the radius are typical for nuclei (these are the units in which we solve the problem). With this potential, the Schrödinger equation (9.3) becomes

(9.6)

(9.7)

To evaluate the ratio of constants here, we insert _c_ 2, the speed of light squared, into both the numerator and the denominator and then these familiar values (Landau, 1996):

(9.8)

**Figure 9.1** Computed wave function and the square-well potential (bold lines). The wave function computed by integration in from the left is matched to the one computed by integration in from the right (dashed curve) at a point near the right edge of the well. Note how the wave function decays rapidly outside the well.

## 9.2 Dual Algorithms: Eigenvalues via ODE Solver + Search

The solution of the eigenvalue problem combines the numerical solution of the ordinary differential equation (9.3) with a trial-and-error search for a wave function that satisfies the boundary conditions (9.4). This is carried out in several steps:2)

  1. Start on the very far _left_ at   where  . Because the potential _V_ = 0 in this region, the analytic solution here is e± _κx_. Accordingly, assume that the wave function there satisfies the left-hand boundary condition:

(9.9)

  2. Use your favorite ODE solver to step   in toward the origin (to the right) from   until you reach the _matching radius x_ match. The exact value of this matching radius is not important, and our final solution should be independent of it. In Figure 9.1, we show a sample solution with   that is, we match at the right edge of the potential well. In Figure 9.2 we see some guesses that do not match.
  3. Start at the extreme _right_ , that is, at   with a wave function that satisfies the right-hand boundary condition:

(9.10)

  4. Use your `**rk4**` ODE solver to step   in toward the origin (to the left) from   until you reach the _matching radius x_ match.
  5. In order for probability and current to be continuous at   and _ψ'(x)_ must be continuous there. Requiring the ratio   called the _logarithmic derivative_ , to be continuous there encapsulates both continuity conditions into a single condition and is independent of _ψ'_ s normalization.
  6. Although we do not know ahead of time which _E_ or _k_ values are eigenvalues, we still need a starting value for the energy in order to use our ODE solver. Such being the case, we start the solution with a guess for the energy. A good guess for ground-state energy would be a value somewhat up from that at the bottom of the well, _E_ > − _V_ 0.
  7. Because it is unlikely that any guess will be correct, the left- and right-wave functions will not quite match at   (Figure 9.2). This is okay because we can use the amount of mismatch to improve the next guess. We measure how well the right and left wave functions match by calculating the difference in logarithmic derivatives:

(9.11)

where the denominator is used to avoid overly large or small numbers. Next, we try a different energy, note how much Δ( _E_ ) has changed, and use this to deduce an intelligent guess at the next energy. The search continues until the left and right _ψ_ '/ _ψ_ match within some set tolerance that depends on the precision in energy desired.

**Figure 9.2** Two guesses for the energy that are either too low or too high to be an eigenvalue. We see that the low- _E_ guess does not oscillate enough to match onto a dying exponential, and that the high- _E_ guess oscillates too much.

### 9.2.1 Numerov Algorithm for Schrödinger ODE

We generally recommend the fourth-order Runge–Kutta method for solving ODEs, and its combination with a search routine for solving the eigenvalue problem. In this section we present the Numerov method, an algorithm that is specialized for ODEs not containing any first derivatives (such as our Schrödinger equation). While this algorithm is not as general as rk4, it is of   and thus speeds up the calculation by providing additional precision.

We start by rewriting the Schrödinger equation (9.3) in the generic form

(9.12)

where   for bound states. Observe that although (9.12) is specialized to a square well, other potentials would have a function _V_ ( _x_ ) in place of _−V 0_. The trick in the Numerov method is to get extra precision in the second derivative by taking advantage of there being no first derivative   in (9.12). We start with the Taylor expansions of the wave functions

(9.13)

(9.14)

where _ψ_ ( _n_ ) signifies the _n_ th derivative  . Because the expansion of _ψ_ ( _x − h_ ) has odd powers of _h_ appearing with negative signs, all odd powers cancel when we add _ψ_ ( _x_ \+ _h_ ) and _ψ_ ( _x_ − _h_ ) together:

(9.15)

(9.16)

To obtain an algorithm for the second derivative, we eliminate the fourth-derivative term by applying the operator   to the Schrödinger equation (9.12):

(9.17)

We eliminate the _ψ_ (4) terms by substituting the derived expression for the _ψ_ (2):

(9.18)

Now we use a central-difference approximation for the second derivative of  :

(9.19)

After this substitution, we obtain the Numerov algorithm:

(9.20)

We see that the Numerov algorithm uses the values of _ψ_ at the two previous steps _x_ and _x − h_ to move _ψ_ forward to _x_ \+ _h_. To step backward in _x_ , we need only to reverse the sign of _h._ Our implementation of this algorithm, `**Numerov.py**`, is given in Listing 9.1.

**Listing 9.1** **QuantumNumerov.py** solves the 1D time-independent Schrödinger equation for bound-state energies using a Numerov method (rk4 also works, as we show in Listing 9.2).

### 9.2.2 Implementation: Eigenvalues via ODE Solver + Bisection Algorithm

  1. Combine your bisection algorithm search program with your `**rk4**` or Numerov ODE solver program to create an eigenvalue solver. Start with a step size _h =_ 0.04.
  2. Write a function that calculates the matching function Δ( _E_ , _x_ ) as a function of energy and matching radius. This subroutine will be called by the bisection algorithm program to search for the energy at which Δ( _E,x =_ 2) (9.11) vanishes.
  3. As a first guess, take _E_   65 MeV.
  4. Search until Δ( _E_ , _x_ ) changes in only the fourth decimal place. We do this in the code `**QuantumEigen.py**` shown in Listing 9.2.
  5. Print out the value of the energy for each iteration. This will give you a feel as to how well the procedure converges, as well as a measure of the precision obtained. Try different values for the tolerance until you are confident that you are obtaining three good decimal places in the energy.
  6. Build in a limit to the number of energy iterations you permit, and print out a warning if the iteration scheme fails.
  7. As we have shown, plot the wave function and potential on the same graph (you will have to scale one plot to get both of them to fit).
  8. Deduce, by counting the number of nodes in the wave function, whether the solution found is a ground state (no nodes) or an excited state (with nodes) and whether the solution is even or odd about the origin (the ground state must be even).
  9. Include in your version of Figure 9.1 a horizontal line within the potential indicating the energy of the ground state relative to the potential's depth.
  10. Increase the value of the initial energy guess and search for excited states. Make sure to examine the wave function for each state found to establish that it is continuous, and to count the number of nodes to see if you have missed a state.
  11. Add each new state found as another horizontal bar within the potential.
  12. Verify that you have solved the _problem_ , that is, the spacing between levels is on the order of MeV for a nucleon bound in a several-fermi potential well.

**Listing 9.2** **QuantumEigen.py** solves the 1D time-independent Schrödinger equation for bound-state energies using the rk4 algorithm.

## 9.3 Explorations

  1. Check to see how well your search procedure works by using arbitrary values for the starting energy. For example, because no bound-state energies can lie below the bottom of the well, try _E ≥ —V 0,_ as well as some arbitrary fractions of _V 0_. In every case, examine the resulting wave function and check that it is both symmetric and continuous.
  2. Increase the depth of your potential progressively until you find several bound states. Look at the wave function in each case and correlate the number of nodes in the wave function with the position of the bound state in the well.
  3. Explore how a bound-state energy changes as you change the depth _V 0_ of the well. In particular, as you keep decreasing the depth, watch the eigenenergy move closer to _E = 0_ and see if you can find the potential depth at which the bound state has _E_   0.
  4. For a fixed well depth _V 0_, explore how the energy of a bound state changes as the well radius _a_ is varied. Larger radius should give increased binding.
  5.   Conduct some explorations in which you discover different combinations of ( _V 0_, _a_ ) that give the same ground-state energies (discrete ambiguities). The existence of several different combinations means that a knowledge of ground-state energy is not enough to determine a unique depth of the well.
  6. Modify the procedures to solve for the eigenvalues and eigenfunctions for odd wave functions.
  7. Solve for the wave function of a linear potential:

(9.21)

There is less potential here than for a square well, so you may expect smaller binding energies and a less confined wave function. (For this potential, there are no analytic results with which to compare.)

  8. Compare the results obtained, and the time the computer took to get them, using both the Numerov and `**rk4**` methods.
  9. _Newton–Raphson extension:_ Extend the eigenvalue search by using the Newton–Raphson method in place of the bisection algorithm. Determine how much faster it is.

## 9.4 Problem: Classical Chaotic Scattering

**Problem** One expects the classical scattering of a projectile from a barrier to be a continuous process. Yet it has been observed in experiments conducted on pinball machines (Figure 9.3) that for certain conditions the projectile undergoes multiple internal scatterings and ends up with a final trajectory that is apparently unrelated  to the initial one. Your _problem_ is to determine if this process can be modeled as scattering from a static potential, or if there must be active mechanisms built into the pinball machines that cause chaotic scattering.

**Figure 9.3** A photograph of a pinball machine in which multiple scatterings occur from the bumpers.

_Although this problem is easy to solve on the computer, the results have some chaotic features that are surprising (chaos is discussed further inChapter 14). In fact, the applet `**Disper2e.html**` (created by Jaime Zuluaga) that simulates this problem continues to be a source of wonderment for readers as well as authors._

### 9.4.1 Model and Theory

Our model for balls bouncing off the electrically activated bumpers in pinball machines is a point particle scattering from the stationary 2D potential (Blehel _et al_., 1990)

(9.22)

As seen in Figure 9.4, this potential has four circularly symmetric peaks in the _xy_ plane. The two signs correspond to repulsive and attractive potentials, respectively (the pinball machine contains only repulsive interactions). Because there are four peaks in this potential, we suspect that it may be possible to have multiple scatterings in which the projectile bounces back and forth between the peaks, somewhat as in a pinball machine.

The _theory_ for this problem is classical dynamics. Visualize a scattering experiment in which a projectile starting out at an infinite distance from a target with a definite velocity _v_ and an impact parameter _b_ (Figure 9.4) is incident on a target. After interacting with the target and moving a large distance from it, the scattered particle is observed at the scattering angle _θ_. Because the potential cannot recoil and thereby carry off energy, the speed of the projectile does not change,  but its direction does. An experiment typically measures the number of particles scattered and then converts this to a function, the differential cross section _σ_ ( _θ_ ), which is independent of the details of the experimental apparatus:

(9.23)

**Figure 9.4** Scattering from the potential   which in some ways models a pinball machine. The incident velocity _v_ is in the _y_ direction, and the impact parameter ( _x_ value) is _b_. The velocity of the scattered particle is _v'_ and its scattering angle is _θ_.

Here _N_ scatt( _θ_ ) is the number of particles per unit time scattered into the detector at angle _θ_ subtending a solid angle Δ _Ω_ , _N_ in is the number of particle per unit time incident on the target of cross-sectional area Δ _A_ in, and the limit in (9.23) is for infinitesimal detector and area sizes.

The definition (9.23) for the cross section is the one that experimentalists use to convert their measurements to a function that can be calculated by theory. We, as theorists, solve for the trajectory of a particle scattered from the potential (9.22) and from that deduce the scattering angle _θ_. Once we have the scattering angle, we predict the differential cross section from the dependence of the scattering angle upon the classical impact parameter _b_ (Marion and Thornton, 2003):

(9.24)

The surprise you should find in the simulation is that for certain potential parameters, d _θ_ /d _b_ can get to be very large or discontinuous, and this, accordingly, leads to large and discontinuous cross sections.

The dynamical equations to solve are just the vector form of Newton's law for the simultaneous _x_ and _y_ motions in the potential (9.22):

(9.25)

The equations for the _x_ and _y_ motions are simultaneous second-order ODEs:

(9.26)

(9.27)

Because the force vanishes at the peaks in Figure 9.4, these equations tell us that the peaks are at   and  . Substituting these values into the potential (9.22) yields  , which sets the energy scale for the problem.

### 9.4.2 Implementation

Although (9.26) and (9.27) are simultaneous second-order ODEs, we can still use our standard `**rk4**` ODE solver on them after expressing them in the standard form, only now the arrays will be 4D rather than the previous 2D:

(9.28)

(9.29)

(9.30)

where the order in which the _y (i)_s are assigned is arbitrary. With these definitions and equations (9.26) and (9.27), we can assign values for the force function:

(9.31)

(9.32)

(9.33)

(9.34)

(9.35)

To deduce the scattering angle from our simulation, we need to examine the trajectory of the scattered particle at a very large separation from the target. To approximate that, we wait until the scattered particle no longer feels the potential (say |PE|/KE ≤ 10−10) and call this infinity. The scattering angle is then deduced from the components of velocity,

(9.36)

Here `**atan2**` is a function that computes the arctangent in the correct quadrant without requiring any explicit divisions (that can blow up).

### 9.4.3 Assessment

  1. Apply the `**rk4**` method to solve the simultaneous second-order ODEs (9.26) and (9.27) with a 4D force function.
  2. The _initial conditions_ are (a) an incident particle with only a _y_ component of velocity and (b) an impact parameter _b_ (the initial _x_ value). You do not need to vary the initial _y_ , but it should be large enough such that PE/KE ≤ 10−10, which means that the KE   _E_.
  3. Good parameters to use are _m_ = 0.5, _v y(0)_ = 0.5, _v x(0)_ = 0.0, _Δ b_ = 0.05, − 1 ≤ _b_ ≤ 1. You may want to lower the energy and use a finer step size once you have found regions of rapid variation in the cross section.
  4. Plot a number of trajectories [ _x(t)_ , _y(t)_ ] that show usual and unusual behaviors. In particular, plot those for which backward scattering occurs, and consequently for which there is much multiple scattering.
  5. Plot a number of phase-space trajectories   and  . How do these differ from those of bound states?
  6. Determine the scattering angle `**_θ_ = atan2(Vx,Vy)**` by determining the velocity of the scattered particle after it has left the interaction region, that is, PE/KE≤ 10−10.
  7. Identify which characteristics of a trajectory lead to discontinuities in dθ/ _db_ and thus _σ (θ)_.
  8. Run the simulations for both attractive and repulsive potentials and for a range of energies less than and greater than _V max_ = exp(−2).
  9. _Time delay:_ Another way to find unusual behavior in scattering is to compute the _time delay T(b)_ as a function of the impact parameter _b_. The time delay is the increase in the time taken by a particle to travel through the interaction region. Look for highly oscillatory regions in the semilog plot of _T(b)_ , and once you find some, repeat the simulation at a finer scale by setting _b_   _b/10_ (the structures are fractals, see Chapter 16).

**Figure 9.5** The trajectories of a projectile fired with initial velocity _V_ 0 in the _θ_ direction. The lower curve includes air resistance.

## 9.5 Problem: Balls Falling Out of the Sky

Golf and baseball players claim that balls appear to fall straight down out of the sky at the end of their trajectories (the solid curve in Figure 9.5). Your _problem_ is to determine whether there is a simple physics explanation for this effect or whether it is "all in the mind's eye." And while you are wondering why things fall out of the sky, see if you can use your new-found numerical tools to explain why planets do not fall out of the sky.

## 9.6 Theory: Projectile Motion with Drag

Figure 9.5 shows the initial velocity _V_ 0 and inclination _θ_ for a projectile launched from the origin. If we ignore air resistance, the projectile has only the force of gravity acting on it and therefore has a constant acceleration _g_ = 9.8 m/s2 in the negative _y_ direction. The analytic solutions to the equations of motion are

(9.37)

(9.38)

where ( _V 0x_, _V 0y_) = _V_ 0(cos _θ_ , sin _θ_ ). Solvingfor _t_ as a function of _x_ and substituting it into the _y_ ( _t_ ) equation show that the trajectory is a parabola:

(9.39)

Likewise, it is easy to show (solid curve in Figure 9.5) that without friction the range   and the maximum height  .

The parabola of frictionless motion is symmetric about its midpoint and so does not describe a ball dropping out of the sky. Maybe air resistance will change that? The basic physics is Newton's second law in two dimensions for a frictional force _**F**_ (f) opposing motion, and a vertical gravitational force − _mgê y_:

(9.40)

(9.41)

where the boldface italic symbols indicate vector quantities.

The frictional force _**F**_ (f) is not a basic force of nature but rather a simple model of a complicated phenomenon. We know that friction always opposes motion, which means it is in the direction opposite to velocity. One model assumes that the frictional force is proportional to a power _n_ of the projectile's speed (Marion and Thornton, 2003; Warburton and Wang, 2004):

(9.42)

where the   factor ensures that the frictional force is always in a direction opposite that of the velocity. Physical measurements indicate that the power _n_ is noninteger and varies with velocity, and so a more accurate model would be a numerical one that uses the empirical velocity dependence _n_ ( _v_ ). With a constant power law for friction, the equations of motion are

(9.43)

We shall consider three values for _n_ , each of which represents a different model for the air resistance: (1) _n_ = 1 for low velocities; (2) _n_ = 3/2, for medium velocities; and (3) _n_ = 2 for high velocities.

### 9.6.1 Simultaneous Second-Order ODEs

Although (9.43) are simultaneous second-order ODEs, we can still use our regular ODE solver on them after expressing them in the standard form

(9.44)

(9.45)

We express the equations of motion in terms of _y_ to obtain the standard form:

(9.46)

(9.47)

And now we just read off the components of the force function _**f**_ ( _t_ , _**y**_ ):

(9.48)

Our implementation, `**ProjectiveAir.py**`, is given in Listing 9.3

**Listing 9.3** **ProjectileAir.py** solves for projectile motion with air resistance as well as analytically for the frictionless case.

### 9.6.2 Assessment

  1. Modify your `**rk4**` program so that it solves the simultaneous ODEs for projectile motion (9.43) with friction ( _n_ = 1).
  2. Check that you obtain graphs similar to those in Figure 9.5.
  3. The model (9.42) with _n_ = 1 is okay for low velocities. Now modify your program to handle _n_ = 3/2 (medium-velocity friction) and _n_ = 2 (high-velocity friction). Adjust the value of _k_ for the latter two cases such that the initial force of friction   is the same for all three cases.
  4. What is your conclusion about balls falling out of the sky?

## 9.7 Exercises: 2- and 3-Body Planet Orbits and Chaotic Weather

**Planets via Two of Newton's Laws** Newton's explanation of the motion of the planets in terms of a universal law of gravitation is one of the greatest achievements of science. He was able to prove that planets traveled in elliptical orbits with the sun at one vertex, and then go on to predict the periods of the motion. All Newton needed to postulate was that the force between a planet of mass _m_ and the sun of mass _M_ is

(9.49)

where _r_ is the planet–CM distance, _G_ is the universal gravitational constant, and the attractive force lies along the line connecting the planet and the sun (Figure 9.6a). Seeing that he had to invent calculus to do it, the hard part for Newton was solving the resulting differential equations. In contrast, the numerical solution is straightforward because even for planets the equation of motion is still

(9.50)

with the force (9.49) having Cartesian components (Figure 9.6

(9.51)

(9.52)

The equation of motion (9.50) becomes two simultaneous second-order ODEs:

(9.53)

  1. Assume units such that _GM_ = 1 and the initial conditions

(9.54)

  2. Modify your ODE solver program to solve (9.53).
  3. Establish that you use small enough time steps so that the orbits are closed and fall upon themselves.
  4. Experiment with the initial conditions until you find the ones that produce a circular orbit (a special case of an ellipse).
  5. Once you have obtained good precision, note the effect of progressively increasing the initial velocity until the orbits open up and the planets become projectiles.
  6. Using the same initial conditions that produced the ellipse, investigate the effect of the power in (9.49) being   with  . Even for small _α_ you should find that the ellipses now rotate or precess (Figure 9.6). (A small value for _α_ is predicted by general relativity.)

**Figure 9.6** (a) The gravitational force on a planet at distance _r_ from the sun. The _x_ and _y_ components of the force are indicated. (b) Output from the applet `Planet` showing the precession of a planet's orbit when the gravitational force ∝ 1/ _r_ 4 (successive orbits do not lie on top of each other).

**Three-Body Problem: The Discovery of Neptune** The planet Uranus was discovered in 1781 by William Herschel and found to have an orbital period of approximately 84 years. By the year 1846, Uranus had just about completed a full orbit around the sun since its discovery, but did not seem to be following precisely the positions predicted by Newton's law of gravity. However, theoretical calculations indicated that if there was a yet-to-be-discovered planet lying about 50% further away from the sun than Uranus, then its perturbation on the orbit of Uranus could explain the disagreement with Newton's law. The planet Neptune was thus discovered theoretically and confirmed experimentally. (If Pluto is discarded as just a dwarf planet, then Neptune is the most distant planet in the solar system.)

Assume that the orbits of Neptune and Uranus are circular and coplanar (as shown in Figure 9.7), and that the initial angular positions with respect to the _x_ -axis are as follows:

**Mass
(×10−5 Solar Masses)** | **Distance
(AU)** | **Orbital period
(Years)** | **Angular position
(in 1690)**
---|---|---|---
Uranus | 4.366 244 | 19.1914 | 84.0110 | ∼ 205.640
Neptune | 5.151 389 | 30.0611 | 164.7901 | ∼ 288.380

Using these data and rk4, find the variation in angular position of Uranus with respect to the Sun as a result of the influence of Neptune during one complete orbit of Neptune. Consider only the forces of the Sun and Neptune on Uranus. In the astronomical units, _M s_ = 1 and _G_ = 4π2.

You can do this calculation following the procedure outlined above in which the problems is reduced to simultaneous ODEs for the _x_ and _y_ Cartesian coordinates, and the components of the forces along _x_ and _y_ are computed. Another approach that you may want to try, computes the explicit values of the derivatives used in the rk4 method (<http://spiff.rit.edu/richmond/nbody/OrbitRungeKutta4_fixed.pdf>):

**Figure 9.7** A snapshot from the animated output of the code `**UranusNeptune.py**` (on Instructor's disk) showing: (a) the orbits of Uranus (inner circle) and of Neptune (outer circle) with the Sun in the center. The arrows indicate the Uranus–Neptune force that causes a perturbation in the orbits. (b) The perturbation in the angular position of Uranus as a result of the presence of Neptune.

(9.55)

(9.56)

(9.57)

(9.58)

(9.59)

(9.60)

with a similar set of **_k_** 's defined for Neptune because the force is equal and opposite.

To help you get started, here is a listing of some of the constants used in our program:

  1) The time-dependent equation requires the solution of a partial differential equation, as discussed in Chapter 22. 2) The procedure outlined here is for a general potential that falls off gradually. For a square well with sharp cutoffs, the analytic solution is valid right up to the walls, and we could start integrating inwards from there in this special case. In contrast, if we were working with a Coulomb potential, its very slow falloff would does not match onto a simple exponential, even at infinity (Landau, 1996).

# 10
High-Performance Hardware and Parallel Computers

* * *

This chapter discusses a number of topics associated with high-performance computing (HPC) and parallel computing. Although this may sound like something only specialists should be reading, using history as a guide, present HPC hardware and software will be desktop machines in less than a decade, and so you may as well learn these things now. We start with a discussion of a high-performance computer's memory and central processor design, and then examine various general aspects of parallel computing. Chapter 11 goes on to discuss some practical programming aspects of HPC and parallel computing. HPC is a broad subject, and our presentation is brief and given from a practitioner's point of view. The text (Quinn, 2004) surveys parallel computing and message passing interface from a computer science point of view. Other references on parallel computing include van de Velde (1994); Fox (1994), and Pancake (1996).

## 10.1 High-Performance Computers

By definition, supercomputers are the fastest and most powerful computers available, and at present, the term refers to machines with hundreds of thousands of processors. They are the superstars of the high-performance class of computers. Personal computers (PCs) small enough in size and cost to be used by an individual, yet powerful enough for advanced scientific and engineering applications, can also be high-performance computers. We define _high-performance computers_ as machines with a good balance among the following major elements:

  * Multistaged (pipelined) functional units,
  * Multiple central processing units (CPUs),
  * Multiple cores,
  * Fast central registers,
  * Very large, fast memories,
  * Very fast communication among functional units,
  * Vector, video, or array processors,
  * Software that integrates the above effectively and efficiently.

As the simplest example, it makes little sense to have a CPU of incredibly high speed coupled to a memory system and software that cannot keep up with it.

## 10.2 Memory Hierarchy

An idealized model of computer architecture is a CPU sequentially executing a stream of instructions and reading from a continuous block of memory. To illustrate, in Figure 10.1 we have a vector `**A[ ]**` and an array `**M[.. , ..]**` loaded in memory and about to be processed. The real world is more complicated than this. First, arrays are not stored in 2D blocks, but rather in the linear order. For instance, in Python, Java, and C it is in _row-major_ order:

[10.1]

In Fortran, it is in _column-major_ order:

[10.2]

Second, as illustrated in Figures 10.2 and 10.3, the values for the matrix elements may not even be in the same physical place. Some may be in RAM, some on the disk, some in cache, and some in the CPU. To give these words more meaning, in Figures 10.3 and 10.2 we show simple models of the memory architecture of a high-performance computer. This hierarchical arrangement arises from an effort to balance speed and cost, with fast, expensive memory supplemented by slow, less expensive memory. The memory architecture may include the following elements:

**CPU** Central processing unit, the fastest part of the computer. The CPU consists of a number of very high-speed memory units called _registers_ containing the _instructions_ sent to the hardware to do things like fetch, store, and operate on data. There are usually separate registers for instructions, addresses, and _operands_ (current data). In many cases, the CPU also contains some specialized parts for accelerating the processing of floating-point numbers.

**Cache** A small, very fast bit of memory that holds instructions, addresses, and data in their passage between the very fast CPU registers and the slower RAM (also called a high-speed buffer). This is seen in the next level down the pyramid in Figure 10.2. The main memory is also called _dynamic RAM_ (DRAM), while the cache is called _static RAM_ (SRAM). If the cache is used properly, it can greatly reduce the time that the CPU waits for data to be fetched from memory.

**Cache lines** The data transferred to and from the cache or CPU are grouped into cache or data lines. The time it takes to bring data from memory into the cache is called _latency._

**Figure 10.1** The logical arrangement of the CPU and memory showing a Fortran array _A_ ( _N_ ) and matrix **M** ( _N, N_ ) loaded into memory.

**Figure 10.2** Typical memory hierarchy for a single-processor, high-performance computer (B = bytes, k, M, G, T = kilo, mega, giga, tera).

**RAM** Random-access or central memory is in the middle of the memory hierarchy in Figure 10.2. RAM is fast because its addresses can be accessed directly in random order, and because no mechanical devices are needed to read it.

**Pages** Central memory is organized into pages, which are blocks of memory of fixed length. The operating system labels and organizes its memory pages much like we do the pages of a book; they are numbered and kept track of with a _table of contents._ Typical page sizes range from 4 to 16 kB, but on supercomputers they may be in the MB range.

**Hard disk** Finally, at the bottom of the memory pyramid is permanent storage on magnetic disks or optical devices. Although disks are very slow compared to RAM, they can store vast amounts of data and sometimes compensate for their slower speeds by using a cache of their own, the _paging storage controller._

**Virtual memory** True to its name, this is a part of memory you will not find in our figures because it is _virtual._ It acts like RAM but resides on the disk.

**Figure 10.3** The elements of a computer's memory architecture in the process of handling matrix storage.

When we speak of "fast" and "slow" memory we are using a time scale set by the clock in the CPU. To be specific, if your computer has a clock speed or cycle time of 1 ns, this means that it could perform a billion operations per second, if it could get its hands on the needed data quickly enough (typically, more than 10 cycles are needed to execute a single instruction). While it usually takes 1 cycle to transfer data from the cache to the CPU, the other types of memories are much slower. Consequently, you can speedup your program by having all needed data available for the CPU when it tries to execute your instructions; otherwise the CPU may drop your computation and go on to other chores while your data gets transferred from lower memory (we talk more about this soon in the discussion of pipelining or cache reuse). Compilers try to do this for you, but their success is affected by your programming style.

As shown in Figure 10.3, virtual memory permits your program to use more pages of memory than can physically fit into RAM at one time. A combination of operating system and hardware _maps_ this virtual memory into pages with typical lengths of 4–16 kB. Pages not currently in use are stored in the slower memory on the hard disk and brought into fast memory only when needed. The separate memory location for this switching is known as _swap space_ (Figure 10.4a).

Observe that when an application accesses the memory location for `**M[i,j]**`, the number of the page of memory holding this address is determined by the computer, and the location of `**M[i,j]**` within this page is also determined. A _page fault_ occurs if the needed page resides on disk rather than in RAM. In this case, the entire page must be read into memory while the least recently used page in RAM is swapped onto the disk. Thanks to virtual memory, it is possible to run programs on small computers that otherwise would require larger machines (or extensive reprogramming). The price you pay for virtual memory is an order-of-magnitude slowdown of your program's speed when virtual memory is actually invoked. But this may be cheap compared to the time you would have to spend to rewrite your program so it fits into RAM, or the money you would have to spend to buy enough RAM for your problem.

**Figure 10.4** (a) Multitasking of four programs in memory at one time. On a SISD computer the programs are executed in round robin order. (b) Four programs in the four separate memories of a MIMD computer.

Virtual memory also allows _multitasking,_ the simultaneous loading into memory of more programs than can physically fit into RAM (Figure 10.4b). Although the ensuing switching among applications uses computing cycles, by avoiding long waits while an application is loaded into memory, multitasking increases total throughout and permits an improved computing environment for users. For example, it is multitasking that permits a windowing system, such as Linux, Apple OS, or Windows, to provide us with multiple windows. Although each window application uses a fair amount of memory, only the single application currently receiving input must actually reside in memory; the rest are _paged out_ to disk. This explains why you may notice a slight delay when switching to an idle window; the pages for the now-active program are being placed into RAM, and the least used application still in memory is simultaneously being paged out.

## 10.3 The Central Processing Unit

How does the CPU get to be so fast? Often, it utilizes _prefetching_ and _pipelining;_ that is, it has the ability to prepare for the next instruction before the current one has finished. It is like an assembly line or a bucket brigade in which the person filling the buckets at one end of the line does not wait for each bucket to arrive at the other end before filling another bucket. In the same way, a processor fetches, reads, and decodes an instruction while another instruction is executing. Consequently, despite the fact that it may take more than one cycle to perform some operations, it is possible for data to be entering and leaving the CPU on each cycle. To illustrate, Table 10.1 indicates how the operation _c_ = ( _a_ \+ _b_ )/( _d_ × _f_ ) is handled. Here the pipelined arithmetic units A1 and A2 are simultaneously doing their jobs of fetching and operating on operands, yet arithmetic unit A3 must wait for the first two units to complete their tasks before it has something to do (during which time the other two sit idle).

**Table 10.1** Computation of _c_ = ( _a_ \+ _b_ )/( _d_ × _f_ ).

**Arithmetic Unit** | **Step 1** | **Step 2** | **Step 3** | **Step 4**
---|---|---|---|---
A1 | Fetch _a_ | Fetch _b_ | Add | —
A2 | Fetch _d_ | Fetch _f_ | Multiply | —
A3 | — | — | — | Divide

## 10.4 CPU Design: Reduced Instruction Set Processors

_Reduced instruction set computer_ (RISC) architecture (also called _superscalar_ ) is a design philosophy for CPUs developedfor high-performance computers and now used broadly. It increases the arithmetic speed of the CPU by decreasing the number of instructions the CPU must follow. To understand RISC, we contrast it with CISC ( _complex instruction set computer_ ) architecture. In the late 1970s, processor designers began to take advantage of _very-large-scale integration_ (VLSI), which allowed the placement of hundreds of thousands of elements on a single CPU chip. Much of the space on these early chips was dedicated to _microcode_ programs written by chip designers and containing machine language instructions that set the operating characteristics of the computer. There were more than 1000 instructions available, and many were similar to higher level programming languages like _Pascal_ and _Forth._ The price paid for the large number of complex instructions was slow speed, with a typical instruction taking more than 10 clock cycles. Furthermore, a 1975 study by Alexander and Wortman of the _XLP_ compiler of the IBM System/360 showed that about 30 low-level instructions accounted for 99% of the use with only 10 of these instructions accounting for 80% of the use.

The RISC philosophy is to have just a small number of instructions available at the chip level, but to have the regular programmer's high-level language, such as Fortran or C, translate them into efficient machine instructions for a particular computer's architecture. This simpler scheme is cheaper to design and produce, lets the processor run faster, and uses the space saved on the chip by cutting down on microcode to increase arithmetic power. Specifically, RISC increases the number of internal CPU registers, thus, making it possible to obtain longer pipelines (cache) for the data flow, a significantly lower probability of memory conflict, and some instruction-level parallelism.

The theory behind this philosophy for RISC design is the simple equation describing the execution time of a program:

[10.3]

Here "CPU time" is the time required by a program, "number of instructions" is the total number of machine-level instructions the program requires (sometimes called the _path length_ ), "cycles/instruction" is the number of CPU clock cycles each instruction requires, and "cycle time" is the actual time it takes for one CPU cycle. After thinking about (10.3), we can understand the CISC philosophy that tries to reduce CPU time by reducing the number of instructions, as well as the RISC philosophy, which tries to reduce the CPU time by reducing cycles/instruction (preferably to 1). For RISC to achieve an increase in performance requires a greater decrease in cycle time and cycles/instruction than is the increase in the number of instructions.

In summary, the elements of RISC are the following:

**Single-cycle execution,** for most machine-level instructions.

**Small instruction set,** of less than 100 instructions.

**Register-based instructions,** operating on values in registers, with memory access confined to loading from and storing to registers.

**Many registers,** usually more than 32.

**Pipelining,** concurrent preparation of several instructions that are then executed successively.

**High-level compilers,** to improve performance.

## 10.5 CPU Design: Multiple-Core Processors

The present time is seeing a rapid increase in the inclusion of multicore (up to 128) chips as the computational engine of computers, and we expect that number to keep rising. As seen in Figure 10.5, a dual-core chip has two CPUs in one integrated circuit with a shared interconnect and a shared level-2 cache. This type of configuration with two or more identical processors connected to a single shared main memory is called _symmetric multiprocessing,_ or SMP.

Although multicore chips were originally designed for game playing and single precision, they are finding use in scientific computing as new tools, algorithms, and programming methods are employed. These chips attain more integrated speed with less heat and more energy efficiency than single-core chips, whose heat generation limits them to clock speeds of less than 4 GHz. In contrast to multiple single-core chips, multicore chips use fewer transistors per CPU and are thus simpler to make and cooler to run.

Parallelism is built into a multicore chip because each core can run a different task. However, because the cores usually share the same communication channel and level-2 cache, there is the possibility of a communication bottleneck if both CPUs use the bus at the same time. Usually the user need not worry about this, but the writers of compilers and software must. Modern compilers automatically make use of the multiple cores, with MPI even treating each core as a separate processor.

**Figure 10.5** (a) A generic view of the Intel core-2 dual-core processor, with CPU-local level-1 caches and a shared, on-die level-2 cache (courtesy of D. Schmitz), (b) The AMD Athlon 64 X2 3600 dual-core CPU (Wikimedia Commons).

## 10.6 CPU Design: Vector Processors

Often the most demanding part of a scientific computation involves matrix operations. On a classic (von Neumann) scalar computer, the addition of two vectors of physical length 99 to form a third, ultimately requires 99 sequential additions (Table 10.2). There is actually much behind-the-scenes work here. For each element _i_ there is the _fetch_ of _a_ ( _i_ ) from its location in memory, the _fetch_ of _b_ ( _i_ ) from its location in memory, the _addition_ of the numerical values of these two elements in a CPU register, and the _storage_ in memory of the sum in _c_ ( _i_ ). This fetching uses up time and is wasteful in the sense that the computer is being told again and again to do the same thing.

When we speak of a computer doing _vector processing,_ we mean that there are hardware components that perform mathematical operations on entire rows or columns of matrices as opposed to individual elements. (This hardware can also handle single-subscripted matrices, that is, mathematical vectors.) In the vector processing of [ _A_ ] + [ _B_ ] = [ _C_ ], the successive fetching of and addition of the elements _A_ and _B_ are grouped together and overlaid, and _Z_   64–256 elements (the _section size)_ are processed with one command, as seen in Table 10.3. Depending on the array size, this method may speedup the processing of vectors by a factor of approximately 10. If all _Z_ elements were truly processed in the same step, then the speedup would be ~ 64-256.

Vector processing probably had its heyday during the time when computer manufacturers produced large mainframe computers designed for the scientific and military communities. These computers had proprietary hardware and software and were often so expensive that only corporate or military laboratories could afford them. While the Unix and then PC revolutions have nearly eliminated these large vector machines, some do exist, as well as PCs that use vector processing in their video cards. Who is to say what the future holds in store?

**Table 10.2** Computation of matrix [ _C_ ] = [ _A_ ] + [ _B_ ]-

**Step 1** | **Step 2** | **...** | **Step 99**
---|---|---|---
_c_ (1) = _a_ (1) + _b_ (1) | _c_ (2) = _a_ (2) + _b_ (2) |... | _c_ (99) = _a_ (99) + _b_ (99)

**Table 10.3** Vector processing of matrix [ _A_ ] + [ _B_ ] = [ _C_ ].

**Step 1** | **Step 2** | **...** | **Step _Z_**
---|---|---|---
_c_ (1) = _a_ (1) + _b_ (1) |  |  |
| _c_ (2) = _a_ (2) + _b_ (2) |  |
|  |... |
|  |  | _c_ ( _Z_ ) = _a_ ( _Z_ ) + _b_ ( _Z_ )

## 10.7 Introduction to Parallel Computing

There is a little question that advances in the hardware for parallel computing are impressive. Unfortunately, the software that accompanies the hardware often seems stuck in the 1960s. In our view, message passing and GPU programming have too many details for application scientists to worry about and (unfortunately) requires coding at an elementary level reminiscent of the early days of computing. However, the increasing occurrence of clusters in which the nodes are symmetric multiprocessors has led to the development of sophisticated compilers that follow simpler programming models; for example, _partitioned global address space_ compilers such as _CoArray Fortran, Unified Parallel C,_ and _Titanium._ In these approaches, the programmer views a global array of data and then manipulates these data as if they were contiguous. Of course, the data really are distributed, but the software takes care of that outside the programmer's view. Although such a program may make use of processors less efficiently than would a hand-coded program, it is a lot easier than redesigning your program. Whether it is worth _your_ time to make a program more efficient depends on the problem at hand, the number of times the program will be run, and the resources available for the task. In any case, if each node of the computer has a number of processors with a shared memory and there are a number of nodes, then some type of a hybrid programming model will be needed.

## 10.8 Parallel Semantics (Theory)

We saw earlier that many of the tasks undertaken by a high-performance computer are run in parallel by making use of internal structures such as pipelined and segmented CPUs, hierarchical memory, and separate I/O processors. While these tasks are run "in parallel," the modern use of _parallel computing_ or _parallelism_ denotes applying multiple processors to a single problem (Quinn, 2004). It is a computing environment in which some number of CPUs are running asynchronously and communicating with each other in order to exchange intermediate results and coordinate their activities.

For an instance, consider the matrix multiplication:

[10.4]

Mathematically, this equation makes no sense unless [ _A_ ] equals the identity matrix [ _I_ ]. However, it does make sense as an algorithm that produces new value of _B_ on the LHS in terms of old values of _B_ on the RHS:

[10.5]

[10.6]

Because the computation of   for specific values of _i_ and _j_ is independent of all other values of   and   can be computed in parallel, or each row or column of [ _B_ new] can be computed in parallel. If _B_ were not a matrix, then we could just calculate _B_ = _AB_ with no further ado. However, if we try to perform the calculation using just matrix elements of [ _B_ ] by replacing the old values with the new values as they are computed, then we must somehow establish that the _B k,j_ on the RHS of (10.6) are the values of [ _B_ ] that existed _before_ the matrix multiplication.

This is an example of _data dependency,_ in which the data elements used in the computation depend on the order in which they are used. A way to account for this dependency is to use a temporary matrix for [ _B_ new], and then to assign [ _B_ ] to the temporary matrix after all multiplications are complete:

[10.7]

[10.8]

In contrast, the matrix multiplication [ _C_ ] = [ _A_ ][ _B_ ] is a _data parallel_ operation in which the data can be used in any order. So already we see the importance of communication, synchronization, and understanding of the mathematics behind an algorithm for parallel computation.

The processors in a parallel computer are placed at the _nodes_ of a communication network. Each node may contain one CPU or a small number of CPUs, and the communication network may be internal to or external to the computer. One way of categorizing parallel computers is by the approach they utilize in handling instructions and data. From this viewpoint there are three types of machines:

**Single instruction, single data (SISD)** These are the classic (von Neumann) serial computers executing a single instruction on a single data stream before the next instruction and next data stream are encountered.

**Single instruction, multiple data (SIMD)** Here instructions are processed from a single stream, but the instructions act concurrently on multiple data elements. Generally, the nodes are simple and relatively slow but are large in number.

**Multiple instructions, multiple data (MIMD)** In this category, each processor runs independently of the others with independent instructions and data. These are the types of machines that utilize _message-passing_ packages, such as MPI, to communicate among processors. They may be a collection of PCs linked via a network, or more integrated machines with thousands of processors on internal boards, such as the Blue Gene computer described in Section 10.15. These computers, which do not have a shared memory space, are also called _multicomputers._ Although these types of computers are some of the most difficult to program, their low cost and effectiveness for certain classes of problems have led to their being the dominant type of parallel computer at present.

The running of independent programs on a parallel computer is similar to the multitasking feature used by Unix and PCs. In multitasking (Figure 10.4a), several independent programs reside in the computer's memory simultaneously and share the processing time in a round robin or priority order. On a SISD computer, only one program runs at a single time, but if other programs are in memory, then it does not take long to switch to them. In multiprocessing (Figure 10.4b), these jobs may all run at the same time, either in different parts of memory or in the memory of different computers. Clearly, multiprocessing becomes complicated if separate processors are operating on different parts of the _same_ program because then synchronization and load balance (keeping all the processors equally busy) are concerns.

In addition to instructions and data streams, another way of categorizing parallel computation is by _granularity_. A _grain_ is defined as a measure of the computational work to be performed, more specifically, the ratio of computation work to communication work.

**Coarse-grain parallel** Separate programs running on separate computer systems with the systems coupled via a conventional communication network. An illustration is six Linux PCs sharing the same files across a network but with a different central memory system for each PC. Each computer can be operating on a different, independent part of one problem at the same time.

**Medium-grain parallel** Several processors executing (possibly different) programs simultaneously while accessing a common memory. The processors are usually placed on a common _bus_ (communication channel) and communicate with each other through the memory system. Medium-grain programs have different, independent, _parallel subroutines_ running on different processors. Because the compilers are seldom smart enough to figure out which parts of the program to run where, the user must include the multitasking routines in the program.1)

**Fine-grain parallel** As the granularity decreases and the number of nodes increases, there is an increased requirement for fast communication among the nodes. For this reason, fine-grain systems tend to be custom-designed machines. The communication may be via a central bus or via shared memory for a small number of nodes, or through some form of high-speed network for massively parallel machines. In the latter case, the user typically divides the work via certain coding constructs, and the compiler just compiles the program. The program then runs concurrently on a user-specified number of nodes. For example, different for loops of a program may be run on different nodes.

## 10.9 Distributed Memory Programming

An approach to concurrent processing that, because it is built from commodity PCs, has gained dominant acceptance for coarse- and medium-grain systems is _distributed memory._ In it, each processor has its own memory and the processors exchange data among themselves through a high-speed switch and network. The data exchanged or _passed_ among processors have encoded _to_ and _from_ addresses and are called _messages._ The _clusters_ of PCs or workstations that constitute a _Beowulf_2) are examples of distributed memory computers (Figure 10.6). The unifying characteristic of a cluster is the integration of highly replicated compute and communication components into a single system, with each node still able to operate independently. In a Beowulf cluster, the components are commodity ones designed for a general market, as are the communication network and its high-speed switch (special interconnects are used by major commercial manufacturers, but they do not come cheaply). _Note:_ A group of computers connected by a network may also be called a cluster, but unless they are designed for parallel processing, with the same type of processor used repeatedly and with only a limited number of processors ( _the front end_ ) onto which users may log in, they are not usually called a Beowulf.

**Figure 10.6** Two views of parallel computing (courtesy of Yuefan Deng).

The literature contains frequent arguments concerning the differences among clusters, commodity clusters, Beowulfs, constellations, massively parallel systems, and so forth (Dongarra _et al_., 2005). Although, we recognize that there are major differences between the clusters on the top 500 list of computers and the ones that a university researcher may set up in his or her lab, we will not distinguish these fine points in the introductory materials we present here.

For a message-passing program to be successful, the data must be divided among nodes so that, at least for a while, each node has all the data it needs to run an independent subtask. When a program begins execution, data are sent to all the nodes. When all the nodes have completed their subtasks, they exchange data again in order for each node to have a complete new set of data to perform the next subtask. This repeated cycle of data exchange followed by processing continues until the full task is completed. Message-passing MIMD programs are also _single-program, multiple-data_ programs, which means that the programmer writes a single program that is executed on all the nodes. Often a separate host program, which starts the programs on the nodes, reads the input files and organizes the output.

## 10.10 Parallel Performance

Imagine a cafeteria line in which all the servers appear to be working hard and fast yet the ketchup dispenser has some relish partially blocking its output and so everyone in line must wait for the ketchup lovers up front to ruin their food before moving on. This is an example of the slowest step in a complex process determining the overall rate. An analogous situation holds for parallel processing, where the ketchup dispenser may be a relatively small part of the program that can be executed only as a series of serial steps. Because the computation cannot advance until these serial steps are completed, this small part of the program may end up being the bottleneck of the program.

As we soon will demonstrate, the speedup of a program will not be significant unless you can get ~90% of it to run in parallel, and even then most of the speedup will probably be obtained with only a small number of processors. This means that you need to have a computationally intense problem to make parallelization worthwhile, and that is one of the reasons why some proponents of parallel computers with thousands of processors suggest that you should not apply the new machines to old problems but rather look for new problems that are both big enough and well-suited for massively parallel processing to make the effort worthwhile.

The equation describing the effect on speedup of the balance between serial and parallel parts of a program is known as Amdahl's law (Amdahl, 1967; Quinn, 2004). Let

[10.9]

The maximum speedup _S p_ attainable with parallel processing is thus

[10.10]

In practice, this limit is never met for a number of reasons: some of the program is serial, data and memory conflicts occur, communication and synchronization of the processors take time, and it is rare to attain a perfect load balance among all the processors. For the moment, we ignore these complications and concentrate on how the _serial_ part of the code affects the speedup. Let _f_ be the fraction of the program that potentially may run on multiple processors. The fraction 1 – _f_ of the code that cannot be run in parallel must be run via serial processing and thus takes time:

[10.11]

The time _T p_ spent on the _p_ parallel processors is related to _T s_ by

[10.12]

That being so, the maximum speedup as a function of _f_ and the number of processors is

[10.13]

Some theoretical speedups are shown in Figure 10.7 for different numbers of processors _p_. Clearly the speedup will not be significant enough to be worth the trouble unless most of the code is run in parallel (this is where the 90% of the in-parallel figure comes from). Even an infinite number of processors cannot increase the speed of running the serial parts of the code, and so it runs at one processor speed. In practice, this means many problems are limited to a small number of processors, and that only 10–20% of the computer's peak performance is often all that is obtained for realistic applications.

**Figure 10.7** The theoretical maximum speedup of a program as a function of the fraction of the program that potentially may be run in parallel. The different curves correspond to different numbers of processors.

### 10.10.1 Communication Overhead

As discouraging as Amdahl's law may seem, it actually _overestimates_ speedup because it ignores the _overhead_ for parallel computation. Here we look at communication overhead. Assume a completely parallel code so that its speedup is

[10.14]

The denominator is based on the assumption that it takes no time for the processors to communicate. However, in reality it takes a finite time, called _latency,_ to get data out of memory and into the cache or onto the communication network. In addition, a communication channel also has a finite _bandwidth,_ that is, a maximum rate at which data can be transferred, and this too will increase the _communication time_ as large amounts of data are transferred. When we include communication time _T_ c, the speedup decreases to

[10.15]

For the speedup to be unaffected by communication time, we need to have

[10.16]

This means that as you keep increasing the number of processors _p_ , at some point the time spent on computation _T_ 1/ _p_ must equal the time _T_ c needed for communication, and adding more processors leads to greater execution time as the processors wait around more to communicate. This is another limit, then, on the maximum number of processors that may be used on any one problem, as well as on the effectiveness of increasing processor speed without a commensurate increase in communication speed.

The continual and dramatic increase in the number of processors being used in computations is leading to a changing view as to how to judge the speed of an algorithm. Specifically, the slowest step in a process is usually the rate-determining step, yet with the increasing availability of CPU power, the slowest step is more often the access to or communication among processors. Such being the case, while the number of computational steps is still important for determining an algorithm's speed, the number and amount of memory access and interprocessor communication must also be mixed into the formula. This is currently an active area of research in algorithm development.

## 10.11 Parallelization Strategies

A typical organization of a program containing both serial and parallel tasks is given in Table 10.4. The user organizes the work into units called _tasks,_ with each task assigning work ( _threads_ ) to a processor. The main task controls the overall execution as well as the subtasks that run independent parts of the program (called _parallel subroutines, slaves, guests,_ or _subtasks_ ). These parallel subroutines can be distinctive subprograms, multiple copies of the same subprogram, or even Python for loops.

It is the programmer's responsibility to establish that the breakup of a code into parallel subroutines is mathematically and scientifically valid and is an equivalent formulation of the original program. As a case in point, if the most intensive part of a program is the evaluation of a large Hamiltonian matrix, you may want to evaluate each row on a different processor. Consequently, the key to parallel programming is to identify the parts of the program that may benefit from parallel execution. To do that the programmer should understand the program's data structures (discussed soon), know in what order the steps in the computation must be performed, and know how to coordinate the results generated by different processors.

The programmer helps speedup the execution by keeping many processors simultaneously busy and by avoiding storage conflicts among different parallel subprograms. You do this _load balancing_ by dividing your program into subtasks of approximately equal numerical intensity that will run simultaneously on different processors. The rule of thumb is to make the task with the largest granularity (workload) dominant by forcing it to execute first and to keep all the processors busy by having the number of tasks an integer multiple of the number of processors. This is not always possible.

**Table 10.4** A typical organization of a program containing both serial and parallel tasks.

The individual parallel threads can have _shared_ or _local_ data. The shared data may be used by all the machines, while the local data are private to only one thread. To avoid storage conflicts, design your program so that parallel subtasks use data that are independent of the data in the main task and in other parallel tasks. This means that these data should not be modified _or even examined_ by different tasks simultaneously. In organizing these multiple tasks, reduce communication _overhead costs_ by limiting communication and synchronization. These costs tend to be high for fine-grain programming where much coordination is necessary. However, _do not_ eliminate communications that are necessary to ensure the scientific or mathematical validity of the results; bad science can do harm!

## 10.12 Practical Aspects of MIMD Message Passing

It makes sense to run only the most numerically intensive codes on parallel machines. Frequently, these are very large programs assembled over a number of years or decades by a number of people. It should come as no surprise, then, that the programming languages for parallel machines are primarily Fortran, which now has explicit structures for the compiler to parallelize, and 0 C. (In the past, we have not obtained good speedup with Java and MPI, yet _FastMPJ_ and _MPJ Express_ have fixed the problems.)

Effective parallel programming becomes more challenging as the number of processors increases. Computer scientists suggest that it is best _not_ to attempt to modify a serial code but instead rewrite one from scratch using algorithms and subroutine libraries best suited to parallel architecture. However, this may involve months or years of work, and surveys find that ~70% of computational scientists revise existing codes instead (Pancake, 1996).

Most parallel computations at present are performed on multiple instruction, multiple-data computers via message passing using MPI. Next we outline some practical concerns based on user experience (Dongarra _et al_., 2005; Pancake, 1996).

**Parallelism carries a price tag** There is a steep learning curve requiring intensive effort. Failures may occur for a variety of reasons, especially because parallel environments tend to change often and get "locked up" by a programming error. In addition, with multiple computers and multiple operating systems involved, the familiar techniques for debugging may not be effective.

**Preconditions for parallelism** If your program is run thousands of times between changes, with execution time in days, and you must significantly increase the resolution of the output or study more complex systems, then parallelism is worth considering. Otherwise, and to the extent of the difference, parallelizing a code may not be worth the time investment.

**The problem affects parallelism** You must analyze your problem in terms of how and when data are used, how much computation is required for each use, and the type of problem architecture.

**Perfectly parallel** This is when the same application is run simultaneously on different data sets, with the calculation for each data set independent (e.g., running multiple versions of a Monte Carlo simulation, each with different seeds, or analyzing data from independent detectors). In this case, it would be straightforward to parallelize with a respectable performance to be expected.

**Fully synchronous** The same operation applied in parallel to multiple parts of the same data set, with some waiting necessary (e.g., determining positions and velocities of particles simultaneously in a molecular dynamics simulation). Significant effort is required, and unless you balance the computational intensity, the speedup may not be worth the effort.

**Loosely synchronous** Different processors do small pieces of the computation but with intermittent data sharing (e.g., diffusion of groundwater from one location to another). In this case, it would be difficult to parallelize and probably not worth the effort.

**Pipeline parallel** Data from earlier steps processed by later steps, with some overlapping of processing possible (e.g., processing data into images and then into animations). Much work may be involved, and unless you balance the computational intensity, the speedup may not be worth the effort.

### 10.12.1 High-Level View of Message Passing

Although it is true that parallel computing programs may become very complicated, the basic ideas are quite simple. All you need is a regular programming language like Python, C, or Fortran, plus four communication statements:3)

  * `**send**`: One processor sends a message to the network.
  * `**receive**`: One processor receives a message from the network.
  * `**myid**`: An integer that uniquely identifies each processor.
  * `**numnodes**`: An integer giving the total number of nodes in the system.

Once you have made the decision to run your program on a computer cluster, you will have to learn the specifics of a message-passing system such as MPI. Here we give a broader view. When you write a message-passing program, you intersperse calls to the message-passing library with your regular Python, Fortran, or C program. The basic steps are as follows:

  1. Submit your job from the command line or a job control system.
  2. Have your job start additional processes.
  3. Have these processes exchange data and coordinate their activities.
  4. Collect these data and have the processes stop themselves.

**Figure 10.8** A master process and two slave processes passing messages. Notice how in this not-well-designed program there are more sends than receives, and consequently the results may depend upon the order of execution, or the program may even lock up.

We show this graphically in Figure 10.8 where at the top we see a _master_ process create two _slave_ processes and then assign work for them to do (arrows). The processes then communicate with each other via message passing, output their data to files, and finally terminate.

**What can go wrong** Figure 10.8 also illustrates some of the difficulties:

  * The programmer is responsible for getting the processes to cooperate and for dividing the work correctly.
  * The programmer is responsible for ensuring that the processes have the correct data to process and that the data are distributed equitably.
  * The commands are at a lower level than those of a compiled language, and this introduces more details for you to worry about.
  * Because multiple computers and multiple operating systems are involved, the user may not receive or understand the error messages produced.
  * It is possible for messages to be sent or received not in the planned order.
  * A _race condition_ may occur in which the program results depend upon the specific ordering of messages. There is no guarantee that slave 1 will get its work performed before slave 2, even though slave 1 may have started working earlier (Figure 10.8).
  * Note in Figure 10.8 how different processors must wait for signals from other processors; this is clearly a waste of time and has potential for deadlock.
  * Processes may _deadlock,_ that is, wait for a message that never arrives.

### 10.12.2 Message Passing Example and Exercise

Start with a simple serial program you have already written that is a good candidate for parallelization. Specifically, one that steps through parameter space in order to generate its results is a good candidate because you can have parallel tasks working on different regions of parameter space. Alternatively, a Monte Carlo calculation that repeats the same step many times is also a good candidate because you can run copies of the program on different processors, and then add up the results at the end. For example, Listing 10.1 is a serial calculation of π by Monte Carlo integration in the C language:

**Listing 10.1** **Serial calculation of π by Monte Carlo integration.**

Modify your serial program so that different processors are used and perform independently, and then have all their results combined. For example, Listing 10.2 is a parallel version of pi.c that uses the message passing interface (MPI). You may want to concentrate on the arithmetic commands and not the MPI ones at this point.

**Listing 10.2** **MPI.c:** Parallel calculation of π by Monte Carlo integration using MPI.

Although this small a problem is not worth your efforts in order to obtain a shorter run time, it is worth investing your time to gain some experience in parallel computing.

## 10.13 Scalability

A common discussion at HPC and Supercomputing conferences of the past heard application scientists get up, after hearing about the latest machine with what seemed like an incredible number of processors, and ask "But how can I use such a machine on my problem, which takes hours to run, but is not trivially parallel like your example?" The response from the computer scientist was often something like "You just need to think up some new problems that are more appropriate to the machines being built. Why use a supercomputer for a problem you can solve on a modern laptop?" It seems that these anecdotal exchanges have now been incorporated into the fabric of parallel computing under the title of _scalability._ In the most general sense, _scalability is defined as the ability to handle more work as the size of the computer or application grows._

As we have already indicated, the primary challenge of parallel computing is deciding how best to break up a problem into individual pieces that can each be computed separately. In an ideal world, a problem would _scale_ in a linear fashion, that is, the program would speedup by a factor of _N_ when it runs on a machine having _N_ nodes. (Of course, as _N_ → ∞ the proportionality cannot hold because communication time must at some point dominate). In present day terminology, this type of scaling is called _strong scaling,_ and refers to a situation in which the _problem size remains fixed_ while the number of number of nodes (the _machine scale_ ) increases. Clearly then, the goal when solving a problem that scales strongly is to decrease the amount of time it takes to solve the problem by using a more powerful computer. These are typically CPU-bound problems and are the hardest ones to yield something close to a linear speedup.

In contrast to strong scaling in which the problem size remains fixed, in _weak scaling_ we have applications of the type our CS colleagues referred to earlier; namely, ones in which we make the problem bigger and bigger as the number of processors increases. So here, we would have linear or perfect scaling if we could increase the size of the problem solved in proportion to the number _N_ of nodes.

To illustrate the difference between strong and weak scaling, consider Figure 10.9 (based on a lecture by Thomas Sterling). We see that for an application that scales perfectly strongly, the work carried out on each node decreases as the scale of the machine increases, which of course means that the time it takes to complete the problem decreases linearly. In contrast, we see that for an application that scales perfectly weakly, the work carried out by each node remains the same as the scale of the machine increases, which means that we are solving progressively larger problems in the same time as it takes to solve smaller ones on a smaller machine.

The concepts of weak and strong scaling are ideals that tend not to be achieved in practice, with real world applications being a mix of the two. Furthermore, it is the combination of application and computer architecture that determines the type of scaling that occurs. For example, shared memory systems and distributed-memory, message passing systems scale differently. Furthermore, a _data parallel_ application (one in which each node can work on its own separate data set) will by its very nature scale weakly.

**Figure 10.9** A graphical representation of weak vs. strong scaling. Weak scaling keeps each node doing the same amount of work as the problem is made bigger. Strong scaling has each node doing less work (running for less time) as the number of nodes is made bigger.

Before we go on and set you working on some examples of scaling, we should introduce a note of caution. Realistic applications tend to have various levels of complexity and so it may not be obvious just how to measure the increase in "size" of a problem. As an instance, it is known that the solution of a set of _N_ linear equations via Gaussian elimination requires   floating-point operations (flops). This means that doubling the number of equations does not make the "problem" twice as large, but rather eight times as large! Likewise, if we are solving partial differential equations on a 3D spatial grid and a 1D time grid, then the problem size would scale like _N_ 4. In this case, doubling the problem size would mean increasing _N_ by only 21/4   1.19.

### 10.13.1 Scalability Exercises

We have given above, and included in the Codes directory, a serial code pi.c that computes π/4 by Monte Carlo integration of a quarter of a unit circle. We have also given the code MPIpi.c that computes π by the same algorithm using MPI to compute the algorithm in parallel. Your exercise is to see how well this application scales. You can modify the codes we have given, or you can write your own.

  1. Determine the CPU time required to calculate π with the serial calculation using 1000 iterations (stone throws). Make sure that this is the actual run time and does not include any system time. (You can get this type of information, depending upon the operating system, by inserting timer calls in your program.)
  2. Get the MPI code running for the same number (1000) of iterations.
  3. First do some running that constitutes a _strong scaling test._ This means keeping the problem size constant, or in other words, keeping _N_ iter = 1000. Start by running the MPI code with only one processor doing the numerical computation. A comparison of this to the serial calculation gives you some idea of the overhead associated with MPI.
  4. Again keeping _N_ iter = 1000, run the MPI code for 2, 4, 8, and 16 computing nodes. In any case, make sure to go up to enough nodes so that the system no longer scales. Record the run time for each number of nodes.
  5. Make a plot of run time vs. number of nodes from the data you have collected.
  6. Strong scalability here would yield a straight line graph. Comment on your results.
  7. Now do some running that constitutes a _weak scaling test._ This means increasing the problem size simultaneously with the number of nodes being used. In the present case, increasing the number of iterations, _N_ iter.
  8. Run the MPI code for 2, 4, 8, and 16 computing nodes, with proportionally larger values for _N_ iter in each case (2000, 4000, 8000, 16 000, etc.). In any case, make sure to go up to enough nodes so that the system no longer scales.
  9. Record the run time for each number of nodes and make a plot of the run time vs. number of computing nodes.
  10. Weak scaling would imply that the run time remains constant as the problem size and the number of compute nodes increase in proportion. Comment on your results.
  11. Is this problem more appropriate for weak or strong scaling?

## 10.14 Data Parallelism and Domain Decomposition

As you have probably realized by this point, there are two basic, but quite different, approaches to creating a program that runs in parallel. In _task parallelism,_ you decompose your program by tasks, with differing tasks assigned to different processors, and with great care taken to maintain _load balance,_ that is, to keep all processors equally busy. Clearly you must understand the internal workings of your program in order to do this, and you must also have made an accurate _profile_ of your program so that you know how much time is being spent in various parts.

In _data parallelism,_ you decompose your program based on the data being created or acted upon, with differing data spaces ( _domains_ ) assigned to different processors. In data parallelism, there often must be data shared at the boundaries of the data spaces, and therefore synchronization among the data spaces. Data parallelism is the most common approach and is well suited to message-passing machines in which each node has its own private data space, although this may lead to a large amount of data transfer at times.

When planning how to decompose global data into subspaces suitable for parallel processing, it is important to divide the data into contiguous blocks in order to minimize the time spent on moving data through the different stages of memory (page faults). Some compilers and operating systems help you in this regard by exploiting _spatial locality,_ that is, by assuming that if you are using a data element at one location in data space, then it is likely that you may use some nearby ones as well, and so they too are made readily available. Some compilers and operating systems also exploit _temporal locality,_ that is, by assuming that if you are using a data element at one time, then there is an increased probability that you may use it again in the near future, and so it too is kept handy. You can help optimize your programs by taking advantage of these localities while programming.

As an example of _domain decomposition,_ consider the solution of a partial differential equation by a finite difference method. It is known from classical electrodynamics that the electric potential _U_ ( _x_ ) in a charge-free region of 2D space satisfies _Laplace's equation_ (fully discussed in Section 19.4):

[10.17]

We see that the potential depends simultaneously on _x_ and _y_ , which is what makes it a partial differential equation. The electric charges, which are the source of the field, enter indirectly by specifying the potential values on some boundaries or charged objects.

As shown in Figure 10.10, we look for a solution on a lattice ( _x, y_ ) values separated by finite difference Δ in each dimension and specified by discrete locations on the lattice:

[10.18]

When the finite difference expressions for the derivatives are substituted into (10.17), and the equation is rearranged, we obtain the finite-difference algorithm for the solution of Laplace's equation:

[10.19]

This equation says that when we have a proper solution, it will be the average of the potential at the four nearest neighbors (Figure 10.10). As an algorithm, (10.19) does not provide a direct solution to Laplace's equation but rather must be repeated many times to converge upon the solution. We start with an initial guess for the potential, improve it by sweeping through all space, taking the average over nearest neighbors at each node, and keep repeating the process until the solution no longer changes to some level of precision or until failure is evident. When converged, the initial guess is said to have _relaxed_ into the solution.

In Listing 10.3, we have a serial code `**laplace.c**` that solves the Laplace equation in two dimensions for a straight wire kept at 100 V in a grounded box, using the relaxation algorithm (10.19). There are five basic elements of the code:

  1. Initialize the potential values on the lattice.
  2. Provide an initial guess for the potential, in this case _U_ = 0 except for the wire at 100 V.
  3. Maintain the boundary values and the source term values of the potential at all times.
  4. Iterate the solution until convergence ((10.19) being satisfied to some level of precision) is obtained.
  5. Output the potential in a form appropriate for 3D plotting.

As you can see, the code is a simple pedagogical example with its essential structure being the array `**p[40][40]**` representing a (small) regular lattice. Industrial strength applications might use much larger lattices as well as adaptive meshes and hierarchical multigrid techniques.

**Figure 10.10** A representation of the lattice in a 2D rectangular space upon which Laplace's equation is solved using a finite difference approach. The lattice sites with white centers correspond to the boundary of the physical system, upon which boundary conditions must be imposed for a unique solution. The large circles in the middle of the lattice represent the algorithm used to solve Laplace's equation in which the potential at the point ( _x, y_ ) = ( _i, j_ )Δ is set to the average of the potential values at the four nearest-neighbor points.

When thinking of parallelizing this program, we note an algorithm being applied to a space of data points, in which case we can divide the domain into sub-spaces and assign each subspace to a different processor. This is domain decomposition or data parallelism. In essence, we have divided a large boundary-value problem into an equivalent set of smaller boundary-value problems that eventually get fit back together. Often extra storage is allocated on each processor to hold the data values that get communicated from neighboring processors. These storage locations are referred to as _ghost cells, ghost rows, ghost columns, halo cells,_ or _overlap areas._

Two points are essential in domain decomposition: (i) Load balancing is critical, and is obtained here by having each domain contain the same number of lattice points. (ii) Communication among the processors should be minimized because this is a slow process. Clearly the processors must communicate to agree on the potential values at the domain boundaries, except for those boundaries on the edge of the box that remain grounded at 0 V. But because there are many more lattice sites that need computing than there are boundaries, communication should not slow down the computation severely for large lattices.

To see an example of how this is carried out, the serial code `**poisson_1d.c**` solves Laplace's equation in 1D, and `**poisson_parallel_ld.c**` solves the same 1D equation in parallel (codes courtesy of Michel Vallieres). This code uses an accelerated version of the iteration algorithm using the parameter _Ω_ , a separate method for domain decomposition, as well as ghost cells to communicate the boundaries.

**Listing 10.3** **laplace.c Serial solution of Laplace's equation using a finite difference technique.**

### 10.14.1 Domain Decomposition Exercises

  1. Get the serial version of either laplace.c or laplace.f running.
  2. Increase the lattice size to 1000 and determine the CPU time required for convergence to six places. Make sure that this is the actual run time and does not include any system time. (You can get this type of information, depending upon the operating system, by inserting timer calls in your program.)
  3. Decompose the domain into four subdomains and get an MPI version of the code running on four compute nodes. [Recall, we give an example of how to do this in the Codes directory with the serial code poisson_1d.c and its MPI implementation, poisson_parallel_1d.c (courtesy of Michel Vallières).]
  4. Convert the serial code to three dimensions. This makes the application more realistic, but also more complicated. Describe the changes you have had to make.
  5. Decompose the 3D domain into four subdomains and get an MPI version of the code running on four compute nodes. This can be quite a bit more complicated than the 2D problem.
  6. Conduct a weak scaling test for the 2D or 3D code.
  7. Conduct a strong scaling test for the 2D or 3D code.

**Figure 10.11** The building blocks of Blue Gene (from Gara _et al._ (2005)).

## 10.15 Example: The IBM Blue Gene Supercomputers

Whatever figures we give to describe the latest supercomputer will be obsolete by the time you read them. Nevertheless, for the sake of completeness, and to set the present scale, we do it anyway. At the time of this writing, one of the fastest computers is the IBM Blue Gene/Q member of the Blue Gene series. In its largest version, its 96 racks contains 98 304 compute nodes with 1.6 million processor cores and 1.6 PB of memory (Gara _et al.,_ 2005). In June 2012, it reached a peak speed of 20.1 PFLOPS.

The name Blue Gene reflects the computer's origin in gene research, although Blue Genes are now general-purpose supercomputers. In many ways, these are computers built by committee, with compromises made in order to balance cost, cooling, computing speed, use of existing technologies, communication speed, and so forth. As a case in point, the compute chip has 18 cores, with 16 for computing, 1 for assisting the operating system with communication, and 1 as a redundant spare in case one of the others was damaged. Having communication on the chip reflects the importance of communication for distributed-memory computing (there are both on- and off-chip distributed memories). And while the CPU is fast with 204.8 GFLOPS at 1.6 GHz, there are faster ones made, but they would generate so much heat that it would not be possible to obtain the extreme scalability up to 98 304 compute nodes. So with the high-efficiency figure of 2.1 GFLOPS/watt, Blue Gene is considered a "green" computer.

We look more closely now at one of the original Blue Genes, for which we were able to obtain illuminating figures (Gara _et al_., 2005). Consider the building-block view in Figure 10.11. We see multiple cores on a chip, multiple chips on a card, multiple cards on a board, multiple boards in a cabinet, and multiple cabinets in an installation. Each processor runs a Linux operating system (imagine what the cost in both time and money would be for Windows!) and utilizes the hardware by running a distributed memory MPI with C, C++, and Fortran90 compilers.

**Figure 10.12** The single-node memory system (as presented by Gara _et al._ (2005)).

Blue Gene has three separate communication networks (Figure 10.13). At the heart of the network is a 3D torus that connects all the nodes, for example, Figure 10.13 a shows a sample torus of 2 × 2 × 2 nodes. The links are made by special link chips that also compute; they provide both direct neighbor–neighbor communications and cut-through communication across the network. The result of this sophisticated communications network is that there are approximately the same effective bandwidth and latencies (response times) between all nodes. However, the bandwidth may be affected by interference among messages, with the actual latency also depending on the number of _hops_ to get from one node to another.4) A rate of 1.4Gb/s is obtained for node-to-node communication. The collective network in Figure 10.13b is used for collective communications among processors, such as a _broadcast._ Finally, the control and gigabit ethernet network (Figure 10.13c) is used for I/O to communicate with the switch (the hardware communication center) and with ethernet devices.

The computing heart of Blue Gene is its integrated circuit and the associated memory system (Figure 10.12). This is essentially an entire computer system on a chip, with the exact specifications depending upon the model, and changing with time.

**Figure 10.13** (a) A 3D torus connecting 2 × 2 × 2 compute nodes. (b) The global collective memory system. (c) The control and Gb-ethernet memory system (from Gara _et al._ (2005)).

## 10.16 Exascale Computing via Multinode-Multicore GPUs

The current architecture of top-end supercomputers (Figure 10.14) uses a very large numbers of nodes, with each node containing a chip set that includes multiple cores as well as a graphical processing unit (GPU) attached to the chip set5). In the near future, we expect to see laptop computers capable of teraflops (1012 floating-point operations per second), deskside computers capable of petaflops, and supercomputers at the exascale, in terms of both flops and memory, probably with millions of nodes.

**Figure 10.14** A schematic of an exascale computer in which, in addition to each chip having multiple cores, a graphical processing unit is attached to each chip (adapted from Dongarra (2011)).

Look again at the schematic in Figure 10.14. As in Blue Gene, there really are large numbers of chip boards and large numbers of cabinets. Here we show just one node and one cabinet and not the full number of cores. The dashed line in Figure 10.14 represents communications, and it is seen to permeate all components of the computer. Indeed, communications have become such an essential part of modern supercomputers, which may contain 100s of 1000s of CPUs, that the network interface "card" may be directly on the chip board. Because a computer of this sort contains shared memory at the node level and distributed memory at the cabinet or higher levels, programming for the requisite data transfer among the multiple elements is a fundamental challenge, with significant new investments likely to occur (Dongarra _et al_., 2014).

1) Some experts define our medium grain as coarse grain yet this distinction changes with time. 2) Presumably there is an analogy between the heroic exploits of the son of Ecgtheow and the nephew of Hygelac in the 1000 C.E. poem _Beowulf_ and the adventures of us common folk assembling parallel computers from common elements that have surpassed the performance of major corporations and their proprietary, multimillion-dollar supercomputers. 3) Personal communication, Yuefan Deng. 4) The number of hops is the number of devices a given data packet passes through. 5) GPUs and their programming are discussed in Chapter 11.

# 11
Applied HPC: Optimization, Tuning, and GPU Programming

* * *

More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason – including blind stupidity.

_W.A. Wulf_

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

_Donald Knuth_

The best is the enemy of the good.

_Voltaire_

This chapter continues the discussion of high-performance computing (HPC) began in the previous chapter. Here we lead the reader through exercises that demonstrates some techniques for writing programs that are optimized for HPC hardware, and how the effects of optimization may be different in different computer languages. We then move on to tips for programming on a multicore computer. We conclude with an example of programming in compute unified device architecture (CUDA) on a graphics processing card, a subject beyond the general scope of this text, but one of high current interest.

* * *

## 11.1 General Program Optimization

**Rule 1** Do not do it.

**Rule 2** (for experts only): Do not do it yet.

**Rule 3** Do not optimize as you go: Write your program without regard to possible optimizations, concentrating instead on making sure that the code is clean, correct, and understandable. If it is too big or too slow when you have finished, then you can consider optimizing it.

**Rule 4** Remember the 80/20 rule: In many fields you can get 80% of the result with 20% of the effort (also called the 90/10 rule – it depends on who you talk to). Whenever you're about to optimize code, use profiling to find out where that 80% of execution time is going, so you know where to concentrate your effort.

**Rule** 5 Always run "before" and "after" benchmarks: How else will you know that your optimizations actually made a difference? If your optimized code turns out to be only slightly faster or smaller than the original version, undo your changes and go back to the original, clear code.

**Rule 6** Use the right algorithms and data structures: Do not use an  ( _n_ 2) DFT algorithm to do a Fourier transform of a 1000 elements when there is an  ( _n_ log _n_ ) FFT available. Similarly, do not store a thousand items in an array that requires an  ( _n_ ) search when you could use an  (log _n_ ) binary tree or an  (1) hash table (Hardwich, 1996).

The type of optimization often associated with _high-performance_ or _numerically intensive_ computing is one in which sections of a program are rewritten and reorganized in order to increase the program's speed. The overall value of doing this, especially as computers have become so fast and so available, is often a subject of controversy between computer scientists and computational scientists. Both camps agree that using the optimization options of compilers is a good idea. Yet on the one hand, CS tends to think that optimization is best left to the compilers, while computational scientists, who tend to run large codes with large amounts of data in order to solve real-world problems, often believe that you cannot rely on the compiler to do all the optimization.

### 11.1.1 Programming for Virtual Memory (Method)

While memory paging makes little appear big, you pay a price because your program's run time increases with each page fault. If your program does not fit into RAM all at once, it will be slowed down significantly. If virtual memory is shared among multiple programs that run simultaneously, they all cannot have the entire RAM at once, and so there will be memory access _conflicts_ which will decrease performance. The basic rules for programming for virtual memory are:

  1. Do not waste your time worrying about reducing the amount of memory used (the _working set size_ ) unless your program is large. In that case, take a global view of your entire program and optimize those parts that contain the largest arrays.
  2. Avoid page faults by organizing your programs to successively perform their calculations on subsets of data, each fitting completely into RAM.
  3. Avoid simultaneous calculations in the same program to avoid competition for memory and consequent page faults. Complete each major calculation before starting another.
  4. Group data elements close together in memory blocks if they are going to be used together in calculations.

### 11.1.2 Optimization Exercises

Many of the optimization techniques developed for Fortran and C are also relevant for Python applications. Yet while Python is a good language for scientific programming and is as universal and portable as Java, at present Python code runs slower than Fortran, C or even Java codes. In part, this is a consequence of the Fortran and C compilers having been around longer and thereby having been better refined to get the most out of a computer's hardware, and in part this is also a consequence of Python not being designed for speed. Because modern computers are so fast, whether a program takes 1s or 3s usually does not matter much, especially in comparison to the hours or days of _your_ time that it might take to modify a program for increased speed. However, you may want to convert the code to C (whose command structure is similar to that of Python) if you are running a computation that takes hours or days to complete and will be doing it many times.

Especially when asked to, compilers may look at your entire code as a single entity and rewrite it in an attempt to make it run faster. In particular, Fortran and C compilers often speed up programs by being careful in how they load arrays into memory. They also are careful in keeping the cache lines full so as not to keep the CPU waiting or having it move on to some other chore. That being said, in our experience compilers still cannot optimize a program as well as can a skilled and careful programmer who understands the order and structure of the code.

There is no fundamental reason why a program written in Java or Python cannot be compiled to produce a highly efficient code, and indeed such compilers are being developed and becoming available. However, such code is optimized for a particular computer architecture and so is not portable. In contrast, the byte code (` **.class**` file in Java and `**.pyc**` file in Python) produced by the compiler is designed to be interpreted or recompiled by the Java or Python _Virtual Machine_ (just another program). When you change from Unix to Windows, for example, the Virtual Machine program changes, but the byte code is the same. This is the essence of portability.

In order to improve the performance of Java and Python, many computers and browsers run _Just-in-Time_ (JIT) compilers. If a JIT is present, the Virtual Machine feeds your byte code `**Prog.class**` or `**Prog.pyc**` to the JIT so that it can be recompiled into native code explicitly tailored to the machine you are using. Although there is an extra step involved here, the total time it takes to run your program is usually 10–30 times faster with a JIT as compared to line-by-line interpretation. Because the JIT is an integral part of the Virtual Machine on each operating system, this usually happens automatically.

In the experiments below, you will investigate techniques to optimize both Fortran and Java or Python programs, and to compare the speeds of both languages for the same computation. If you run your program on a variety of machines you should also be able to compare the speed of one computer to that of another. Note that a knowledge of Fortran or C is not required for these exercises; if you keep an open mind you should be able to look at the code and figure out what changes may be needed.

#### 11.1.2.1 Good and Bad Virtual Memory Use (Experiment)

To see the effect of using virtual memory, convert these simple pseudocode examples (Listings 11.1 and 11.2) into actual code in your favorite language, and then run them on your computer. Use a command such as `**time, time.clock()**` or timeit to measure the time used for each example. These examples call functions `**force12**` and `**force21**`. You should write these functions and make them have significant memory requirements.

**Listing 11.1** BAD program, too simultaneous.

You see (Listing 11.1) that each iteration of the `**for**` loop requires the data and code for all the functions as well as access to all the elements of the matrices and arrays. The working set size of this calculation is the sum of the sizes of the arrays `**f12(N,N), f21(N,N),**` and `**pion(N)**` plus the sums of the sizes of the functions `**force12**` and `**force21**`.

A better way to perform the same calculation is to break it into separate components (Listing 11.2):

**Listing 11.2** GOOD program, separate loops.

Here, the separate calculations are independent and the _working set size,_ that is, the amount of memory used, is reduced. However, you do pay the additional overhead costs associated with creating extra `**for**` loops. Because the working set size of the first `**for**` loop is the sum of the sizes of the arrays `**f12(N, N)**` and `**pion(N)**`, and of the function `**force12**`, we have approximately half the previous size. The size of the last `**for**` loop is the sum of the sizes for the two arrays. The working set size of the entire program is larger than of the working set sizes `**for**` the different for loops.

As an example of the need to group data elements close together in memory or common blocks if they are going to be used together in calculations, consider the following code (Listing 11.3):

**Listing 11.3** BAD Program, discontinuous memory.

Here the variables `**zed, ylt,**` and `**part**` are used in the same calculations and are adjacent in memory because the programmer grouped them together in `**Common**` (global variables). Later, when the programmer realized that the array `**med2**` was needed, it was tacked onto the end of `**Common**`. All the data comprising the variables `**zed, ylt,**` and `**part**` fit onto one page, but the `**med2**` variable is on a different page because the large array `**zpart2(50000)**` separates it from the other variables. In fact, the system may be forced to make the entire 4kB page available in order to fetch the 72 B of data in `**med2**`. While it is difficult for a Fortran or C programmer to establish the placement of variables within page boundaries, you will improve your chances by grouping data elements together (Listing 11.4):

**Listing 11.4** GOOD program, continuous memory

## 11.2 Optimized Matrix Programming with NumPy

In Chapter 6, we demonstrated several ways of handling matrices with Python. In particular, we recommended using the `**array**` structure and the NumPy package. In this section, we extend that discussion somewhat by demonstrating two ways in which NumPy may speed up a program. The first is by using NumPy arrays rather than Python ones, and the second is by using Python slicing to reduce stride.

**Listing 11.5** **TuneNumPy.py** compares the time it takes to evaluate a function of each element of an array using a for loop, as well as using a vectorized call with NumPy. To see the effect of fluctuations, the comparison is repeated three times.

A powerful feature of NumPy is its high-level _vectorization._ This is the process in which the single call of a function operates not on a variable but on an array object as a whole. In the latter case, NumPy automatically _broadcasts_ the operation across all elements of the array with efficient use of memory. As we shall see, the resulting speed up can be more than an order of magnitude! While this may sound complicated, it really is quite simple because NumPy does this for you automatically.

For example, in Listing 11.5 we present the code `**TuneNumPy.py**`. It compares the speed of a calculation using a _for_ loop to evaluate a function for each of 100 000 elements in an array, vs. the speed using NumPy's vectorized evaluation of that function for an array object (Perez _et al.,_ 2010). And to see the effects of fluctuations as a result of things like background processes, we repeat the comparison three times. We obtained the following results:

Though a simple calculation, these results show that vectorization speeds the calculation up by a factor of nearly 50; really!

Now, recall from Chapter 10 our discussion of _stride_ (the amount of memory skipped in order to get to the next element needed in a calculation). It is important to have your program minimize stride in order not waste time jumping through memory, as well as not to load unneeded data into memory. For example, for a (1000, 1000) array, there is a passage of 1 word to get to the next column, but of 1000 words to get to the next row. Clearly better to do a column-by-column calculation than a row-by-row one.

We start our example by setting up a 3 × 3 array of integers using NumPy's `**arange**` to create a 1D array. We then reshape it into a 3 × 3 array and determine the strides of the matrix for rows and columns:

Line 11 tells us that it takes 12 bytes (three values) to get to the same position in the next row, but only 4 bytes (one value) to get to the same position in the next column. Clearly cheaper to go from column to column. Now, we show you an easy way to do that.

Recall Python's _slice_ operator that extracts just the desired part of a list (like taking a "slice" through the center of a jelly doughnut):

`**ListName[StartIndex:StopBeforeIndex:Step]**` .

The convention here is that if no argument is given, then the slice starts at 0 and stops at the end of the list. For example:

Once sliced, Python does not have to jump through memory to work with these elements, or even set up separate arrays for them. This is called _view-based indexing,_ with the indexed notation returning a new array object that _points_ to the address of the original data rather than store the values of the new array (think "pointers" in C). This does lead to improved speed, but you must remember that if you alter the new array, you are also altering the original array to which it points (think "pointers" in C).

For instance, you can optimize a finite difference calculation of forward- and central-difference derivatives quite elegantly (Perez _et al.,_ 2010):

(We note that the values of the derivatives look quite different, yet the forward difference is evaluated at the start of the interval and the central difference at the center.)

### 11.2.1 NumPy Optimization Exercises

  1. We have just demonstrated how NumPy's vectorized function evaluation can speed up a calculation by a factor of 50 via broadcast of an operation over an array rather than performing that operation on each individual element in the array. Determine the speedup for the matrix multiplication [A][B] where the matrices are at least 105 in size and contain floating-point numbers. Compare the direct multiplication to application of the elementary rule for each element:

(11.1)

  2. We have just demonstrated how Python's slice operator can be used to reduce the stride of a calculation of derivatives. Determine the speedup obtained in evaluating the forward-difference and central-difference derivatives over an array of at least 105 floating-point numbers using stripping to reduce stride. Compare to the same calculation without any stripping.

## 11.3 Empirical Performance of Hardware

In this section, you conduct an experiment in which you run a complete program in several languages and on as many computers as are available to you. In this way, you will explore how a computer's architecture and software affect a program's performance.

The first step in optimization is to try asking the compiler to optimize your program. You control how completely the compiler tries to do this by adding _optimization options_ to the compile command. For example, in Fortran (where this works better than in Python):

`**> f90 –O tune.f90**`

Here `**–O**` turns on optimization (` **O**` is the capital letter "oh," not zero). The actual optimization that is turned on differs from compiler to compiler. Fortran and C compilers have a bevy of such options and directives that let you truly customize the resulting compiled code. Sometimes optimization options make the code run faster, sometimes not, and sometimes the faster-running code gives the wrong answers (but does so quickly).

Because computational scientists may spend a good fraction of their time running compiled codes, the compiler options tend to become quite specialized. As a case in point, most compilers provide a number of levels of optimization for the compiler to attempt (there are no guarantees with these things). Although the speedup obtained depends upon the details of the program, higher levels may give greater speedup. However, we have had the experience of higher levels of optimization sometimes giving wrong answers (presumably this may be related to our programs not following the rules of grammar perfectly).

Some typical Fortran compiler options include the following:

  1. `**–O**` Use the default optimization level (` **–O3**`)
  2. `**–O1**` Provide minimum statement-level optimizations
  3. `**–O2**` Enable basic block-level optimizations
  4. `**–O3**` Add loop unrolling and global optimizations
  5. `**–O4**` Add automatic inlining of routines from the same source file
  6. `**–O5**` Attempt aggressive optimizations (with profile feedback).

The _gnu compilers_ gcc, g77, and g90 accept `**–O**` options as well as

  1. `**–malign–double**` Align doubles on 64-bit boundaries
  2. `**–ffloat–store**` For codes using IEEE-854 extended precision
  3. `**–fforce–mem, –fforce–addr**` Improves loop optimization
  4. `**–fno–inline**` Do not compile statement functions inline
  5. `**–nffast–math**` Try non-IEEE handling of floats
  6. `**–funsafe–math–optimizations**` Speeds up float operations; incorrect result possible
  7. `**–fno–trapping–math**` Assume no floating-point traps generated
  8. `**–fstrength–reduce**` Makes some loops faster
  9. `**–frerun–cse–after–loop**`
  10. `**–fexpensive–optimizations**`
  11. `**–fdelayed–branch**`
  12. `**–fschedule–insns**`
  13. `**–fschedule–insns2**`
  14. `**–fcaller–saves**`
  15. `**–funroll–loops**` Unrolls iterative DO loops
  16. `**–funroll–all–loops**` Unrolls DO WHILE loops

### 11.3.1 Racing Python vs. Fortran/C

The various versions of the program tune that are given in the Codes/HPC directory solve the matrix eigenvalue problem

(11.2)

for the eigenvalues _E_ and eigenvectors _c_ of a Hamiltonian matrix H. Here the individual Hamiltonian matrix elements are assigned the values

(11.3)

(11.4)

**Listing 11.6** **tune.f90** is meant to be numerically intensive enough to show the results of various types of optimizations, but you may have to increase sizes in it to make it more intensive. The programsolves the eigenvalue problem iteratively for a nearly diagonal Hamiltonian matrix using a variation of the power method.

Because the Hamiltonian is almost diagonal, the eigenvalues should be close to the values of the diagonal elements, and the eigenvectors should be close to a set of _N-_ dimensional unit vectors. For example, let us say that H has dimensions of _N_ × _N_ with _N_ = 2000. The number of elements in the matrix is then 2000 × 2000 = 4 000 000, and so it will take 4 million × 8B = 32 MB to store this many double precision numbers. Because modern PCs have 4GB or more of RAM, this small a matrix should not have memory issues. Accordingly, _determine the size of RAM on your computer and increase the dimension of the_ H _matrix until it surpasses that size._ (On Windows, this will be indicated as one of the "Properties" of "Computer" or in the information about "System" in the Control Panel.)

We find the solution to (11.2) via a variation of the _power_ or _Davidson method._ We start with an arbitrary first guess for the eigenvector _c_ and use it to calculate the energy corresponding to this eigenvector,1)

(11.5)

where   is the row vector adjoint of c0. Because **H** is nearly diagonal with diagonal elements that increase as we move along the diagonal, this guess should be close to the eigenvector with the smallest eigenvalue. The heart of the algorithm is the guess that an improved eigenvector has the _k_ th component

(11.6)

where _k_ ranges over the length of the eigenvector. If repeated, this method converges to the eigenvector with the smallest eigenvalue. It will be the smallest eigenvalue because it gets the largest weight (smallest denominator) in (11.6) each time. For the present case, six places of precision in the eigenvalue are usually obtained after 11 iterations. Here are the steps to follow:

  1. Vary the value of `**err**` in tune that controls precision and note how it affects the number of iterations required.
  2. Try some variations on the initial guess for the eigenvector (11.6) and see if you can get the algorithm to converge to some of the other eigenvalues.
  3. Keep a table of your execution times vs. technique.
  4. Compile and execute `**tune.f90**` and record the run time (Listing 11.6). On Unix systems, the compiled program will be placed in the file `**a.out**`. From a Unix shell, the compilation, timing, and execution can all be carried out with the commands

Here the compiled Fortran program is given the (default) name of `**a.out**`, and the `**time**` command gives you the execution (` **user**`) time and `**system**` time in seconds to execute `**a.out**`.

  5. As indicated in Section 11.3, you can ask the compiler to produce a version of your program optimized for speed by including the appropriate compiler option:

`**> f90 –O tune.f90**`

Execute and time the optimized code, checking that it still gives the same answer, and note any speedup in your journal.

  6. Try out optimization options up to the highest levels and note the run time and accuracy obtained. Usually `**–O3**` is pretty good, especially for as simple a program as tune with only a main method. With only one program unit, we would not expect `**–O4**` or `**–O5**` to be an improvement over `**–O3**`. However, we do expect `**–O3**`, with its loop unrolling, to be an improvement over `**–O2**`.
  7. The program `**tune4**` does some _loop unrolling_ (we will explore that soon). To see the best we can do with Fortran, record the time for the most optimized version of `**tune4.f95**`.
  8. The program `**Tune.py**` in Listing 11.7 is the Python equivalent of the Fortran program `**tune.f90**`.
  9. To get an idea of what Tune.py does (and give you a feel for how hard life is for the poor computer), assume ldim `**=2**` and work through one iteration of `**Tune**` _by hand._ Assume that the iteration loop has converged, follow the code to completion, and write down the values assigned to the variables.
  10. Compile and execute `**Tune.py**`. You do not have to issue the `**time**` command because we built a timer into the Python program. Check that you still get the same answer as you did with Fortran and note how much longer it takes with Python. You might be surprised how much slower Python is than Fortran.
  11. We now want to perform a little experiment in which we see what happens to performance as we fill up the computer's memory. In order for this experiment to be reliable, it is best for you _not_ to be sharing the computer with any other users. On Unix systems, the `**who –a**` command shows you the other users (we leave it up to you to figure out how to negotiate with them).
  12. To get some idea of what aspect of our little program is making it so slow, compile and run `**Tune.py**` for the series of matrix sizes `**ldim**` = 10, 100, 250, 500, 750, 1025, 2500, and 3000. You may get an error message that Python is out of memory at 3000. This is because you have not turned on the use of virtual memory.

**Listing 11.7** **Tune.py** is meant to be numerically intensive enough to show the results of various types of optimizations, but you may need to increase sizes to make it more intensive. The program solves the eigenvalue problem iteratively for a nearly diagonal Hamiltonian matrix using a variation of the power method.

  13. Make a graph of run time vs.matrix size. It should be similar to Figure 11.1, although if there is more than one user on your computer while you run, you may get erratic results. Note that as our matrix becomes larger than ~ 1000 × 1000 in size, the curve sharply increases in slope with execution time, in our case increasing like the _third_ power of the dimension. Because the number of elements to compute increases as the _second_ power of the dimension, something else is happening here. It is a good guess that the additional slowdown is as a result of page faults in accessing memory. In particular, accessing 2D arrays, with their elements scattered all through memory, can be very slow.

**Figure 11.1** Running time `**vs.**` dimension for an eigenvalue search using `**Tune.java**` and `**tune.f90**`.

**Listing 11.8** **Tune4.py** does some loop unrolling by explicitly writing out two steps of a for loop `**(steps of 2)**`. This results in better memory access and faster execution.

**Listing 11.9 tune4.f95** does some loop unrolling by explicitly writing out two steps of a Do loop `**(steps of 2)**`. This results in better memory access and faster execution.

  14. Repeat the previous experiment with `**tune.f90**` that gauges the effect of increasing the `**ham**` matrix size, only now do it for `**ldim**` = 10, 100, 250, 500, 1025, 3000, 4000, 6000,... You should get a graph like ours. Although our implementation of Fortran has automatic virtual memory, its use will be exceedingly slow, especially for this problem (possibly a 50-fold increase in time!). So if you submit your program and you get nothing on the screen (though you can hear the disk spin or see it flash busy), then you are probably in the virtual memory regime. If you can, let the program run for one or two iterations, kill it, and then scale your run time to the time it would have taken for a full computation.
  15. To test our hypothesis that the access of the elements in our 2D array ham[i,j] is slowing down the program, we have modified `**Tune.py**` into `**Tune4.py**` in Listing 11.8 (and similar modification with the Fortran versions).
  16. Look at `**Tune4**` and note where the nested `**for**` loop over `**i**` and `**j**` now takes step of Δ _i_ = 2 rather the unit steps in `**Tune.py**`. If things work as expected, the better memory access of `**Tune4.py**` should cut the run time nearly in half. Compile and execute `**Tune4.py**`. Record the answer in your table.
  17. In order to cut the number of calls to the 2D array in half, we employed a technique known as _loop unrolling_ in which we explicitly wrote out some of the lines of code that otherwise would be executed implicitly as the `**for**` loop went through all the values for its counters. This is not as clear a piece of code as before, but it evidently permits the compiler to produce a faster executable. To check that `**Tune**` and `**Tune4**` actually do the same thing, assume `**ldim = 4**` and run through one iteration of `**Tune4**` _by hand._ Hand in your manual trial.

## 11.4 Programming for the Data Cache (Method)

Data caches are small, very fast memory banks used as temporary storage between the ultrafast CPU registers and the fast main memory. They have grown in importance as high-performance computers have become more prevalent. For systems that use a data cache, this may well be the single most important programming consideration; continually referencing data that are not in the cache _(cache misses)_ may lead to an order-of-magnitude increase in CPU time.

**Figure 11.2** The cache manager's view of RAM. Each 128-B cache line is read into one of four lines in cache.

As indicated in Figures 10.3 and 11.2, the data cache holds a copy of some of the data in memory. The basics are the same for all caches, but the sizes are manufacturer dependent. When the CPU tries to address a memory location, the _cache manager_ checks to see if the data are in the cache. If they are not, the manager reads the data from memory into the cache, and then the CPU deals with the data directly in the cache. The cache manager's view of RAM is shown in Figure 11.2.

When considering how a matrix operation uses memory, it is important to consider the _stride_ of that operation, that is, the number of array elements that are stepped through as the operation repeats. For instance, summing the diagonal elements of a matrix to form the trace

(11.7)

involves a large stride because the diagonal elements are stored far apart for large _N._ However, the sum

(11.8)

has stride 1 because adjacent elements of _x_ are involved. Following is the basic rule in programming for a cache:

  * Keep the stride low, preferably at 1, which in practice means:
    * – Vary the leftmost index first on Fortran arrays.
    * – Vary the rightmost index first on Python and C arrays.

### 11.4.1 Exercise 1: Cache Misses

We have said a number of times that your program will be slowed down if the data it needs are in virtual memory and not in RAM. Likewise, your program will also be slowed down if the data required by the CPU are not in the cache. For high-performance computing, you should write programs that keep as much of the data being processed as possible in the cache. To do this, you should recall that Fortran matrices are stored in successive memory locations with the row index varying most rapidly (column-major order), while Python and C matrices are stored in successive memory locations with the column index varying most rapidly (row-major order). While it is difficult to isolate the effects of the cache from other elements of the computer's architecture, you should now estimate its importance by comparing the time it takes to step through the matrix elements row by row to the time it takes to step through the matrix elements column by column.

Run on machines available to you a version of each of the two simple codes given in Listings 11.10 and 11.11. Check that although each has the same number of arithmetic operations, one takes significantly more time to execute because it makes large jumps through memory, with some of the memory locations addressed not yet read into the cache:

**Listing 11.10** Sequential column and row references.

**Listing 11.11** Sequential column and row references.

### 11.4.2 Exercise 2: Cache Flow

Below in Listings 11.12 and 11.13, we give two simple code fragments that you should place into full programs in whatever computer language you are using. Test the importance of cache flow on your machine by comparing the time it takes to run these two programs. Run for increasing column size `**idim**` and compare the times for loop _A_ vs. those for loop _B._ A computer with very small caches may be most sensitive to stride.

**Listing 11.12** Loop A: GOOD f90 (min stride), BAD Python/C (max stride).

**Listing 11.13** Loop B: BAD f90 (max stride), GOOD Python/C (min stride).

Loop _A_ steps through the matrix `**Vec**` in column order. Loop _B_ steps through in row order. By changing the size of the columns (the leftmost Python index), we change the step size ( _stride_ ) taken through memory. Both loops take us through all the elements of the matrix, but the stride is different. By increasing the stride in any language, we use fewer elements already present in the cache, require additional swapping and loading of the cache, and thereby slow down the whole process.

### 11.4.3 Exercise 3: Large-Matrix Multiplication

As you increase the dimensions of the arrays in your program, memory use increases geometrically, and at some point you should be concerned about efficient memory use. The penultimate example of memory usage is large-matrix multiplication:

(11.9)

(11.10)

**Listing 11.14** BAD f90 (max stride), GOOD Python/C (min stride).

This involves all the concerns with different kinds of memory. The natural way to code (11.9) follows from the definition of matrix multiplication (11.10), that is, as a sum over a row of **A** times a column of **B**. Try out the two codes in Listings 11.14 and 11.15 on your computer. In Fortran, the first code uses matrix **B** with stride 1, but matrix **C** with stride _N._ This is corrected in the second code by performing the initialization in another loop. In Python and C, the problems are reversed. On one of our machines, we found a factor of 100 difference in CPU times despite the fact that the number of operations is the same!

**Listing 11.15** GOOD f90 (min stride), BAD Python/C (max stride).

## 11.5 Graphical Processing Units for High Performance Computing

In Section 10.16, we discussed how the trend toward exascale computing appears to be one using multinode-multicore-GPU computers, as illustrated in Figure 10.14. The GPU component in this figure extends a supercomputer's architecture beyond that of computers such as IBM Blue Gene. The GPUs in these future supercomputers are electronic devices designed to accelerate the creation of visual images. A GPU's efficiency arises from its ability to create many different parts of an image in parallel, an important ability because there are millions of pixels to display simultaneously. Indeed, these units can process 100s of millions of polygons in a second. Because GPUs are designed to assist the video processing on commodity devices such as personal computers, game machines, and mobile phones, they have become inexpensive, high performance, parallel computers in their own right. Yet because GPUs are designed to assist in video processing, their architecture and their programming are different from that of the general purpose CPUs usually used for scientific applications, and it takes some work to use them for scientific computing.

Programming of GPUs requires specialized tools specific to the GPU architecture being used, and while we do discuss them in Section 11.6.2, their low-level details places it beyond the normal level of this book. What is often called "compute unified device architecture (CUDA) programming" refers to programming for the architecture developed by the Nvidia corporation, and is probably the most popular approach to GPU programming at present (Zeller, 2008). However, some of the complications are being reduced via extensions and wrappers developed for popular programming languages such as C, Fortran, Python, Java, and Perl. However, the general principles involved are just an extension of those used already discussed, and after we have you work through some examples of those general principles, in Section 11.6 we give some practical tips for programming multinode-multicore-GPU computers.

In Chapter 10, we discussed some high-level aspects regarding parallel computing. In this section, we discuss some practical tips for multicore, GPU Programming. In the next section, we provide some actual examples of GPU programming using Nvidias _CUDA_ language, with access to it from within Python programs via the use of _PyCUDA_ (Klöckner, 2014). But do not think this is nothing more than adding in another package to Python. That section is marked as optional because using CUDA goes beyond the usual level of this text, and because, in our experience, it can be difficult for a regular scientist to get CUDA running on their personal computer without help. As is too often the case with parallel computing, one does have to get one's hands dirty with lower level commands. Our presentation should give the reader some general understanding of CUDA. To read more on the subject, we recommend the CUDA tutorial (Zeller, 2008), as well as (Kirk and Wen-Mei, 2013; Sanders and Kandrot, 2011; Faber, 2010).

### 11.5.1 The GPU Card

GPUs are hardware components that are designed to accelerate the storage, manipulation, and display of visual images. GPUs tend to have more core chips and more arithmetic data processing units than do multicore central processing units (CPUs), and so are essentially fast, but limited, parallel computers. Because GPUs have become commodity items on most PCs (think gaming), they hold the promise of being a high powered, yet inexpensive, parallel computing environment. At present an increasing number of scientific applications are being adapted to GPUs, with the CPU being used for the sequential part of a program and the GPU for the parallel part.

The most popular GPU programming language is CUDA. CUDA is both a platform and a programming model created by Nvidia for parallel computing on their GPU. The model defines:

  1. threads, clocks, and grids,
  2. a memory system with registers, local, shared and global memories, and
  3. an execution environment with scheduling of threads and blocks of threads.

Figure 11.3 shows a schematic of a typical Nvidia card containing four streaming multiprocessors (SMs), each with eight streaming processors (SPs), two special function units (SFUs), and 16 kB of shared memory. The SFUs are used for the evaluation of the, otherwise, time-consuming transcendental functions such as sin, cosine, reciprocal, and square root. Each group of dual SMs form a texture processing cluster (TPC) that is used for pixel shading or general-purpose graphical processing. Also represented in the figure is the transfer of data among the three types of memory structures on the GPU and those on the host (Figure 11.3a). Having memory on the chip is much faster than accessing remote memory structures.

## 11.6 Practical Tips for Multicore and GPU Programming

We have already described some basic elements of exascale computing in Section 10.16. Some practical tips for programming multinode-multicore-GPU computers follow along the same lines as we have been discussing, but with an even greater emphasis on minimizing communication costs.2) Contrary to the traditional view on optimization, this means that the "faster" of two algorithms may be the one that takes more steps, but requires less communications. Because the effort in programming the GPU directly can be quite high, many application programmers prefer to let compiler extensions and wrappers deal with the GPU. But if you must, here is how.

**Figure 11.3** A schematic of a GPU (b) and its interactions with a CPU (a). The GPU is seen to have streaming multiprocessors, special function units and several levels of memory. Also shown is the communication between memory levels and with the host CPU.

Exascale computers and computing are expected to be _"disruptive technologies"_ in that they lead to drastic changes from previous models for computers and computing. This is in contrast to the more evolving technology of continually increasing the clock speed of the CPU, which was the practice until the power consumption and associated heat production imposed a roadblock. Accordingly, we should expect that software and algorithms will be changing (and we will have to rewrite our applications), much as it did when supercomputers changed from large vector machines with proprietary CPUs to cluster computers using commodity CPUs and message passing. Here are some of the major points to consider:

**Exacascale data movement is expensive** The time for a floating-point operation and for a data transfer can be similar, although if the transfer is not local, as Figures 10.13 and 10.14 show happens quite often, then communication will be the rate-limiting step.

**Exacascale flops are cheap and abundant** GPUs and local multicore chips provide many, very fast flops for us to use, and we can expect even more of these elements to be included in future computers. So do not worry about flops as much as communication.

**Synchronization-reducing algorithms are essential** Having many processors stop in order to synchronize with each other, while essential in ensuring that the proper calculation is being performed, can slow down processing to a halt (literally). It is better to find or derive an algorithm that reduces the need for synchronization.

**Break the fork-join model** This model for parallel computing takes a queue of incoming jobs, divides them into subjobs for service on a variety of servers, and then combines them to produce the final result. Clearly, this type of model can lead to completed subjobs on various parts of the computer that must wait for other subjobs to complete before recombination. A big waste of resources.

**Communication-reducing algorithms** As already discussed, it is best to use methods that lessen the need for communication among processors, even if more flops are needed.

**Use mixed precision methods** Most GPUs do not have native double precision floating-point numbers (or even full single precision) and correspondingly slow down by a factor-of-two or more when made to perform double precision calculation, or when made to move double-precision data. The preferred approach then is to use single precision. One way to do this is to utilize the perturbation theory in which the calculation focuses on the small (single precision) correction to a known, or separately calculated, large (double precision) basic solution. The rule-of-thumb then is to use the lowest precision required to achieve the required accuracy.

**Push for and use autotuning** The computer manufactures have advanced the hardware to incredible speeds, but have not produced concordant advances in the software that scientists need to utilize in order to make good use of these machines. It often takes years for people to rewrite a large program for these new machines, and that is a tremendous investment that not many scientists can afford. We need smarter software to deal with such complicated machines, and tools that permit us to optimize experimentally our programs for these machines.

**Fault resilient algorithms** Computers containing millions or billions of components do make mistakes at times. It does not make sense to have to start a calculation over, or hold a calculation up, when some minor failure such as a bit flip occurs. The algorithms should be able to recover from these types of failures.

**Reproducibility of results** Science is at its heart the search for scientific truth, and there should be only one answer when solving a problem whose solution is mathematically unique. However, approximations in the name of speed are sometimes made and this can lead to results whose exact reproducibility cannot be guaranteed. (Of course exact reproducibility is not to be expected for Monte Carlo calculations involving chance.)

**Data layout is critical** As we discussed with Figures 10.3, 11.2, and 11.4, much of HPC deals with matrices and their arrangements in memory. With parallel computing, we must arrange the data into tiles such that each data tile is contiguous in memory. Then, the tiles can be processed in a fine-grained computation. As we have seen in the exercises, the best size for the tiles depends upon the size of the caches being used, and these are generally small.

**Figure 11.4** A schematic of how the contiguous elements of a matrix must be tiled for parallel processing (from Dongarra, 2011).

### 11.6.1 CUDA Memory Usage

CUDA commands are extensions of the C programming language and are used to control the interaction between the GPU's components. CUDA supports several data types such as dim2, dim3, and 2D textures, for 2D, 3D, and 2D shaded objects, respectively. PyCUDA (Klöckner, 2014), in turn, provides access to CUDA from Python programs as well as providing some additional extensions of C. Key concepts when programming with CUDA are following:

**Kernel** A part or parts of a program that are executed in parallel by the GPU using threads. The kernel is called from the host and executed on the _device_ (another term for the GPU).

**Thread** The basic element of data to be processed on the GPU. Sometimes defined as an execution of a kernel with a given index, or the abstraction of a function call acting on data. The numbers of threads that are able to run concurrently on a GPU depends upon the specific hardware being used.

**Index** All threads see the same kernel, but each kernel is executed with a different parameter or subscript, called an index.

As a working example, consider the summation of the arrays `**a**` and `**b**` by a kernel with the result assigned to the array `**c:**`

(11.11)

If these were calculated using a sequential programming model, then we might use a `**for**` loop to sum over the index `**i**`. However, this is not necessary in CUDA where each thread sees the same kernel, but there is a different ID assigned to each thread. Accordingly, one thread associated with an ID would see

(11.12)

while another thread assigned to a different ID might see

(11.13)

and so forth. In this way, the summation is performed in parallel by the GPU. You can think of this model as analogous to an orchestra in which all violins plays the same part with the conductor setting the timing (the clock cycles in the computer). Each violinist has the same sheet music (the kernel) yet plays his/her own violin (the threads).

As we have already seen in Figure 11.3, CUDA's GPU architecture contains scalable arrays of multithreaded SMs. Each SM contains eight SPs, also called CUDA cores, and these are the parts of the GPU that run the kernels. The SPs are execution cores like CPUs, but simpler. They typically can perform more operations in a clock cycle than the CPU, and being dedicated just to data processing, constitute the arithmetic logic units of the GPU. The speed of dedicated SMs, along with their working in parallel, leads to the significant processing power of GPUs, even when compared to a multicore CPU computer.

In the CUDA approach, the threads are assigned in _blocks_ that can accommodate from one to thousands of threads. For example, the CUDA GeForce GT540M can handle 1024 threads per block, and has 96 cores. As we have said, each thread runs the same program, which make this a single program multiple data computer. The threads of a block are partitioned into _warps_ (set of lengthwise yarns in weaving), with each warp containing 32 threads. The blocks can be 1D, 2D, or 3D, and are organized into _grids._ The grids, in turn, can be ID or 2D.

When a grid is created or _launched,_ the blocks are assigned to the SMs in an arbitrary order, with each block partitioned into warps, and each thread going to a different streaming processor. If more blocks are assigned to an SM than it can process at once, the excess blocks are scheduled for a later time. For example, the GT200 architecture can process 8 blocks or 1024 threads per SM, and contain 30 SMs. This means that the 8 × 30 = 240 CUDA cores (SPs) can process 30 × 1024 = 30 720 threads in parallel. Compare that to a 16 core CPU!

### 11.6.2 CUDA Programming

The installation of CUDA development tools follows a number of steps:

  1. Establish that the video card on your system has a GPU with Nvidia CUDA. Just having an Nvidia card might not do because not all Nvidia cards have a CUDA architecture.
  2. Verify that your operating system can support CUDA and PyCuda (Linux seems to be the preferred system).
  3. On Linux, verify that your system has gcc installed. On Windows, you will need the Microsoft Visual Studio compiler.
  4. Download the Nvidia CUDA Toolkit.
  5. Install the Nvidia CUDA Toolkit.
  6. Test that the installed software runs and can communicate with the CPU.
  7. Establish that your Python environment contains PyCuda. If not, install it.

Let us now return to our problem of summing two arrays and storing the result, `**a[i]+b[i]=c[i]**`. The steps needed to solve this problem using a host (CPU) and a device (GPU) are:

  1. Declare single precision arrays in the host and initialize the arrays.
  2. Reserve space in the device memory to accommodate the arrays.
  3. Transfer the arrays from the host to the device.
  4. Define a kernel on the host to sum the arrays, with the execution of the kernel to be performed on the device.
  5. Perform the sum on the device, transfer the result to the host and print it out.
  6. Finally, free up the memory space on the device.

**Listing 11.16** The PyCUDA program **SumArraysCuda.py** uses the GPU to do the array sum _a + b = c._

Listing 11.16 presents a PyCUDA program `**SummArraysCuda.py**` that performs our array summation on the GPU. The program is seen to start on line 3 with the `**import pycuda.autoinit**` command that initializes CUDA to accept the kernel. Line 4, `**import pycuda.driver as drv**`, is needed for Pycuda to be able to find the available GPU. Line 7, `**from pycuda.compiler import SourceModule**` prepares the Nvidia kernel compiler `**nvcc**`. The SourceModule, which nvcc compiles, is given in CUDA C on lines 10–14.

As indicated before, even though we are summing indexed arrays, there is no explicit `**for**` loop running over an index `**i**`. This is because CUDA and the hardware know about arrays and so take care of the indices automatically. This works by having all threads seeing the same kernel, and also by having each thread process a different value of `**i**`. Specifically, notice the command `**const int i = threadIdx.x**` on line 11 in the source module. The suffix `**.x**` indicates that we have a one dimensional collections of threads. If we had a 2D or 3D collection, you would also need to include `**threadIdx.y**` and `**threadIdx.z**` commands. Also notice in this same `**SourceModule**` the prefix `**__global__**` on line 11. This indicates that the execution of the following summation is to be spread among the processors on the device.

In contrast to Python, where arrays are double precision floating-point numbers, in CUDA arrays are single precision. Accordingly, on lines 17 and 18 of Listing 11.16, we specify the type of arrays with the command

`**a = numpy.array(range(N)).astype(numpy.float32)**` .

The prefix numpy indicates where the array data type is defined, while the suffix indicates that the array is single precision (32 bits). Next on lines 22–24 are the `**drv.mem_alloc(a.nbytes)**` statements needed to reserve memory on the device for the arrays. With a already defined, `**a.nbytes**` translates into the number of bytes in the array. After that, the `**drv.memcp_htod()**` commands are used to copy a and b to the device. The sum is performed with the `**sum_ab()**` command, and then the result is sent back to the host for printing.

Several of these steps can be compressed by using some PyCUDA-specific commands. Specifically, the memory allocation, the performance of the summation and the transmission of the results on lines 22–28 can be replaced by the statements

with the rest of the program remaining the same. The statement `**grid=(1,1)**` describes the dimension of the grid in blocks as 1 in _x_ and 1 in _y,_ which is the same as saying that the grid is composed of one block. The `**block(N,1,1)**` command indicates that there are `**N (= 32)**` threads, with the 1,1 indicating that the threads are only in the _x_ direction.

**Listing 11.17** The PyCUDA program **SumArraysCuda2.py** uses the GPU to do the array sum _a + b = c,_ using four blocks with a (4,1) grid.

**Figure 11.5** (a) A 1 × 1 grid with a 1D block containing 32 threads, and the corresponding threadIdx.x that goes from 0 to 31. (b) A 1D 4 × 1 grid, that has 4 blocks with 8 threads each.

Our example is so simple that we have organized it into just a single 1D block of 32 threads in a 1 × 1 grid (Figure 11.5a). However, in more complicated problems, it often makes sense to distribute the threads over a number of different block, with each block performing a different operation. For example, the program `**SumArraysCuda2.py**` in Listing 11.17 performs the same sum with the four-block structure illustrated in Figure 11.5b. The difference in the program here is the statement on line 23, `**block=(8,1,1), grid=(4,1)**` that specifies a grid of four blocks in 1D `**x**`, with each 1D block having eight threads. To use this structure of the program, on line 10 we have the command `**const int i = threadIdx.x+blockDim.x*blockIdx.x;**`. Here `**blockDim.x**` is the dimension of each block in threads (8 in this case numbered 0, 1,...,7), with `**blockIdx.x**` indicating the block numbering. The threads in each block have `**treadIdx.x**` from 0 to 7 and the corresponding IDs of the blocks are `**blockIdx.x**` = 0, 1, 2, and 3.

1) Note that the codes refer to the eigenvector _c_ 0 as `coef`. 2) Much of the material in this section comes from talks by John Dongarra (Dongarra, 2011).

# 12
Fourier Analysis: Signals and Filters

* * *

We start this chapter with a discussion of Fourier series and Fourier transforms, the standard tools for decomposing periodic and nonperiodic motions, respectively. We find that, as implemented for numerical computation, both the series and the transform become the same discrete Fourier transform (DFT) algorithm, which has a beautiful simplicity in its realization as a program. We then show how Fourier tools can be used to reduce noise in measured or simulated signals. We end the chapter with a discussion of the fast Fourier transform (FFT), a technique so efficient that it permits nearly instantaneous evaluations of DFTs on various devices.

## 12.1 Fourier Analysis of Nonlinear Oscillations

Consider a particle oscillating either in the nonharmonic potential of (8.5):

(12.1)

or in the perturbed harmonic oscillator potential (8.2),

(12.2)

While free oscillations in these potentials are always periodic, they are not sinusoidal. Your _problem_ is to take the solution of one of these nonlinear oscillators and expand it in a Fourier basis:

(12.3)

For example, if your oscillator is sufficiently nonlinear to behave like the sawtooth function (Figure 12.1a), then the Fourier spectrum you obtain should be similar to that shown in Figure 12.1b.

In general, when we undertake such a spectral analysis we want to analyze the steady-state behavior of a system. This means that we have waited for the initial transient behavior to die out. It is easy to identify just what the initial transient is for linear systems, but may be less so for nonlinear systems in which the "steady state" jumps among a number of configurations. In the latter case, we would have different Fourier spectra at different times.

**Figure 12.1** (a) A sawtooth function that repeats infinitely in time. (b) The Fourier spectrum of frequencies contained in this function (natural units). Also see Figure 1.9 in which we show the result of summing a finite number of terms of this series.

## 12.2 Fourier Series (Math)

Part of our interest in nonlinear oscillations arises from their lack of study in traditional physics courses where linear oscillations, despite the fact that they are just a first approximation, are most often studied. If the force on a particle is always toward its equilibrium position (a restoring force), then the resulting motion will be _periodic,_ but not necessarily _harmonic._ A good example is the motion in a highly anharmonic potential with   in (12.1) that produces an _x_ ( _t_ ) looking like a series of pyramids; this motion is periodic but not harmonic.

In a sense, our approach is the inverse of the traditional one in which the _fundamental_ oscillation is determined analytically and the higher frequency _overtones_ are determined by perturbation theory (Landau and Lifshitz, 1976). We start with the full (numerical) periodic solution and then decompose it into what may be called _harmonics._ When we speak of fundamentals, overtones, and harmonics, we speak of solutions to the linear _boundary-value problem,_ for example, of waves on a plucked violin string. In the latter case, and when given the correct conditions (enough musical skill), it is possible to excite individual harmonics or sums of them in the series

(12.4)

Anharmonic oscillators vibrate at a single frequency (which may vary with amplitude) but not with a sinusoidal waveform. Although it is mathematically proper to expand nonlinear oscillations in a Fourier series, this does not imply that the individual harmonics can be excited (played).

You may recall from classical mechanics that the general solution for a vibrating system can be expressed as the sum of the _normal modes_ of that system. These expansions are possible only if we have _linear operators_ and, subsequently, the _principle of superposition:_ If   are solutions of some linear equation, then   is also a solution. The principle of linear superposition does not hold when we solve nonlinear problems. Nevertheless, it is always possible to expand a _periodic_ solution of a _nonlinear_ problem in terms of trigonometric functions with frequencies that are integer multiples of the true frequency of the nonlinear oscillator.1) This is a consequence of _Fourier's theorem_ being applicable to any single-valued periodic function with only a finite number of discontinuities. We assume we know the period _T,_ that is,

(12.5)

This tells us the _true frequency ω_ :

(12.6)

A periodic function (usually designated as the _signal_ ) can be expanded as a series of harmonic functions with frequencies that are multiples of the true frequency:

(12.7)

This equation represents the signal _y_ ( _t_ ) as the simultaneous sum of pure tones of frequency _nω._ The coefficients _a n_ and _b n_ measure the amount of _cos nωt_ and sin _nωt_ present in _y_ ( _t_ ), respectively. The intensity or power at each frequency is proportional to  .

The Fourier series (12.7) is a "best fit" in the least-squares sense of Chapter 7, because it minimizes  , where _i_ denotes different measurements of the signal. This means that the series converges to the _average_ behavior of the function, but misses the function at discontinuities (at which points it converges to the mean) or at sharp corners (where it overshoots). A general function _y_ ( _t_ ) may contain an infinite number of Fourier components, although low-accuracy reproduction is usually possible with a small number of harmonics.

The coefficients _a n_ and _b n_ in (12.7) are determined by the standard techniques for orthogonal function expansion. To find the coefficients, multiply both sides of (12.7) by cos _n  t_ or sin _n  t_, integrate over one period, and project a single _a n_ or _b n_:

(12.8)

As seen in the _b n_ coefficients (Figure 12.1b), these coefficients usually decrease in magnitude as the frequency increases, and can enter with a negative sign, the negative sign indicating the relative phase.

Awareness of the _symmetry_ of the function _y_ ( _t_ ) may eliminate the need to evaluate all the expansion coefficients. For example,

  * _a_ 0 is twice the average value of _y_ :

(12.9)

  * For an _odd function,_ that is, one for which  , and only half of the integration range is needed to determine _b n_:

(12.10)

However, if there is no input signal for  , we do not have a truly odd function, and so small values of _a n_ may occur.
  * For an _even function,_ that is, one for which  , and only half the integration range is needed to determine _a n_:

(12.11)

### 12.2.1 Examples: Sawtooth and Half-Wave Functions

The sawtooth function (Figure 12.1) is described mathematically as

(12.12)

It is clearly periodic, nonharmonic, and discontinuous. Yet it is also odd and so can be represented more simply by shifting the signal to the left:

(12.13)

Although the general shape of this function can be reproduced with only a few terms of the Fourier components, many components are needed to reproduce the sharp corners. Because the function is odd, the Fourier series is a sine series and (12.8) determines the _b n_ values:

(12.14)

(12.15)

The half-wave function

(12.16)

is periodic, nonharmonic (the upper half of a sine wave), and continuous, but with discontinuous derivatives. Because it lacks the sharp corners of the sawtooth function, it is easier to reproduce with a finite Fourier series. Equation 12.8 determines

(12.17)

## 12.3 Exercise: Summation of Fourier Series

_Hint:_ The program **FourierMatplot.py** written by Oscar Restrepo performs a Fourier analysis of a sawtooth function and produces the visualization shown in Figure 1.9b. You may want to use this program to help with this exercise.

  1. _Sawtooth function:_ Sum the Fourier series for the _sawtooth function_ up to order _n_ = 2, 4, 10, 20, and plot the results over two periods.
    1. a) Check that in each case, the series gives the mean value of the function _at_ the points of discontinuity.
    2. b) Check that in each case the series _overshoots_ by about 9% the value of the function on either side of the discontinuity (the _Gibbs phenomenon_ ).
  2. _Half-wave function:_ Sum the Fourier series for the _half-wave function_ up to order _n_ = 2, 4, 10, 50 and plot the results over two periods. (The series converges quite well, doesn't it?)

## 12.4 Fourier Transforms (Theory)

Although a Fourier _series_ is the right tool for approximating or analyzing periodic functions, the Fourier _transform_ or _integral_ is the right tool for nonperiodic functions. We convert from series to transform by imagining a system described by a continuum of "fundamental" frequencies. We thereby deal with _wave packets_ containing continuous rather than discrete frequencies.2) While the difference between series and transform methods may appear clear mathematically, when we approximate the Fourier integral as a finite sum, the two become equivalent.

By analogy with (12.7), we now imagine our function or signal _y_ ( _t_ ) expressed in terms of a continuous series of harmonics ( _inverse Fourier transform_ ):

(12.18)

where for compactness we use a complex exponential function.3) The expansion amplitude _Y_ ( _ _) is analogous to the Fourier coefficients ( _a n, bn_), and is called the _Fourier transform_ of _y_ ( _t_ ). The integral (12.18) is the inverse transform because it converts the transform to the signal. The _Fourier transform_ converts the signal _y_ ( _t_ ) to its transform _Y_ ( _ω_ ):

(12.19)

The   factor in both these integrals is a common normalization in quantum mechanics, but maybe not in engineering where only a single 1/2π factor is used. Likewise, the signs in the exponents are also conventions that do not matter as long as you maintain consistency.

If _y_ ( _t_ ) is the measured response of a system (signal) as a function of time, then _Y_ ( _ω_ ) is the _spectral function_ that measures the amount of frequency _ω_ present in the signal. In many cases, it turns out that _Y_ ( _ω_ ) is a complex function with both positive and negative values, and with powers-of-ten variation in magnitude. Accordingly, it is customary to eliminate some of the complexity of _Y_ ( _ω_ ) by making a semilog plot of the squared modulus   vs _ω_. This is called a _power spectrum_ and provides an immediate view of the amount of power or strength in each component.

If the Fourier transform and its inverse are consistent with each other, we should be able to substitute (12.18) into (12.19) and obtain an identity:

(12.20)

(12.21)

For this to be an identity, the term in braces must be the _Dirac delta function_ :

(12.22)

While the delta function is one of the most common and useful functions in theoretical physics, it is not well behaved in a mathematical sense and misbehaves terribly in a computational sense. While it is possible to create numerical approximations to  , they may well be borderline pathological. It is certainly better for you to do the delta function part of an integration analytically and leave the nonsingular leftovers to the computer.

## 12.5 The Discrete Fourier Transform

If _y_ ( _t_ ) or _Y_ ( _ω_ ) is known analytically or numerically, integrals (12.18) and (12.19) can be evaluated using the integration techniques studied earlier. In practice, the signal _y_ ( _t_ ) is measured at just a finite number _N_ of times _t,_ and these are all we have as input to approximate the transform. The resultant _discrete Fourier transform_ is an approximation both because the signal is not known for all times, and because we must integrate numerically (Briggs and Henson, 1995). Once we have a discrete set of (approximate) transform values, they can be used to reconstruct the signal for any value of the time. In this way, the DFT can be thought of as a technique for interpolating, compressing, and extrapolating the signal.

We assume that the signal _y_ ( _t_ ) is sampled at ( _N_ \+ 1) discrete times ( _N_ time intervals), with a constant spacing   between times:

(12.23)

(12.24)

In other words, we measure the signal _y_ ( _t_ ) once every _h_ th of a second for a total time of _T_. This correspondingly define the signal's period _T_ and the _sampling rate s_ :

(12.25)

Regardless of the true periodicity of the signal, when we choose a period _T_ over which to sample the signal, the mathematics will inevitably produce a _y_ ( _t_ ) that is periodic with period _T,_

(12.26)

We recognize this periodicity, and ensure that there are only _N_ independent measurements used in the transform, by defining the first and last _y_ 's to be equal:

(12.27)

If we are analyzing a truly periodic function, then the _N_ points should span one complete period, but not more. This guarantees their independence. Unless we make further assumptions, the _N_ independent data _y_ ( _t k_) can determine no more than _N_ independent transform values  .

The time interval _T_ (which should be the period for periodic functions) is the largest time over which we measure the variation of _y_ ( _t_ ). Consequently, it determines the lowest frequency contained in our Fourier representation of _y_ ( _t_ ),

(12.28)

The full range of frequencies in the spectrum _ω n_ are determined by the number of samples taken, and by the total sampling time _T = Nh_ as

(12.29)

Here _ω_ 0 = 0 corresponds to the zero-frequency or DC component of the transform, that is, the part of the signal that does not oscillate.

The discrete Fourier transform (DFT) algorithm follows from two approximations. First we evaluate the integral in (12.19) from time 0 to time _T,_ over which the signal is measured, and not from  . Second, the trapezoid rule is used for the integration4):

(12.30)

(12.31)

To keep the final notation more symmetric, the step size _h_ is factored from the transform _Y_ and a discrete function _Y n_ is defined as

(12.32)

With this same care in accounting, and with  , we invert the _Y n_'s:

(12.33)

(12.34)

Once we know the _N_ values of the transform, we can use (12.34) to evaluate _y_ ( _t_ ) for any time _t._ There is nothing illegal about evaluating _Y n_ and _y k_ for arbitrarily large values of _n_ and _k,_ yet there is also nothing to be gained either. Because the trigonometric functions are periodic, we just get the old answers:

(12.35)

(12.36)

Another way of stating this is to observe that none of the equations change if we replace  . There are still just _N_ independent output numbers for _N_ independent inputs, and so the transform and the reconstituted signal are periodic.

We see from (12.29) that the larger we make the time _T_ = _Nh_ over which we sample the function, the smaller will be the frequency steps or resolution.5) Accordingly, if you want a smooth frequency spectrum, you need to have a small frequency step 2π/ _T_ , which means a longer observation time _T_. While the best approach would be to measure the input signal for all times, in practice a measured signal _y_ ( _t_ ) is often extended in time ("padded") by adding zeros for times beyond the last measured signal, which thereby increases the value of _T_ artificially. Although this does not add new information to the analysis, it does build in the experimentalist's view that the signal has no existence, or no meaning, at times after the measurements are stopped.

While periodicity is expected for a Fourier _series,_ it is somewhat surprising for Fourier a _integral,_ which have been touted as the right tool for nonperiodic functions. Clearly, if we input values of the signal for longer lengths of time, then the inherent period becomes longer, and if the repeat period _T_ is very long, it may be of little consequence for times short compared to the period. If _y_ ( _t_ ) is actually periodic with period _Nh,_ then the DFT is an excellent way of obtaining Fourier series. If the input function is not periodic, then the DFT can be a bad approximation near the endpoints of the time interval (after which the function will repeat) or, correspondingly, for the lowest frequencies.

The DFT and its inverse can be written in a concise and insightful way, and be evaluated efficiently, by introducing a complex variable _Z_ for the exponential and then raising _Z_ to various powers:

(12.37)

(12.38)

With this formulation, the computer needs to compute only powers of _Z_. We give our DFT code in Listing 12.1. If your preference is to avoid complex numbers, we can rewrite (12.37) in terms of separate real and imaginary parts by applying Euler's theorem with  :

(12.39)

(12.40)

(12.41)

Readers new to DFTs are often surprised when they apply these equations to practical situations and end up with transforms _Y_ having imaginary parts, despite the fact that the signal _y_ is real. Equation 12.40 should make it clear that a real signal   will yield an imaginary transform unless  . This occurs only if _y_ ( _t_ ) is an _even_ function over   _and_ we integrate exactly. Because neither condition holds, the DFTs of real, even functions may have small imaginary parts. This is not as a result of an error in programming, and in fact is a good measure of the approximation error in the entire procedure.

The computation time for a DFT can be reduced even further by using _fast Fourier transform_ algorithm, as discussed in Section 12.9. An examination of (12.37) shows that the DFT is evaluated as a matrix multiplication of a vector of length _N_ containing the _Z_ values by a vector of length _N_ of _y_ value. The time for this DFT scales like _N_ 2, while the time for the FFT algorithm scales as _N_ log2 _N_. Although this may not seem like much of a difference, for _N_ = 102–3, the difference of 103–5 is the difference between a minute and a week. For this reason, it is the FFT is often used for online spectrum analysis.

**Listing 12.1** **DFTcomplex.py** uses the built-in complex numbers of Python to compute the DFT for the signal in method f(signal).

### 12.5.1 Aliasing (Assessment)

The sampling of a signal by DFT for only a finite number of times (large Δ _t_ ) limits the accuracy of the deduced high-frequency components present in the signal. Clearly, good information about very high frequencies requires sampling the signal with small time steps so that all the wiggles can be included. While a poor deduction of the high-frequency components may be tolerable if all we care about are the low-frequency components, the inaccurate high-frequency components remain present in the signal and may contaminate the low-frequency components that we deduce. This effect is called _aliasing_ and is the cause of the Moiré pattern distortion in digital images.

As an example, consider Figure 12.2 showing the two functions  , with their points of overlap in bold. If we were unfortunate enough to sample a signal containing these functions at the times _t_ = 0, 2, 4, 6, 8, then we would measure   and assume that there was no signal at all. However, if we were unfortunate enough to measure the signal at the filled dots in Figure 12.2 where  , specifically, _t_ = 0, 12/10, 4/3,..., then our Fourier analysis would completely miss the high-frequency component. In DFT jargon, we would say that the high-frequency component has been _aliased_ by the low-frequency component. In other cases, some high-frequency values may be included in our sampling of the signal, but our sampling rate may not be high enough to include enough of them to separate the high-frequency component properly. In this case some high-frequency signals would be included spuriously as part of the low-frequency spectrum, and this would lead to spurious low-frequency oscillations when the signal is synthesized from its Fourier components.

More precisely, aliasing occurs when a signal containing frequency _f_ is sampled at a rate of _s_ = _N/T_ measurements per unit time, with  . In this case, the frequencies _f_ and _f_ – 2 _s_ yield the same DFT, and we would not be able to determine that there are two frequencies present. That being the case, to avoid aliasing we want no frequencies _f_ > _s_ /2 to be present in our input signal. This is known as the _Nyquist criterion._ In practice, some applications avoid the effects of aliasing by filtering out the high frequencies from the signal and then analyzing only the remaining low-frequency part. (The low-frequency _sinc filter_ discussed in Section 12.8.1 is often used for this purpose.) Although filtering eliminates some high-frequency information, it lessens the distortion of the low-frequency components, and so may lead to improved reproduction of the signal.

**Figure 12.2** A plot of the functions sin(π _t_ /2) and sin(2π _t_ ). If the sampling rate is not high enough, these signalsmay appear indistinguishable in a Fourier decomposition. If the sample rate is too low and if both signals are present in a sample, the deduced low-frequency component may be contaminated by the higher frequency component signal.

If accurate values for the high frequencies are required, then we will need to increase the sampling rate _s_ by increasing the number _N_ of samples taken within the fixed sampling time _T_ = _Nh._ By keeping the sampling time constant and increasing the number of samples taken, we make the time step _h_ smaller and we pick up the higher frequencies. By increasing the number _N_ of frequencies and that you compute, you move the previous higher frequency components in closer to the middle of the spectrum, and thus away from the error-prone ends.

If we increase the total time sampling time _T_ = _Nh_ and keep _h_ the same, then the sampling rate _s_ = _N/T_ = 1/ _h_ remains the same. Because  , this makes _ω_ 1 smaller, which means we have more low frequencies recorded and a smoother frequency spectrum. And as we said, this is often carried out, after the fact, by padding the end of the data set with zeros.

**Listing 12.2** **DFTreal.py** computes the discrete Fourier transform for the signal in method f(signal) using real numbers.

### 12.5.2 Fourier Series DFT (Example)

For simplicity, let us consider the Fourier cosine series:

(12.42)

Here   is the actual period of the system (not necessarily the period of the simple harmonic motion occurring for a small amplitude). We assume that the function _y_ ( _t_ ) is sampled for a discrete set of times

(12.43)

Because we are analyzing a periodic function, we retain the conventions used in the DFT and require the function to repeat itself with period _T_ = _Nh_ ; that is, we assume that the amplitude is the same at the first and last points:

(12.44)

This means that there are only _N_ independent values of _y_ being used as input. For these _N_ independent _y k_ values, we can determine uniquely only _N_ expansion coefficients _a k_. If we use the trapezoid rule to approximate the integration in (12.42), we determine the _N_ independent Fourier components as

(12.45)

Because we can determine only _N_ Fourier components from _N_ independent _y_ ( _t_ ) values, our Fourier series for the _y_ ( _t_ ) must be in terms of only these components:

(12.46)

In summary, we sample the function _y_ ( _t_ ) at _N_ times, _t_ 1,..., _t N_. We see that all the values of _y_ sampled contribute to each _a k_. Consequently, if we increase _N_ in order to determine more coefficients, we must recompute all the _a n_ values. In the wavelet analysis in Chapter 13, the theory is reformulated so that additional samplings determine higher spectral components without affecting lower ones.

### 12.5.3 Assessments

**Simple analytic input** It is always good to do simple checks before examining more complex problems, even if you are using a package's Fourier tool.

  1. Sample the even signal

(12.47)

    1. a) Decompose this into its components.
    2. b) Check that the components are essentially real and in the ratio 3 : 2 : 1 (or 9 : 4 : 1 for the power spectrum).
    3. c) Verify that the frequencies have the expected values (not just ratios).
    4. d) Verify that the components resum to give the input signal.
    5. e) Experiment on the separate effects of picking different values of the step size _h_ and of enlarging the measurement period _T_ = _Nh_.
  2. Sample the odd signal

(12.48)

Decompose this into its components; then check that they are essentially imaginary and in the ratio 1 : 2 : 3 (or 1 : 4 : 9 if a power spectrum is plotted) and that they resum to give the input signal.
  3. Sample the mixed-symmetry signal

(12.49)

Decompose this into its components; then check that there are three of them in the ratio 5 : 2 : 1 (or 25 : 4 : 1 if a power spectrum is plotted) and that they resum to give the input signal.
  4. Sample the signal

Compare and explain the results obtained by sampling (a) without the 5, (b) as given but without the 2, and (c) without the 5 and without the 2.
  5. In our discussion of aliasing, we examined Figure 12.2 showing the functions  . Sample the function

(12.50)

and explore how aliasing occurs. Explicitly, we know that the true transform contains peaks at  . Sample the signal at a rate that leads to aliasing, as well as at a higher sampling rate at which there is no aliasing. Compare the resulting DFTs in each case and check if your conclusions agree with the Nyquist criterion.

**Highly nonlinear oscillator** Recall the numerical solution for oscillations of a spring with power _p_ = 12 (see (12.1)). Decompose the solution into a Fourier series and determine the number of higher harmonics that contribute at least 10%; for example, determine the _n_ for which  . Check that resuming the components reproduces the signal.

**Nonlinearly perturbed oscillator** Remember the harmonic oscillator with a nonlinear perturbation (8.2):

(12.51)

For very small amplitudes of oscillation  , the solution _x_ ( _t_ ) essentially should be only the first term of a Fourier series.

  1. We want the signal to contain "approximately 10% nonlinearity." This being the case, fix your value of _α_ so that   is the maximum amplitude of oscillation. For the rest of the problem, keep the value of _α_ fixed.
  2. Decompose your numerical solution into a discrete Fourier spectrum.
  3. Plot a graph of the percentage of importance of the first _two,_ non-DC Fourier components as a function of the initial displacement for  . You should find that higher harmonics are more important as the amplitude increases. Because both even and odd components are present, _Y n_ should be complex. Because a 10% effect in amplitude becomes a 1% effect in power, make sure that you make a semilog plot of the power spectrum.
  4. As always, check that resumations of your transforms reproduce the signal.

( _Warning:_ The _ω_ you use in your series must correspond to the _true_ frequency of the system, not the _ω_ 0 of small oscillations.)

### 12.5.4 Nonperiodic Function DFT (Exploration)

Consider a simple model (a wave packet) of a "localized" electron moving through space and time. A good model for an electron initially localized around _x_ = 5 is a Gaussian multiplying a plane wave:

(12.52)

This wave packet is not an eigenstate of the momentum operator 6) _p_ = id/d _x_ and in fact contains a spread of momenta. Your _problem_ is to evaluate the Fourier transform

(12.53)

as a way of determining the momenta components in (12.52).

## 12.6 Filtering Noisy Signals

You measure a signal _y_ ( _t_ ) that obviously contains noise. Your _problem_ is to determine the frequencies that would be present in the spectrum of the signal if the signal did not contain noise. Of course, once you have a Fourier transform from which the noise has been removed, you can transform it to obtain a signal _s_ ( _t_ ) with no noise.

_In the process of solving this problem, we examine two simple approaches: the use of autocorrelation functions and the use of filters. Both approaches find wide applications in science, with our discussion not doing the subjects justice. We will see filters again in the discussion of wavelets inChapter 13._

## 12.7 Noise Reduction via Autocorrelation (Theory)

We assume that the measured signal is the sum of the true signal _s_ ( _t_ ), which we wish to determine, plus some unwelcome _noise n_ ( _t_ ):

(12.54)

Our first approach at removing the noise relies on that fact that noise is a random process and thus should not be correlated with the signal. Yet what do we mean when we say that two functions are not _correlated?_ Well, if the two tend to oscillate with their nodes and peaks in much the same places, then the two functions are clearly correlated. An analytic measure of the correlation of two arbitrary functions _y_ ( _t_ ) and _x_ ( _t_ ) is the _correlation function_

(12.55)

where _τ_ , the _lag time,_ is a variable. Even if the two signals have different magnitudes, if they have similar time dependences except for one lagging or leading the other, then for certain values of _τ_ the integrand in (12.55) will be positive for all values of _t_. For those values of _τ_ , the two signals interfere constructively and produce a large value for the correlation function. In contrast, if both functions oscillate independently regardless of the value of _τ_ , then it is just as likely for the integrand to be positive as to be negative, in which case the two signals interfere destructively and produce a small value for the integral.

Before we apply the correlation function to our problem, let us study some of its properties. We use (12.18) to express _c_ , _y_ *, and _x_ in terms of their Fourier transforms:

(12.56)

Because _ω, ω_ ′, and _ω_ ″ are dummy variables, other names may be used for these variables without changing any results. When we substitute these representations into definition (12.55) of the correlation function and assume that the resulting integrals converge well enough to be rearranged, we obtain

(12.57)

where the last line follows because _ω_ ″ and _ω_ are equivalent dummy variables. Equation 12.57 says that the Fourier transform of the correlation function between two signals is proportional to the product of the transform of one signal and the complex conjugate of the transform of the other. (We shall see a related convolution theorem for filters.)

**Figure 12.3** (a) A function that is a signal plus noise _s_ ( _t_ ) + _n_ ( _t_ ); (b) the autocorrelation function vs. time deduced by processing this signal; (c) the power spectrum obtained from autocorrelation function; (d) the signal plus noise after passage through a low-pass filter.

A special case of the correlation function _c_ ( _τ_ ) is the _autocorrelation function A_ ( _τ_ ). It measures the correlation of a time signal with itself:

(12.58)

This function is computed by taking a signal _y_ ( _t_ ) that has been measured over some time period and then averaging it over time using _y_ ( _t_ \+ _τ_ ) as a weighting function. This process is also _called folding_ a function onto itself (as might be done with dough) or a _convolution._ To see how this folding removes noise from a signal, we go back to the measured signal (12.54), which was the sum of pure signal plus noise _s_ ( _t_ ) + _n_ ( _t_ ). As an example, in Figure 12.3a we show a signal that was constructed by adding random noise to a smooth signal. When we compute the autocorrelation function for this signal, we obtain a function (Figure 12.3b) that looks like a broadened, smoothed version of the signal _y_ ( _t_ ).

We can understand how the noise is removed by taking the Fourier transform of _s_ ( _t_ ) + _n_ ( _t_ ) to obtain a simple sum of transforms:

(12.59)

(12.60)

Because the autocorrelation function (12.58) for   involves the second power of _y,_ is not a linear function, that is,  , but instead

(12.61)

If we assume that the noise _n_ ( _t_ ) in the measured signal is truly random, then it should average to zero over long times and be uncorrelated at times _t_ and _t_ \+ _τ_. This being the case, both integrals involving the noise vanish, and so

(12.62)

Thus, the part of the noise that is random tends to be averaged out of the autocorrelation function, and we are left with an approximation of the autocorrelation function of the pure signal.

So how does this help us? Application of (12.57) with   tells us that the Fourier transform _A_ ( _ω_ ) of the autocorrelation function is proportional to  :

(12.63)

The function   is the _power spectrum_ of the pure signal. Thus, evaluation of the autocorrelation function of the noisy signal gives us the pure signal's power spectrum, which is often all that we need to know. For example, in Figure 12.3a we see a noisy signal, the autocorrelation function (Figure 12.3b), which clearly is smoother than the signal, and finally, the deduced power spectrum (Figure 12.3c). Note that the broadband high-frequency components characteristic of noise are absent from the power spectrum.

You can easily modify the sample program `**DFTcomplex.py**` in Listing 12.1 to compute the autocorrelation function and then the power spectrum from _A_ ( _τ_ ). We present a program `**NoiseSincFilter.py**` on the instructor's site that does this.

### 12.7.1 Autocorrelation Function Exercises

  1. Imagine that you have sampled the pure signal

(12.64)

Although there is just a single sine function in the denominator, there is an infinite number of overtones as follows from the expansion

(12.65)

    1. a) Compute the DFT _S_ ( _ω_ ). Make sure to sample just one period but to cover the entire period. Make sure to sample at enough times (fine scale) to obtain good sensitivity to the high-frequency components.
    2. b) Make a semilog plot of the power spectrum  .
    3. c) Take your input signal _s_ ( _t_ ) and compute its autocorrelation function _A_ ( _τ_ ) for a full range of _τ_ values (an analytic solution is okay too).
    4. d) Compute the power spectrum indirectly by performing a DFT on the autocorrelation function. Compare your results to the spectrum obtained by computing   directly.
  2. Add some random noise to the signal using a random number generator:

(12.66)

where _α_ is an adjustable parameter. Try several values of _α_ , from small values that just add some fuzz to the signal to large values that nearly hide the signal.
    1. a) Plot your noisy data, their Fourier transform, and their power spectrum obtained directly from the transform with noise.
    2. b) Compute the autocorrelation function _A_ ( _τ_ ) and its Fourier transform _A_ ( _ω_ ).
    3. c) Compare the DFT of _A_ ( _τ_ ) to the true power spectrum and comment on the effectiveness of reducing noise by use of the autocorrelation function.
    4. d) For what value of _α_ do you essentially lose all the information in the input?

## 12.8 Filtering with Transforms (Theory)

A filter (Figure 12.4) is a device that converts an input signal _f_ ( _t_ ) to an output signal _g_ ( _t_ ) with some specific property for _g_ ( _t_ ). More specifically, an _analog filter_ is defined (Hartmann, 1998) as integration over the input function:

(12.67)

The operation indicated in (12.67) occurs often enough that it is given the name _convolution_ and is denoted by an asterisk *. The function _h_ ( _t_ ) is called the _response_ or _transfer function_ of the filter because it is the response of the filter to a unit impulse:

(12.68)

Equation 12.67 states that the output _g_ ( _t_ ) of a filter equals the input _f_ ( _t_ ) convoluted with the transfer function _h_ ( _t_ – _τ_ ). Because the argument of the response function is delayed by a time _τ_ relative to that of the signal in integral (12.67), _τ_ is called the _lag time._ While the integration is over all times, the response of a good detector usually peaks around zero time. In any case, the response must equal zero for _τ_ > _t_ because events in the future cannot affect the present (causality).

The _convolution theorem_ states that the Fourier transform of the convolution _g_ ( _t_ ) is proportional to the product of the transforms of _f_ ( _t_ ) and _h_ ( _t_ ):

(12.69)

**Figure 12.4** A schematic of an input signal _f_ ( _t_ ) passing through a filter _h_ that outputs the function _g_ ( _t_ ).

The theorem results from expressing the functions in (12.67) by their transforms and using the resulting Dirac delta function to evaluate an integral (essentially what we did in our discussion of correlation functions).

Regardless of the domain used, filtering as we have defined it is a linear process involving just the first powers of _f_. This means that the output at one frequency is proportional to the input at that frequency. The constant of proportionality between the two may change with frequency and thus suppress specific frequencies relative to others, but that constant remains fixed in time. Because the law of linear superposition is valid for filters, if the input to a filter is represented as the sum of various functions, then the transform of the output will be the sum of the functions' Fourier transforms.

Filters that remove or decrease high-frequency components more than they do low-frequency ones, are called _low-pass_ filters. Those that filter out the low frequencies are called _high-pass filters._ A simple low-pass filter is the RC circuit shown in Figure 12.5a. It produces the transfer function

(12.70)

where _τ_ = RC is the time constant. The _ω_ 2 in the denominator leads to a decrease in the response at high frequencies and therefore makes this a low-pass filter (the i _ω_ affects only the phase). A simple high-pass filter is the RC circuit shown in Figure 12.5b. It produces the transfer function

(12.71)

_H_ = 1 at large _ω_ , yet _H_ vanishes as _ω_ → 0, as expected for a high-pass filter.

Filters composed of resistors and capacitors are fine for analog signal processing. For digital processing we want a _digital filter_ that has a specific response function for each frequency range. A physical model for a digital filter may be constructed from a delay line with taps at various spacing along the line (Figure 12.6)

**Figure 12.5** (a) An RC circuit arranged as a low-pass filter. (b) An RC circuit arranged as a high-pass filter.

**Figure 12.6** A delay-line filter in which the signal at different times is scaled by different amounts _c i_.

(Hartmann, 1998). The signal read from tap _n_ is just the input signal delayed by time _nτ_ , where the delay time _τ_ is a characteristic of the particular filter. The output from each tap is described by the transfer function  , possibly with scaling factor _c n._ As represented by the triangle in Figure 12.6b, the signals from all taps are ultimately summed together to form the total response function:

(12.72)

In the frequency domain, the Fourier transform of a delta function is an exponential, and so (12.72) results in the transfer function

(12.73)

where the exponential indicates the phase shift from each tap.

If a digital filter is given a continuous time signal _f_ ( _t_ ) as input, its output will be the discrete sum

(12.74)

And of course, if the signal's input is a discrete sum, its output will remain a discrete sum. In either case, we see that knowledge of the filter coefficients _c i_ provides us with all we need to know about a digital filter. If we look back at our work on the DFT in Section 12.5, we can view a digital filter (12.74) as a Fourier transform in which we use an _N_ -point approximation to the Fourier integral. The _c ns_ then contain both the integration weights and the values of the response function at the integration points. The transform itself can be viewed as a filter of the signal into specific frequencies.

### 12.8.1 Digital Filters: Windowed Sinc Filters (Exploration)

**Problem** Construct digital versions of high-pass and low-pass filters and determine which filter works better at removing noise from a signal.

**Figure 12.7** The rectangle function rect( _ω_ ) that is constant for a finite frequency interval. The Fourier transform of this function is sinc( _t_ ).

A popular way to separate the bands of frequencies in a signal is with a _windowed sinc filter_ (Smith, 1999). This filter is based on the observation that an ideal _low-pass_ filter passes all frequencies below a cutoff frequency _ω_ c, and blocks all frequencies above this frequency. And because there tends to be more noise at high frequencies than at low frequencies, removing the high frequencies tends to remove more noise than signal, although some signal is inevitably lost. One use for windowed sinc filters is in reducing aliasing in DFTs by removing the high-frequency component of a signal before determining its Fourier components. The graph in Figure 12.1b was obtained by passing our noisy signal through a sinc filter (using the program `**NoiseSincFilter.py**`).

If both positive and negative frequencies are included, an ideal low-frequency filter will look like the rectangular pulse in frequency space (Figure 12.7):

(12.75)

Here rect( _ω_ ) is the rectangular function. Although maybe not obvious, a rectangular pulse in the frequency domain has a Fourier transform that is proportional to the _sinc function_ in the time domain (Smith, 1991)

(12.76)

where the πs are sometimes omitted. Consequently, we can filter out the high-frequency components of a signal by convoluting it with  , a technique also known as the _Nyquist–Shannon_ interpolation formula. In terms of discrete transforms, the time-domain representation of the sinc filter is simply

(12.77)

Because all frequencies below the cutoff frequency _ω_ c are passed with unit amplitude, while all higher frequencies are blocked, we can see the importance of a sinc filter.

In practice, there are a number of problems in using sinc function as the filter. First, as formulated, the filter is _noncausal;_ that is, there are coefficients at negative times, which is nonphysical because we do not start measuring the signal until _t_ = 0. Second, in order to produce a perfect rectangular response, we would have to sample the signal at an infinite number of times. In practice, we sample at ( _M_ \+ 1) points ( _M_ even) placed symmetrically around the main lobe of sin(π _t_ )/π _t_ , and then shift times to purely positive values via

(12.78)

As might be expected, a penalty is incurred for making the filter discrete; instead of the ideal rectangular response, we obtain some _Gibbs overshoot,_ with rounded corners and oscillations beyond the corner.

There are two ways to reduce the departures from the ideal filter. The first is to increase the length of times for which the filter is sampled, which inevitably leads to longer compute times. The other way is to smooth out the truncation of the sinc function by multiplying it with a smoothly tapered curve like the _Hamming window function:_

(12.79)

In this way, the filter's kernel becomes

(12.80)

The cutoff frequency _ω_ c should be a fraction of the sampling rate. The time length _M_ determines the _bandwidth_ over which the filter changes from 1 to 0.

**Exercise** Repeat the exercise that added random noise to a known signal, this time using the sinc filter to reduce the noise. See how small you can make the signal and still be able to separate it from the noise.

## 12.9 The Fast Fourier Transform Algorithm

We have seen in (12.37) that a discrete Fourier transform can be written in the compact form as

(12.81)

Even if the signal elements _y k_ to be transformed are real, _Z_ is complex, and therefore we must process both real and imaginary parts when computing transforms. Because both _n_ and _k_ range over _N_ integer values, the   multiplications in (12.81) require some _N_ 2 multiplications and additions of complex numbers. As _N_ gets large, as happens in realistic applications, this geometric increase in the number of steps leads to long computation times.

In 1965, Cooley and Tukey discovered an algorithm7) that reduces the number of operations necessary to perform a DFT from _N_ 2 to roughly _N_ log2 _N_ (Cooley, 1965; Donnelly and Rust, 2005). Although this may not seem like such a big difference, it represents a 100-fold speedup for 1000 data points, which changes a full day of processing into 15 min of work. Because of its widespread use (including cell phones), the fast Fourier transform algorithm is considered one of the 10 most important algorithms of all time.

The idea behind the FFT is to utilize the periodicity inherent in the definition of the DFT (12.81) to reduce the total number of computational steps. Essentially, the algorithm divides the input data into two equal groups and transforms only one group, which requires ~ ( _N_ /2)2 multiplications. It then divides the remaining (nontransformed) group of data in half and transforms them, continuing the process until all the data have been transformed. The total number of multiplications required with this approach is approximately _N_ log2 _N._

Specifically, the FFT's time economy arises from the computationally expensive complex factor   having values that are repeated as the integers _n_ and _k_ vary sequentially. For instance, for _N_ = 8,

where we include   for clarity. When we actually evaluate these powers of _Z_ , we find only four independent values:

(12.82)

When substituted into the definitions of the transforms, we obtain

(12.83)

(12.84)

(12.85)

(12.86)

(12.87)

(12.88)

(12.89)

(12.90)

(12.91)

We see that these transforms now require 8 × 8 = 64 multiplications of complex numbers, in addition to some less time-consuming additions. We place these equations in an appropriate form for computing by regrouping the terms into sums and differences of the _y_ 's:

(12.92)

(12.93)

(12.94)

(12.95)

(12.96)

(12.97)

(12.98)

(12.99)

(12.100)

Note the repeating factors inside the parentheses, with combinations of the form  . These symmetries are systematized by introducing the _butterfly operation_ (Figure 12.8). This operation takes the _y p_ and _y q_ data elements from the left wing and converts them to the _y p_ \+ _Zy q_ elements in the right wings. In Figure 12.9 we show what happens when we apply the butterfly operations to an entire FFT process, specifically to the pairs ( _y_ 0, _y_ 4), ( _y_ 1 _y_ 5), ( _y_ 2, _y_ 6), and ( _y_ 3, _y_ 7). Note how the number of multiplications of complex numbers has been reduced: For the first butterfly operation there are 8 multiplications by _Z_ 0; for the second butterfly operation there are 8 multiplications, and so forth, until a total of 24 multiplications are made in four butterflies. In contrast, 64 multiplications are required in the original DFT (12.91).

**Figure 12.8** The basic butterfly operation in which elements _y p_ and _y q_ on the left are transformed into _y p_ \+ _Zy q_ and _y p_ – _Zy q_ on the right.

### 12.9.1 Bit Reversal

The reader may have observed in Figure 12.9 that we started with eight data elements in the order 0–7 and that after three butterfly operators we obtained transforms in the order 0, 4, 2, 6, 1, 5, 3, 7. The astute reader may further have observed that these numbers correspond to the bit-reversed order of 0–7. Let us look into this further. We need 3 bits to give the order of each of the 8 input data elements (the numbers 0–7). Explicitly, on the left in Table 10.1 we give the binary representation for decimal numbers 0–7, their bit reversals, and the corresponding decimal numbers. On the right we give the ordering for 16 input data elements, where we need 4 bits to enumerate their order. Notice that the order of the first 8 elements differs in the two cases because the number of bits being reversed differs. Also note that after the reordering, the first half of the numbers are all even and the second half are all odd.

**Figure 12.9** The butterfly operations performing a FFT on the eight data on the left leading to eight transforms on the right. The transforms are different linear combinations of the input data.

The fact that the Fourier transforms are produced in an order corresponding to the bit-reversed order of the numbers 0–7 suggests that if we process the data in the bit-reversed order 0, 4, 2, 6, 1, 5, 3, 7, then the output Fourier transforms will be ordered (see Table 10.1). We demonstrate this conjecture in Figure 12.10, where we see that to obtain the Fourier transform for the eight input data, the butterfly operation had to be applied three times. The number 3 occurs here because it is the power of 2 that gives the number of data; that is, 23 = 8. In general, in order for a FFT algorithm to produce transforms in the proper order, it must reshuffle the input data into bit-reversed order. As a case in point, our sample program starts by reordering the 16 (24) data elements given in Table 12.1. Now the four butterfly operations produce sequentially ordered output.

**Table 12.1** Reordering for 16 data complex points.

Order | Input data | New order
---|---|---
0 | 0.0 + 0.0i | 0.0 + 0.0i
1 | 1.0 + 1.0i | 8.0 + 8.0i
2 | 2.0 + 2.0i | 4.0 + 4.0i
3 | 3.0 + 3.0i | 12.0 + 12.0i
4 | 4.0 + 4.0i | 2.0 + 2.0i
5 | 5.0 + 5.0i | 10.0 + 10.i
6 | 6.0 + 6.0i | 6.0 + 6.0i
7 | 7.0 + 7.0i | 14.0 + 14.0i
8 | 8.0 + 8.0i | 1.0 + 1.0i
9 | 9.0 + 9.0i | 9.0 + 9.0i
10 | 10.0 + 10.i | 5.0 + 5.0i
11 | 11.0 + 11.0i | 13.0 + 13.0i
12 | 12.0 + 12.0i | 3.0 + 3.0i
13 | 13.0 + 13.0i | 11.0 + 11.0i
14 | 14.0 + 14.i | 7.0 + 7.0i
15 | 15.0 + 15.0i | 15.0 + 15.0i
Binary-Reversed 0–7 | Binary-Reversed 0–16
---|---
Dec | Bin | Rev | Dec Rev | Rev | Dec Rev
0 | 000 | 000 | 0 | 0000 | 0
1 | 001 | 100 | 4 | 1000 | 8
2 | 010 | 010 | 2 | 0100 | 4
3 | 011 | 110 | 6 | 1100 | 12
4 | 100 | 001 | 1 | 0010 | 2
5 | 101 | 101 | 5 | 1010 | 10
6 | 110 | 011 | 3 | 0110 | 6
7 | 111 | 111 | 7 | 1110 | 14
8 | 1000 | — | — | 0001 | 1
9 | 1001 | — | — | 1001 | 9
10 | 1010 | — | — | 0101 | 5
11 | 1011 | — | — | 1101 | 13
12 | 1100 | — | — | 0011 | 3
13 | 1101 | — | — | 1011 | 11
14 | 1101 | — | — | 0111 | 7
15 | 1111 | — | — | 1111 | 15

## 12.10 FFT Implementation

The first FFT program we are aware of was written in Fortran IV by Norman Brenner at MIT's Lincoln Laboratory (Higgins, 1976) and was hard for us to follow. Our (easier-to-follow) Python version is in Listing 12.3. Its input is _N_ = 2 _n_ data to be transformed (FFTs always require that the number of input data are a power of 2). If the number of your input data is not a power of 2, then you can make it so by concatenating some of the initial data to the end of your input until a power of 2 is obtained; because a DFT is always periodic, this just starts the period a little earlier. Our program assigns complex numbers at the 16 data points

**Figure 12.10** A modified FFT in which the eight input data on the left are transformed into eight transforms on the right. The results are the same as in the previous figure, but now the output transforms are in numerical order whereas in the previous figure the input signals were in numerical order.

(12.101)

reorders the data via bit reversal, and then makes four butterfly operations. The data are stored in the array `**dtr[max,2]**`, with the second subscript denoting real and imaginary parts. We increase the speed further by using the 1D array data to make memory access more direct:

(12.102)

which also provides storage for the output. The FFT transforms data using the butterfly operation and stores the results back in `**dtr[,]**`, where the input data were originally.

## 12.11 FFT Assessment

  1. Compile and execute `**FFT.py**`. Make sure you understand the output.
  2. Take the output from `**FFT.py**`, inverse-transform it back to signal space, and compare it to your input. (Checking that the double transform is proportional to itself is adequate, although the normalization factors in (12.37) should make the two equal.)
  3. Compare the transforms obtained with a FFT to those obtained with a DFT (you may choose any of the functions studied before). Make sure to compare both precision and execution times.

**Listing 12.3** **FFT.py** computes the FFT or inverse transform depending upon the sign of `**isign**`.

    1) We remind the reader that every periodic system by definition has a period _T_ and consequently a true frequency _ω_. Nonetheless, this does not imply that the system behaves like sin _ωt_. Only harmonic oscillators do that. 2) We follow convention and consider time _t_ the function's variable and frequency _ω_ the transform's variable. Nonetheless, these can be reversed or other variables such as position _x_ and wave vector _k_ may also be used. 3) Recall that  , and with the law of linear superposition this means that the real part of _y_ gives the cosine series, and the imaginary part the sine series. 4) The alert reader may be wondering what has happened to the _h_ /2 with which the trapezoid rule weights the initial and final points. Actually, they are there, but because we have set   terms have been added to produce one _h_ term. 5) See also Section 12.5.1 where we discuss the related phenomenon of aliasing. 6) We use natural units in which  . 7) Actually, this algorithm has been discovered a number of times, for instance, in 1942 by Danielson and Lanczos (Danielson and Lanczos, 1942), as well as much earlier by Gauss.

# 13
Wavelet and Principal Components Analyses: Nonstationary Signals and Data Compression

* * *

There are a number of techniques that extend Fourier analysis to signals whose forms change in time. This chapter introduces _wavelet analysis_ , a field that has seen extensive development and application in areas as diverse as brain waves, stock-market trends, gravitational waves, and compression of photographic images. The first part of the chapter deals with wavelet basics, and covers the essential materials. The second part of the chapter explores the discrete wavelet transform, and is marked as optional as a result of its technical nature. However, it is a beautiful bit of analysis and the basis of much of the digital revolution. The chapter ends with a discussion of principal component analysis, another powerful technique for analyzing signals with space and time correlations.

* * *

## 13.1 Problem: Spectral Analysis of Nonstationary Signals

**Problem** You have sampled the signal in Figure 13.1 that seems to contain an increasing number of frequencies as time increases. Your _problem_ is to undertake a spectral analysis of this signal that tells you, in the most compact way possible, the amount of each frequency present at each instant of time. _Hint:_ Although we want the method to be general enough to work with numerical data, for pedagogical purposes it is useful to know that the signal is

(13.1)

## 13.2 Wavelet Basics

The Fourier analysis we used in Chapter 12 reveals the amount of the harmonic functions sin( _ωt_ ) and cos( _ωt_ ), and their overtones, that are present in a signal. An expansion in periodic functions is fine for _stationary_ signals (those whose forms do not change in time) but has shortcomings for the variable form of our _problem_ signal (13.1). One such problem is that the Fourier reconstruction has all constituent frequencies occurring simultaneously and so does not contain _time resolution_ information indicating when each frequency occurs. Another shortcoming is that all the Fourier components are correlated, which results in more information being stored than may be needed to reconstruct the measured signal.

**Figure 13.1** The input time signal (13.1) we wish to analyze. The signal is seen to contain additional frequencies as time increases. The boxes are possible placements of windows for short-time Fourier transforms.

There are a number of techniques that extend simple Fourier analysis to non-stationary signals. The idea behind wavelet analysis is to expand a signal in a complete set of functions (wavelets), each of which oscillates for a finite period of time, and each of which is centered at a different time. To give you a preview before we get into the details, we show four sample wavelets in Figure 13.2. Because each wavelet is local in time, it is a wave packet,1) with its time localization leading to a spectrum with a range of frequencies. These wave packets are called "wavelets" because they exist for only short periods of time (Polikar, 2001).

Although wavelets are required to oscillate in time, they are not restricted to a particular functional form (Addison, 2002; Goswani and Chan, 1999; Graps, 1995). As a case in point, they may be oscillating Gaussians (Morlet: in Figure 13.2a),

(13.2)

the second derivative of a Gaussian (Mexican hat, Figure 13.2b),

(13.3)

an up-and-down step function (Figure 13.2c), or a fractal shape (Figure 13.2d). All of these wavelets are _localized_ in both time and frequency, that is, they are large for just a limited time and contain a limited range of frequencies. As we shall see, translating and scaling a _mother wavelets_ generates an entire set of _daughter wavelet_ or basis functions, with the daughters covering different frequency ranges at different times.

**Figure 13.2** Four possible mother wavelets that can be used to generate entire sets of daughter wavelets. (a–d) Morlet (real part), Mexican hat, Daub4 e6 (explained later), and Haar. The daughter wavelets are generated by scaling and translating these mother wavelets.

## 13.3 Wave Packets and Uncertainty Principle (Theory)

A _wave packet_ or _wave train_ is a collection of waves of differing frequencies added together in such a way as to produce a pulse of width Δ _t_. As we shall see, the Fourier transform of a wave packet is a pulse in the frequency domain of width Δ _ω_. We will first study such wave packets analytically, and then use others numerically. An example of a simple wave packet is just a sine wave that oscillates at frequency _ω_ 0 for _N_ periods (Figure 13.3a) (Arfken and Weber, 2001)

(13.4)

where we relate the frequency to the period via the usual _ω_ 0 = 2π/ _T_. In terms of these parameters, the width of the wave packet is

(13.5)

The Fourier transform of the wave packet (13.4) is calculated via a straightforward application of the transform formula (12.19):

(13.6)

**Figure 13.3** (a) A wave packet in time corresponding to the functional form (13.4) with _ω_ 0 = 5 and _N_ = 6. (b) The Fourier transform in frequency of this same wave packet.

where we have dropped a factor of –i that affects only the phase. While at first glance (13.6) appears to be singular at _ω = ω_ 0, it actually just peaks there (Figure 13.3b), reflecting the predominance of frequency _ω_ 0. Note that although the signal _y_ ( _t_ ) appears to have only one frequency, it does drop off sharply in time (Figure 13.3a), and these corners give _Y_ ( _ω_ ) a width Δ _ω_.

There is a fundamental relation between the widths Δ _t_ and Δ _ω_ of a wave packet. Although we use a specific example to determine that relation, it is true in general. While there may not be a precise definition of "width" for all functions, one can usually deduce a good measure of the width (say, within 25%). To illustrate, if we look Figure 13.3b, it makes sense to use the distance between the first zeros of the transform _Y_ (ω) (13.6) as the frequency width Δ _ω_. The zeros occur at

(13.7)

where _N_ is the number of cycles in our original wave packet. Because the wave packet in time makes _N_ oscillations each of period _T_ , a reasonable measure of the time width Δ _t_ of the signal _y_ ( _t_ ) is

(13.8)

When the products of the frequency width (13.7) and the time width (13.8) are combined, we obtain

(13.9)

The greater than sign is used to indicate that this is a minimum, that is, that _y_ ( _t_ ) and _Y_ ( _ω_ ) extend beyond Δ _t_ and Δ _ω_ , respectively. Nonetheless, most of the signal and transform should lie within the bound (13.9).

A relation of the form (13.9) also occurs in quantum mechanics, where it is known as the _Heisenberg uncertainty principle_ , with Δ _t_ and Δ _ω_ being called the uncertainties in _t_ and _ω_. It is true for transforms in general and states that as a signal is made more localized in time (smaller Δ _t_ ) the transform becomes less localized (larger Δ _ω_ ). Conversely, the sine wave _y_ ( _t_ ) = sin _ω_ 0 _t_ is completely localized in frequency, and consequently has an infinite extent in time,  .

### 13.3.1 Wave Packet Assessment

Consider the following wave packets:

(13.10)

For each wave packet:

  1. Estimate the width Δ _t_. A good measure might be the _full width at half-maxima_ (FWHM) of | _y_ ( _t_ )|.
  2. Use your DFT program to evaluate and plot the Fourier transform _Y_ ( _ω_ ) for each wave packet. Make _both_ a linear and a semilog plot (small components are often important, yet not evident in linear plots). Make sure that your transform has a good number of closely spaced frequency values over a range that is large enough to show the periodicity of _Y_ ( _ω_ ).
  3. What are the units for _Y_ ( _ω_ ) and _ω_ in your DFT?
  4. For each wave packet, estimate the width Δ _ω_. A good measure might be the _full width at half-maxima_ of | _Y_ ( _ω_ )|.
  5. For each wave packet determine approximate value for the constant _C_ of the uncertainty principle

(13.11)

## 13.4 Short-Time Fourier Transforms (Math)

The constant amplitude of the functions sin _nωt_ and cos _nωt_ for all times can limit the usefulness of Fourier analysis for reproducing signals. Because these functions and their overtones extend over all times with a constant amplitude, there is considerable overlap among them, and consequently the information present in various Fourier components are correlated. This is undesirable for data storage and compression, where you want to store a minimum number of data information and also want to adjust the amount stored based on the desired quality of the reconstructed signal.2) In _lossless compression_ , which exactly reproduces the original signal, you save space by storing how many times each data element is repeated, and where each element is located. In _lossy compression_ , in addition to removing repeated elements, you also eliminate some transform components consistent with the uncertainty relation (13.9) and with the level of resolution required in the reproduction. This leads to yet greater compression.

In Section 12.5, we defined the Fourier transform _Y_ ( _ω_ ) of signal _y_ ( _t_ ) as

(13.12)

As is true for simple vectors, you can think of (13.12) as giving the overlap or scalar product of the basis function   and the signal _y_ ( _t_ ) (notice that the complex conjugate of the exponential basis function appears in (13.12)). Another view of (13.12) is as the mapping or projection of the signal _y_ into _ω_ space. In this latter case the overlap projects out the amount of the periodic function   in the signal. In other words, the Fourier component _Y_ ( _ω_ ) is also the correlation between the signal _y_ ( _t_ ) and the basis function  , which is the same as what results from filtering the signal _y_ ( _t_ ) through a frequency- _ω_ filter. If there is no exp(i _ωt_ ) in the signal, then the integral vanishes and there is no output. If  , the signal is at only one frequency, and the integral is accordingly singular.

The signal in Figure 13.1 for our problem clearly has different frequencies present at different times and for different lengths of time. In the past, this signal might have been analyzed with a precursor of wavelet analysis known as the _short-time Fourier transform_. With that technique, the signal _y_ ( _t_ ) is "chopped up" into different segments along the time axis, with successive segments centered about successive times _τ_ 1, _τ_ 2,..., _τ N_. For instance, we show three such segments in the boxes of Figure 13.1. Once we have the dissected signal, a Fourier analysis is made for each segment. We are then left with a sequence of transforms  , one for each short-time interval, where the superscript ST indicates short time.

Rather than chopping up a signal, we can express short-time Fourier transforming mathematically by imagining translating a _window function w_ ( _t – τ_ ), which is zero outside of some chosen interval, over the signal in Figure 13.1:

(13.13)

Here the values of the translation time _τ_ correspond to different locations of the window _w_ over the signal, and the window function is essentially a transparent box of small size on an opaque background. Any signal within the width of the window is transformed, while the signal lying outside the window is not seen. Note that in (13.13), the extra variable _τ_ in the Fourier transform indicates the location of the time around which the window was placed. Clearly, because the short-time transform is a function of two variables, a surface or 3D plot is needed to view the amplitude as a function of both _ω_ and _τ_.

## 13.5 The Wavelet Transform

The wavelet transform of a time signal _y_ ( _t_ ) is defined as

(13.14)

and is similar in concept and notation to a short-time Fourier transform. The difference is rather than using exp(i _ωt_ ) as the basis functions, here we are using wave packets or wavelets _ψ S,τ_( _t_ ) localized in time, such as the those shown in Figure 13.2. Because each wavelet is localized in time, each acts as its own window function. Because each wavelet is oscillatory, each contains its own small range of frequencies.

Equation 13.14 says that the wavelet transform _Y_ ( _s, τ_ ) is a measure of the amount of basis function _ψ s, τ_( _t_ ) present in the signal _y_ ( _t_ ). The _τ_ variable indicates the time portion of the signal being decomposed, while the _s_ variable is equivalent to the frequency present during that time:

(13.15)

Because it is key to much that follows, it is a good idea to think about (13.15) for a while. If we are interested in the time _details_ of a signal, then this is another way of saying that we are interested in what is happening at small values of the _scale s_. Equation 13.15 indicates that small values of s correspond to high-frequency components of the signal. That being the case, the time details of the signal are in the high-frequency, or low-scale, components.

### 13.5.1 Generating Wavelet Basis Functions

The conceptual discussion of wavelets is over, and it is time to get down to work. We first need a technique for generating wavelet basis functions, and then we need to discretize this technique. As is often the case, the final formulation will turn out to be simple and short, but it will be a while before we get there.

Just as the expansion of an arbitrary function in a complete set of orthogonal functions is not restricted to any particular set, so too is the wavelet transform not restricted to any particular wavelet basis, although some might be better than others for a given signal. The standard way to generate a family of wavelet basis functions starts with _Ψ_ ( _t_ ), a _mother_ or _analyzing_ function of the real variable _t_ , and then to use this to generate _daughter_ wavelets. As a case in point, we start with the mother wavelet

(13.16)

By scaling, translating, and normalizing this mother wavelet,

(13.17)

we generate the four wavelet basis functions (daughters) displayed in Figure 13.4. We see that larger or smaller values of _s_ , respectively, expand or contract the mother wavelet, while different values of _τ_ shift the center of the wavelet. Because the wavelets are inherently oscillatory, the scaling leads to the same number of oscillations occurring in different time spans, which is equivalent to having basis states with differing frequencies. We see that _s_ < 1 produces a higher frequency wavelet, while _s_ > 1 produces a lower frequency one, both of the same shape. As we shall see, we do not need to store much information to outline the large-time-scale _s_ behavior of a signal (its _smooth envelope_ ), but we do need more information to specify its short-time-scale _s_ behavior ( _details_ ). And if we want to resolve yet finer features in the signal, then we will need to have more information on yet finer details. Here the division by   is made to ensure that there is equal "power" (or energy or intensity) in each region of _s_ , although other normalizations can also be found in the literature. After substituting in the definition of daughters, the wavelet transform (13.14) and its inverse (van den Berg, 1999) are

(13.18)

(13.19)

where the normalization constant _C_ depends on the wavelet used. In summary, wavelet bases are functions of the time variable _t_ , as well as of the two parameters _s_ and _τ_. The _t_ variable is integrated over to yield a transform that is a function of the time scale _s_ (frequency 2π/ _s_ ) and window location _τ_. You can think of scale as being like the scale on a map (also discussed in Section 16.5.2 in relation to fractal analysis) or in terms of _resolution_ , as might occur in photographic images. Regardless of the words, as we see in Chapter 16, if we have a fractal, then we have a self-similar object that looks the same at all scales or resolutions. Similarly, each wavelet in a set of basis functions is self-similar to the others, but at a different scale or location. The general requirements for a mother wavelet _Ψ_ are (Addison, 2002; van den Berg, 1999)

  1. _Ψ_ ( _t_ ) is real.
  2. _Ψ_ ( _t_ ) oscillates around zero such that its average is zero:

(13.20)

**Figure 13.4** Four wavelet basis functions (daughters) generated by scaling ( _s_ ) and translating ( _τ_ ) an oscillating Gaussian mother wavelet. (a–d) (s = 1, _τ_ = 0), ( _s_ = 1/2, _τ_ = 0), ( _s_ = 1, _τ_ = 6), and ( _s_ = 2, _τ_ = 60). Note how _s_ < 1 is a wavelet with higher frequency, while _s_ > 1 has a lower frequency than the _s_ = 1 mother. Likewise, the _τ_ = 6 wavelet is just a translated version of the _τ_ = 0 one directly above it.

  3. _Ψ_ ( _t_ ) is local, that is, a wave packet, and is square integrable:

(13.21)

  4. The transforms of low powers of _t_ vanish, that is, the first _p_ moments:

(13.22)

This makes the transform more sensitive to details than to general shape.

As an example of how we use the _s_ and _τ_ degrees of freedom in a wavelet transform, consider the analysis of a chirp signal _y_ ( _t_ ) = sin(60 _t_ 2) (Figure 13.5). We see that a slice at the beginning of the signal is compared to our first basis function. (The comparison is carried out via the _convolution_ of the wavelet with the signal.) This first comparison is with a narrow version of the wavelet, that is, at low scale, and yields a single coefficient. The comparison at this scale continues with the next signal slice, and eventually ends when the entire signal has been covered (the top row in Figure 13.5). Then in the second row, the wavelet is expanded to larger _s_ values, and comparisons are repeated. Eventually, the data are processed at all scales and at all time intervals. The narrow wavelets correspond to a high-resolution analysis, while the broad wavelets correspond to low resolution. As the scales get larger (lower frequencies, lower resolution), fewer details of the time signal remain visible, but the overall shape or gross features of the signal become clearer.

**Figure 13.5** A schematic representation of the steps followed in performing a wavelet transformation over all time displacements and scales. The upper signal is first analyzed by evaluating its overlap with a narrow wavelet at the signal's beginning. This produces a coefficient that measures the similarity of the signal to the wavelet. The wavelet is successively shifted over the length of the signal and the overlaps are successively evaluated. After the entire signal is covered, the wavelet is expanded and the entire analysis is repeated.

### 13.5.2 Continuous Wavelet Transform Implementation

We want to develop some intuition as to what wavelet transforms look like before going on to apply them in unknown situations and to develop a discrete algorithm. Accordingly, modify the program you have been using for the Fourier transform so that it now computes the continuous wavelet transform.

  1. You will want to see the effect of using different mother wavelets. Accordingly, write a method that calculates the mother wavelet for
    1. a) a Morlet wavelet (13.2),
    2. b) a Mexican hat wavelet (13.3),
    3. c) a Haar wavelet (the square wave in Figure 13.2).
  2. Try out your transform for the following input signals and see if the results make sense:
    1. a) A pure sine wave _y_ ( _t_ ) = sin 2π _t_ ,
    2. b) A sum of sine waves _y_ ( _t_ ) = 2.5 sin 2π _t_ \+ 6 sin 4π _t_ \+ 10 sin 6π _t_ ,
    3. c) The nonstationary signal for our problem (13.1)

(13.23)

**Figure 13.6** Comparison of an input and reconstituted signal (13.23) using Morlet wavelets (the curves overlap nearly perfectly). As expected for Fourier transforms, the reconstruction is least accurate near the endpoints.

    4. d) The half-wave function

(13.24)

  3.   Use (13.19) to invert your wavelet transform and compare the reconstructed signal to the input signal (you can normalize the two to each other). In Figure 13.6 we show our reconstruction.

In Listing 11.1, we give our _continuous wavelet transformation_ `CWT.py` (Lang and Forinash, 1998). Because wavelets, with their transforms in two variables, are somewhat hard to grasp at first, we suggest that you write your own code and include a portion that does the inverse transform as a check. In the next section we will describe the _discrete wavelet transformation_ that makes optimal discrete choices for the scale and time translation parameters _s_ and _τ_. Figure 13.7 shows a surface plot of the spectrum produced for the input signal (13.1) in Figure 13.1. In realization of our goal, we see predominantly one frequency at short times, two frequencies at intermediate times, and three frequencies at longer times.

## 13.6 Discrete Wavelet Transforms, Multiresolution Analysis

As was true for DFTs, if a time signal is measured at only _N_ discrete times,

(13.25)

then we can determine only _N_ independent components of the transform _Y_. The trick with wavelets is to remain consistent with the uncertainty principle as we compute only the _N_ independent components required to reproduce the signal. The _discrete wavelet transform_ (DWT) evaluates the transforms with discrete values for the scaling parameter _s_ and the time translation parameter _τ_ :

**Figure 13.7** The continuous wavelet spectrum obtained by analyzing the input signal with Morelet wavelets. Observe how at small values of time _τ_ there is predominantly one frequency present, how a second, higher frequency (smaller scale) component enters at intermediate times, and how at larger times a still higher frequency components enter (figure courtesy of `Z. Dimcovic`).

(13.26)

(13.27)

Here _j_ and _k_ are integers whose maximum values are yet to be determined, and we have assumed that the total time interval _T_ = 1, so that time is always measured in fractions. This choice of _s_ and _τ_ based on powers of 2 is called a _dyadic grid_ arrangement and will be seen to automatically perform the scalings and translations at the different time scales that are at the heart of wavelet analysis.3) The DWT now becomes

(13.28)

where the discreteness here refers to the wavelet basis set and _not_ the time variable. For an orthonormal wavelet basis, the inverse discrete transform is then

(13.29)

This inversion will exactly reproduce the input signal at the _N_ input points, but only if we sum over an infinite number of terms (Addison, 2002). Practical calculations will be less exact.

**Listing 13.1** `**CWT.py**` computes a normalized continuous wavelet transform of the signal data in input (here assigned as a sum of sine functions) using Morlet wavelets (courtesy of `Z. Dimcovic`). The discrete wavelet transform (DWT) is faster and yields a compressed transform, but is less transparent.

Note in (13.26) and (13.28) that we have kept the time variable _t_ in the wavelet basis functions continuous, despite the fact that _s_ and _τ_ have been made discrete. This is useful in establishing the orthonormality of the basis functions

(13.30)

where _δ m,n_ is the Kronecker delta function. Being normalized to 1 means that each wavelet basis has "unit energy"; being orthogonal means that each basis function is independent of the others. And because wavelets are localized in time, the different transform components have low levels of correlation with each other. Altogether, this leads to efficient and flexible data storage.

The use of a discrete wavelet basis makes it clear that we sample the input signal at the discrete values of time determined by the integers _j_ and _k_. In general, you want time steps that sample the signal at enough times in each interval to obtain the desired level of precision. A rule of thumb is to start with 100 steps to cover each major feature. Ideally, the needed times correspond to the times at which the signal was sampled, although this may require some forethought.

Consider Figure 13.8. We measure a signal at a number of discrete times within the intervals ( _k_ or _τ_ values) corresponding to the vertical columns of fixed width along the time axis. For each time interval we want to sample the signal at a number of scales (frequencies or _j_ values). However, as discussed in Section 13.3, the basic mathematics of Fourier transforms indicates that the width Δ _t_ of a wave packet _ψ_ ( _t_ ) and the width Δ _ω_ of its Fourier transform _Y_ ( _ω_ ) are related by an uncertainty principle

This relation places a constraint on the intervals in which can measure times based on the intervals in which we deduce frequencies. So while we may want a high-resolution reproduction of our signal, we do not want to store more data than are needed to obtain that reproduction. If we sample the signal for times centered about some _τ_ in an interval of width Δ _τ_ (Figure 13.8) and then compute the transform at a number of scales _s_ or frequencies   covering a range of height Δ _ω_ , then the relation between the height and width is restricted by the uncertainty relation, which means that each of the rectangles in Figure 13.8 has the same area Δ _ω_ Δ _t_ = 2π. The increasing heights of the rectangles at higher frequencies means that a larger range of frequencies should be sampled as the frequency increases. The premise here is that the low-frequency components provide the gross or _smooth_ outline of the signal which, being smooth, does not require much detail, while the high-frequency components give the details of the signal over a short time interval and so require many components in order to record these details with high resolution.

**Figure 13.8** A graphical representation of the relation between time and frequency resolutions (the uncertainty relation). Each box represents an equal portion of the time-frequency plane but with different proportions of time and frequency.

**Figure 13.9** A multifrequency dyadic (power-of-2) filter tree used for discrete wavelet transformations. The L boxes represent low-pass filters and the H boxes represent high-pass filters. Each filter performs a convolution (transform). The circles containing "↓ 2" filter out half of the signal that enters them, which is called _subsampling_ or _factor-of-2 decimation_. The signal on the left yields a transform with a single low and two high components (less information is needed about the low components for a faithful reproduction).

Industrial-strength wavelet analyses do not compute explicit integrals, but instead apply a technique known as _multiresolution analysis_ (MRA) (Mallat, 1989). We give an example of this technique in Figure 13.9 and in the code DWT.py in Listing 13.2. It is based on a _pyramid algorithm_ that samples the signal at a finite number of times, and then passes it successively through a number of _filters_ , with each filter representing a digital version of a wavelet.

Filters were discussed in Section 12.8, where in (12.67) we defined the action of a linear filter as a convolution of the filter response function with the signal. A comparison of the definition of a filter to the definition of a wavelet transform (13.14) shows that the two are essentially the same. Such being the case, the result of the transform operation is a weighted sum over the input signal values, with each weight the product of the integration weight times the value of the wavelet function at the integration point. Therefore, _rather than tabulate explicit wavelet functions, a set of filter coefficients is all that is needed for DWTs_.

Because each filter in Figure 13.9 changes the relative strengths of different frequency components, passing the signal through a series of filters is equivalent, in wavelet language, to analyzing the signal at different scales. This is the origin of the name "multiresolution analysis." Figure 13.9 shows how the pyramid algorithm passes the signal through a series of high-pass filters (H) and then through a series of low-pass filters (L). Each filter changes the scale to that of the level below. Also note that the circles containing ↓ 2 in Figure 13.9. This operation filters out half of the signal and so is called _subsampling_ or _factor-of-2 decimation_. It is the way we keep the areas of each box in Figure 13.8 constant as we vary the scale and translation times. We consider subsampling further when we discuss the pyramid algorithm.

In summary, the DWT process decomposes the signal into _smooth_ information stored in the low-frequency components and _detailed_ information stored in the high-frequency components. Because _high-resolution_ reproductions of signals require more information about details than about gross shape, the pyramid algorithm is an effective way to compress data while still maintaining high resolution. In addition, because components of different resolutions are independent of each other, it is possible to lower the number of data stored by systematically eliminating higher resolution components. And finally, the use of wavelet filters builds in progressive scaling, which is particularly appropriate for fractal-like reproductions.

### 13.6.1 Pyramid Scheme Implementation

We now implement the pyramid scheme outlined in Figure 13.9. The _H_ and _L_ filters will be represented by matrices, which is an approximate way to perform the integrations or convolutions. Then there is a decimation of the output by one-half, and finally an interleaving of the output for further filtering. This process simultaneously cuts down on the number of points in the data set and changes the scale and the resolution. The decimation reduces the number of values of the remaining signal by one half, with the low-frequency part discarded because the details are in the high-frequency parts. As indicated in Figure 13.10, the pyramid DWT algorithm follows five steps:

  1. Successively applies the (soon-to-be-derived) _c_ matrix (13.41) to the whole _N_ -length vector

(13.31)

  2. Applies it to the _N_ /2-length smooth vector.
  3. Repeats the application until only two smooth components remain.
  4. After each filtering, the elements are ordered, with the newest two smooth elements on top, the newest detailed elements below, and the older detailed elements below that.
  5. The process continues until there are just two smooth elements left.

**Figure 13.10** An input signal at the top is processed by a tree of high- and low-band filters. The outputs from each filtering are down-sampled with half the data kept. The process continues until there are only two data of high-band filtering and two data of low-band filtering. When complete, the total number of output data equals the total number of signal data. The process of wavelet analysis is thus equivalent to a series of filterings.

To illustrate, here we filter and reorder an initial vector of length _N_ = 8:

(13.32)

The discrete inversion of a transform vector back to a signal vector is made using the transpose (inverse) of the transfer matrix at each stage. For instance,

(13.33)

As a more realistic example, imagine that we have sampled the chirp signal _y_ ( _t_ ) = sin(60 _t_ 2) for 1024 times. The filtering process through which we place this signal is illustrated as a passage from the top to the bottom in Figure 13.10. First the original 1024 samples are passed through a single low band and a single high band (which is mathematically equivalent to performing a series of convolutions). As indicated by the down arrows, the output of the first stage is then downsampled, that is, the number is reduced by a factor of 2. This results in 512 points from the high-band filter as well as 512 points from the low-band filter. This produces the first-level output. The output coefficients from the high-band filters are called   to indicate that they show details, and   that they show smooth features. The superscript indicates that this is the first level of processing. The detail coefficients   are stored to become part of the final output.

In the next level down, the 512 smooth data   are passed through new low-and high-band filters using a broader wavelet. The 512 outputs from each are downsampled to form a smooth sequence   of size 256 and a detailed sequence   of size 256. Again the detail coefficients   are stored to become part of the final output. (Note that this is only half the size of the previously stored details.) The process continues until there are only two numbers left for the detail coefficients and two numbers left for the smooth coefficients. Because this last filtering is carried out with the broadest wavelet, it is of the lowest resolution and therefore requires the least information.

In Figure 13.11, we show the actual effects on the chirp signal of pyramid filtering for various levels in the processing. (The processing is carried out with four-coefficient _Daub4_ wavelets, which we will discuss soon.) At the uppermost level, the wavelet is narrow, and so convoluting this wavelet with successive sections of the signal results in smooth components that still contain many large high-frequency parts. The detail components, in contrast, are much smaller in magnitude. In the next stage, the wavelet is dilated to a lower frequency and the analysis is repeated _on just the smooth (low-band) part_. The resulting output is similar, but with coarser features for the smooth coefficients and larger values for the details. Note that in the upper graphs we have connected the points to make the output look continuous, while in the lower graphs, with fewer points, we have plotted the output as histograms to make the points more evident. Eventually the downsampling leads to just two coefficients output from each filter, at which point the filtering ends.

**Figure 13.11** In successive passes, the filtering of the original signal at the top goes through the pyramid algorithm and produces the outputs shown. The sampling is reduced by a factor of 2 in each step. Note that in the upper graphs, we have connected the points to emphasize their continuous nature while in the lower graphs we plot the individual output points as histograms.

To reconstruct the original signal (called _synthesis_ or _transformation_ ) a reversed process is followed: Begin with the last sequence of four coefficients, upsample them, pass them through low- and high-band filters to obtain new levels of coefficients, and repeat until all the _N_ values of the original signal are recovered. The inverse scheme is the same as the processing scheme (Figure 13.10), only now the directions of all the arrows are reversed.

### 13.6.2 Daubechies Wavelets via Filtering

We should now be able to understand that digital wavelet analysis has been standardized to the point where classes of wavelet basis functions are specified not by their analytic forms, but rather by their _wavelet filter coefficients_. In 1988, the Belgian mathematician Ingrid Daubechies discovered an important class of such filter coefficients (Daubechies, 1995; Rowe and Abbott, 1995). We will study just the Daub4 class containing the four coefficients _c_ 0, _c_ 1, _c_ 2, and _c_ 3.

Imagine that our input contains the four elements   corresponding to measurements of a signal at four times. We represent a low-pass filter _L_ and a high-pass filter _H_ in terms of the four filter coefficients as

(13.34)

(13.35)

To see how this works, we form an input vector by placing the four signal elements in a column and then multiply the input by _L_ and _H_ :

We see that if we choose the values of the _c_ _i_ 's carefully, the result of _L_ acting on the signal vector is a single number that may be viewed as a weighted average of the four input signal elements. Because an averaging process tends to smooth out data, the low-pass filter may be thought of as a _smoothing filter_ that outputs the general shape of the signal.

In turn, we see that if we choose the _c i_ values carefully, the result of _H_ acting on the signal vector is a single number that may be viewed as the weighted differences of the input signal. Because a differencing process tends to emphasize the variation in the data, the high-pass filter may be thought of as a _detail_ filter that produces a large output when the signal varies considerably, and a small output when the signal is smooth.

We have just seen how the individual _L_ and _H_ filters, each represented by a single row of the filter matrix, outputs one number when acting upon an input signal containing four elements in a column. If we want the output of the filtering process _Y_ to contain the same number of elements as the input (four _y_ 's in this case), we just stack the _L_ and _H_ filters together:

(13.36)

Of course, the first and third rows of the **_Y_** vector will be identical, as will the second and fourth, but we will take care of that soon.

Now we go about determining the values of the filter coefficients _c_ _i_ by placing specific demands upon the output of the filter. We start by recalling that in our discussion of discrete Fourier transforms we observed that a transform is equivalent to a rotation from the time domain to the frequency domain. Yet we know from our study of linear algebra that rotations are described by orthogonal matrices, that is, matrices whose inverses are equal to their transposes. In order for the inverse transform to return us to the input signal, the transfer matrix must be orthogonal. For our wavelet transformation to be orthogonal, we must have the 4×4 filter matrix times its transpose equal to the identity matrix:

(13.37)

Two equations in four unknowns are not enough for a unique solution, so we now include the further requirement that the detail filter   must output a zero if the input is smooth. We define "smooth" to mean that the input is constant or linearly increasing:

(13.38)

This is equivalent to demanding that the moments up to order _p_ are zero, that is, that we have an "approximation of order _p_." Explicitly,

(13.39)

(13.40)

These are the basic Daub4 filter coefficients. They are used to create larger filter matrices by placing the row versions of _L_ and _H_ along the diagonal, with successive pairs displaced two columns to the right. For example, for eight elements

(13.41)

Note that in order not to lose any information, the last pair on the bottom two rows is wrapped over to the left. If you perform the actual multiplications indicated in (13.41), you will note that the output has successive _smooth_ and _detailed_ information. The output is processed with the pyramid scheme.

The time dependences of two Daub4 wavelets is displayed in Figure 13.12. To obtain these from our filter coefficients, first imagine that an elementary wavelet   input into the filter. This should result in a transform _Y_ 1,l = 1. Inversely, we obtain _y_ 1,1( _t_ ) by applying the inverse transform to a _Y_ vector with a 1 in the first position and zeros in all the other positions. Like-wise, the _i_ th member of the Daubechies class is obtained by applying the inverse transform to a **_Y_** vector with a 1 in the _i_ th position and zeros in all the other positions.

**Figure 13.12** (a) The Daub4 e6 wavelet constructed by inverse transformation of the wavelet coefficients. This wavelet has been found to be particularly effective in wavelet analyses. (b) The sum of Daub4 e10 and Daub4 1e58 wavelets of different scale and time displacements.

The wavelet for coefficient 6 (thus the e6 notation) is shown in Figure 13.12a and in Figure 13.12b the sum of two wavelets corresponding to the coefficients 10 and 58. We see that the two wavelets have different levels of scale as well as different time positions. So despite the fact that the time dependence of the wavelets is not evident when wavelet (filter) coefficients are used, it is there.

### 13.6.3 DWT Implementation and Exercise

Listing 13.2 gives our program for performing a DWT on the chirp signal _y_ ( _t_ ) = sin(60 _t_ 2). The method pyram calls the `daube4` method to perform the DWT or inverse DWT, depending upon the value of `sign`.

  1. Modify the program so that you output to a file the values for the input signal that your code has read in. It is always important to check your input.
  2. Try to reproduce Figure 13.11 by using various values for the variable nend that controls when the filtering ends. A value `nend=1024` should produce just the first step in the downsampling (top row in Figure 13.11). Selecting `nend=512` should produce the next row, while `nend=4` should output just two smooth and detailed coefficients.
  3. Reproduce the scale-time diagram shown on the right of Figure 13.11. This diagram shows the output at different scales and serves to interpret the main components of the signal and the time in which they appear. The time line at the bottom of the figure corresponds to a signal of length 1 over which 256 samples were recorded. The low-band (smooth) components are shown on the left, and the high-band components on the right.
    1. a) The bottommost figure results when `nend` = 256.
    2. b) The figure in the second row up results from `nend` = 128, and we have the output from two filterings. The output contains 256 coefficients but divides time into four intervals and shows the frequency components of the original signal in more detail.
    3. c) Continue with the subdivisions for `nend` = 64, 32, 16, 8, and 4.
  4. For each of these choices except the topmost, divide the time by 2 and separate the intervals by vertical lines.
  5. The topmost spectrum is your final output. Can you see any relation between it and the chirp signal?
  6. Change the sign of sign and check that the inverse DWT reproduces the original signal.
  7. Use the code to visualize the time dependence of the Daubechies mother function at different scales.
    1. a) Start by performing an inverse transformation on the eight-component signal `[0,0,0,0,1,0,0,0]`. This hould yield a function with a width of about 5 units.
    2. b) Next perform an inverse transformation on a unit vector with _N_ = 32 but with all components except the fifth equal to zero. The width should now be about 25 units, a larger scale but still covering the same time interval.
    3. c) Continue this procedure until you obtain wavelets of 800 units.
    4. d) Finally, with _N_ = 1024, select a portion of the mother wavelet with data in the horizontal interval [590,800]. This should show some self similarity.

**Listing 13.2** `**DWT.py**` computes the DWT using the pyramid algorithm for the 2 _n_ signal values stored in `f[ ]` (here assigned as the chirp signal sin 60 _t_ 2). The Daub4 digital wavelets are the basis functions, and `sign = ±1` for transform/inverse.

## 13.7 Principal Components Analysis

We have seen that Fourier analysis has a shortcoming of having all of its components correlated. This slows down the calculation of transforms and makes compression and reconstitution of data problematic. Wavelets, on the other hand, appear excellent at data compression, but not appropriate for high-dimensionality data sets or for all physical situations. Principal components analysis (PCA) is excellent for situations in which there are correlation among the variable in the data, and especially for the type of space-time correlations as might be found in brain waves, facial patterns and ocean currents. The same basic PCA approach is used in many fields with names such as the Karhunen–Loève transform, the Hotelling transform, the proper orthogonal decomposition, singular value decomposition, factor analysis, empirical orthogonal functions, empirical component analysis, empirical modal analysis, and so forth (Wikipedia, 2014).

**Figure 13.13** (a) The normalized data and eigenvectors of covariance matrix. (b) The normalized data using the PCA eigenvectors as basis.

A key element in PCA is viewing a collection of data as defining a multidimensional _data space_ , and then finding a few basic components in those data that contain most of the "power". For example, imagine the output of 32 detectors of magnetic brain waves recorded every tenth of a second for an hour. In this case there will be 33 space-time dimensions to the data and 10 × 60 × 60 × 32 = 1152 000 data elements. And so we are now motivated to transform from the detector-time interval basis to a new set of basis functions known as the _principal components_ which are pretty much guaranteed to concentrate most of the signal strength into a few components (Jackson, 1991; Jolliffe, 1991; Smith, 2002). This is analogous to the principal axes theorem of mechanics, in which the description of the rotation of a solid object is greatly simplified if moments of inertia relative to the principal axes are used.

As follows from the example that we will work out soon, on the left of Figure 13.13, we see a bunch of data point along with the two principal component eigenvectors for these data. This figure shows how the first component accounts for most of the variability of the data (has largest possible variance), while the next component is orthogonal, and thus not correlated with the first component. The second component accounts for much less of the data, namely the most possible variance in the data within the constraint of being orthogonal to the first component.

**Table 13.1** PCA data.

Data | Adjusted data | In PCA basis
---|---|---
_x_ | _y_ | _x_ | _y_ | _x_ 1 | _x_ 2
2.5 | 2.4 | 0.69 | 0.49 | −0.828 | −0.175
0.5 | 0.7 | −1.31 | −1.21 | 1.78 | 0.143
2.2 | 2.9 | 0.39 | 0.99 | −0.992 | 0.484
1.9 | 2.2 | 0.09 | 0.29 | −0.274 | 0.130
3.1 | 3.0 | 1.29 | 1.09 | −1.68 | −0.209
2.3 | 2.7 | 0.49 | 0.79 | 0.913 | 0.175
2 | 1.6 | 0.19 | −0.31 | 0.0991 | −0.350
1.0 | 1.1 | −0.81 | −0.81 | 1.14 | 0.464
1.6 | 1.6 | −0.31 | −0.31 | 0.438 | 0.0178
1.1 | 0.9 | −0.71 | −1.01 | 1.22 | −0.163

### 13.7.1 Demonstration of Principal Component Analysis

The derivation of PCA and proofs of its theorems can get quite involved. Instead, as our introduction to the subject we take a purely operational approach and work through a simple PCA analysis following the example of (Smith, 2002). We assume that the data have two dimension, and we call these _x_ and _y_ , but they need not be related to spatial positions.

**1. Enter Data** We start with a data set, in our case the first two columns of Table 13.1.

**2. Subtract the Mean** PCA analysis assumes that the data in each dimension has zero mean. Accordingly, as shown in columns two and three in Table 13.1, we calculate the mean for each column,  , and then subtract them from the data in each column. The resulting adjusted data are given in the third and fourth columns of the table, and placed here into a data matrix:

(13.42)

**3. Calculate Covariance Matrix** Recall that for a data set with _N_ members, the _variance_ is a measure of the deviation of the data from their mean:

(13.43)

When a data set contains multiple dimensions (variables), there may well be a dependence of the data in one-dimensional variable to that in another. The _covariance_ gives a measure of how much the deviation of one variable from the mean varies with the deviation of another variable from the mean:

(13.44)

so that a positive covariance indicates that the _x_ and _y_ variables tend to change together in the same direction. Also, we see that the variance (13.43) can be viewed as a special case of the covariance, var( _x_ ) = cov( _x, x_ ), and that there is a symmetry here with cov( _x, y_ ) = cov( _y,x_ ). All of these possible covariance values are combined into a symmetric covariance matrix, which for our 2×2 case is just

(13.45)

The next step in a PSA is to compute the covariance matrix for all of the data, which in our case turns out to be

(13.46)

**4. Compute Unit Eigenvector and Eigenvalues of C** Easy to do with NumPy:

(13.47)

(13.48)

where we have ordered the eigenvalues and eigenvectors so that the largest eigen-value is first. As we shall see, the eigenvector corresponding to this largest eigen-value is in fact the principal component in the data, typically with ∼ 80% of the power in it.

In Figure 13.13, we show the normalized data and the two unit eigenvectors of the covariance matrix (scaled to fill frame). Note that the _e_ 1 eigenvector looks very much like a straight-line best fit to the data. This is the major trend in the data. The other eigenvector _e_ 2 is clearly orthogonal to _e_ 1 and contain much less of the signal strength than _e_ 1 It is in the direction of the variation from the straight line fit, and cleaerly contains less information about the data. These are the essential ideas behind PSA.

**5. Expressing Data in Terms of Principal Components** Now that we have the principal components, we need to do something with them, namely, express the data in terms of them. Clearly, one choice is to ignore the eigenvectors corresponding to the smaller eigenvalues (only one in our simple example). This is useful for focusing attention on the key elements in the data, as well as for compressing the data. What one usually does now is to form a _feature vector F_ made up of the eigenvectors that we wish to keep, for example,

(13.49)

(13.50)

where _F_ 1 keeps just one principal component, while _F_ 2 keeps two. The matrix gets its name because by deciding which eigenvectors we wish to keep, we are deciding which features of the data we wish to display.

Next, we form the transpose of feature matrix composed of the eigenvectors we wish to keep, and of the adjusted data matrix:

(13.51)

(13.52)

To express the data in terms of the principal components, we multiply the transposed feature matrix by the transposed adjusted data matrix:

(13.53)

(13.54)

(13.55)

(13.56)

In Table 13.1, we place the transform data elements back into standard form, along side the original data. On the right of Figure 13.13 we show the normalized data plotted using the eigenvectors _e_ 1 and _e_ 2 as basis. This plot shows just where each datum point sits relative to the trend in the data. If we use only the principal component, we would have all of the data on a straight line (we leave that as an exercise). Of course, our data are so simple that this example does not show the power of the technique. But if we have millions of data, it would most valuable to be able to categorize them in terms of a few components.

### 13.7.2 PCA Exercises

  1. Use just the principal eigenvector to perform the PCA analysis just completed with two eigenvectors.
  2. Store data from ten cycles of the chaotic pendulum studied in Chapter 15, but do not include transients. Perform a PCA of these data and plot the results using principal component axes.

1) We discuss wave packets further in Section 13.3. 2) Wavelets have proven to be a highly effective approach to data compression, with the Joint Photographic Experts Group (JPEG) 2000 standard being based on wavelets. 3) Note that some references scale down with increasing _j_ , in contrast to our scaling up.

# 14
Nonlinear Population Dynamics

* * *

Nonlinear dynamics is one of the success stories of computational physics. It has been explored by scientists, and engineers with computers as an essential tool, often then followed by mathematicians (Motter and Campbell, 2013). The computations have led to the discovery of new phenomena such as solitons, chaos, and fractals, as you will discover on your own. In addition, because biological systems often have complex interactions and may not be in thermodynamic equilibrium states, models of them are often nonlinear, with properties similar to those of other complex systems. In this chapter, we look at discrete models of population dynamics that are simple yet produce surprising complex behavior. In the next chapter, we explore chaos for a continuous system.

* * *

## 14.1 Bug Population Dynamics

**Problem** Populations of insects and patterns of weather do not appear to follow any simple laws.1) At times, the populations patterns appear stable, at other times they vary periodically, and at other times they appear chaotic, with no discernable regularity, only to settle down to something simple again. Your _problem_ is to deduce if a simple law can produce such complicated behaviors.

## 14.2 The Logistic Map (Model)

Imagine a bunch of insects reproducing generation after generation. We start with _N_ 0 bugs, then in the next generation we have to live with _N_ 1 of them, and after _i_ generations there are _N i_ bugs to bug us. We want to define a model of how _N n_ varies with the discrete generation number _n_. Clearly, if the rates of breeding and dying are the same, then a stable population should occur. Yet bugs cannot live on love alone, they must also eat, and bugs not being farmers must compete for the available food supply. This tends to restrict their number to lie below some maximum population. We want to build these observations into our model.

For guidance, we look to the radioactive decay simulation in Chapter 4, where the discrete decay law, Δ _N_ /Δ _t_ = − _λ_ N, led to exponential-like decay. Likewise, if we reverse the sign of _λ_ , we should get exponential-like _growth,_ which is a good place to start our modelling. We assume that the bug-breeding rate is proportional to the number of bugs:

(14.1)

Because we know the empirical fact that exponential growth usually tapers off, we extend the model by incorporating the observation that for a given environment there must be maximum population _N_ ∗, which is called the _carrying capacity_. Consequently, we modify the exponential growth model (14.1) by modifiying the growth rate so that it decreases as the population _N i_ approaches _N_ ∗:

(14.2)

(14.3)

We expect that when _N i_ is small compared to _N_ ∗, the population will grow exponentially. We also expect that as _N i_ approaches _N_ ∗, the growth rate will decrease, eventually becoming negative if _N i_ exceeds the carrying capacity _N_ ∗.

Equation 14.3 is a form of the _logistic map_. It is usually written as a relation between the number of bugs in future and present generations:

(14.4)

(14.5)

This relation looks simpler when expressed in terms of dimensionless variables:

(14.6)

(14.7)

(14.8)

Here _μ_ is a dimensionless growth parameter and _x i_ is a dimensionless population variable. Observe from (14.7) that the _growth rate μ_ = 1 when the breeding rate _λ_ ' equals 0, and is otherwise expected to be larger than 1. If the number of bugs born per generation _λ_ 'Δ _t_ is large, then   that is, _x i_ is essentially the fraction of the maximum population _N_ ∗. Consequently, realistic _x_ values generally lie in the range   corresponding to no bugs, and _x_ = 1 corresponding to the maximum population.

The map (14.6) is seen to be the sum of a linear and a quadratic dependence on _x i_ In general, we employ the term "map" to refer to a function _f_ ( _x_ ) that converts one number in a sequence to the next

(14.9)

For the logistic map,   with the quadratic dependence on _x_ making this a nonlinear map, and the dependence on only the one variable _x_ making it a _one-dimensional_ map.

Just by looking at (14.6) there is no way of knowing how good a model this will be. Being as simple as it is, the model cannot be expected to be a complete description of the population dynamics of bugs. However, if it exhibits some features similar to those found in nature, then it may well form the foundation for a more complete description.

## 14.3 Properties of Nonlinear Maps (Theory and Exercise)

Rather than going through some fancy mathematical analysis to learn about the properties of the logistic map (Rasband, 1990), we suggest that you explore it yourself on a computer or a calculator by generating and plotting sequences of values. You should get results similar to those shown in Figures 14.1 and 14.2.

**Stable Populations** We want to see if the model can produce a stable population, that is, one that remains the same from generation to generation.

  1. Calculate and plot _x_ _i_ +1 as a function of the generation number _i_.
  2. The initial population _x_ 0 is called the _seed_ , and we suggest _x_ 0 = 0.75 as a good starting value (the dynamical effects are not sensitive to the seed).

**Figure 14.1** The insect population _x n_ vs. the generation number _n_ for the two growth rates: (a) _μ_ = 2.8, a single attractor; (b) _μ_ = 3.3, a double attractor.

**Figure 14.2** The insect population _x n_ vs. the generation number _n_ for two different growth rates: (a) _μ_ = 3.5, a quadruple attractor; (b) _μ_ = 3.8, a chaotic regime.

  3. Your results should be highly sensitive to the value for the growth rate _μ_. Too large a value may lead to instabilities, while too small a value may lead to extinction. To make sure that the model is behaving reasonably, try some cases for which we can be fairly sure of what the results should be. By trying some negative and zero values for _μ_ (for example, -1, −0.75, −0.5, −0.25, 0) we should obtain decaying populations.
  4. Now that you have some confidence in the model, see if you can increase the population to some stable values. With the same initial population as before, try _μ_ = 0, 0.5, 1.0, 1.5, 2. Make plots of _x i_ vs. _i_ for each of these cases.
  5. Take note of the _transient_ behavior in these plots that occur for early generations before a steady state or more regular behavior sets in. In general, these are not the long-term dynamical behaviors of interest.
  6. For a fixed value of _μ_ , try different values for the seed population _x_ 0. Verify that differing values of _x_ 0 do affect the transients, but _not_ the values of the stable populations.

You should have found that with positive growth rates _μ_ , this model yields stable populations, with the bugs approaching the maximum population more rapidly as _μ_ gets larger. This is a good validation of the model. Some typical behaviors are shown in Figures 14.1 and 14.2. In Figure 14.1a, we see equilibration into a single population; in Figure 14.1b, we see oscillation between two population levels; in Figure 14.2a we see oscillation among four levels; and in Figure 14.2b we see a chaotic system.

### 14.3.1 Fixed Points

An important property of the map (14.6) is the possibility of the sequence _x i_ reaching a _fixed_ point _x_ ∗, that is, a value of the population at which the system remains. At a _one-cycle_ fixed point, there is no change in the population from generation _i_ to generation _i_ \+ 1, that is,

(14.10)

Substituting the logistic map (14.6) into this equation produces an algebraic equation that we can solve

(14.11)

(14.12)

The nonzero fixed point _x_ ∗ = ( _μ_ − 1)/ _μ_ corresponds to a stable population with a balance between birth and death that is reached regardless of the initial population (Figure 14.1a). In contrast, the _x_ ∗ = 0 point is unstable and the population remains static only as long as no bugs exist; if even a few bugs are introduced, exponential growth occurs. Further analysis (Section 14.8) tells us that the stability of a population is determined by the magnitude of the derivative of the mapping function _f_ ( _x i_) at the fixed point (Rasband, 1990):

(14.13)

For the one cycle of the logistic map (14.6), the derivative is

(14.14)

### 14.3.2 Period Doubling, Attractors

Equation 14.14 tells us that while the equation for fixed points (14.12) may be satisfied for all values of _μ_ , the populations will not be stable if _μ_ > 3. In this latter case, the system's long-term population _bifurcates_ into two populations, a so called _two-cycle_. The effect is known as _period doubling_ and is evident in Figure 14.1b. Because the system now acts as if it were attracted to two populations, these populations are called _attractors_ or _cycle points_. We can easily predict the _x_ values for these two-cycle attractors by demanding that generation _i_ \+ 2 have the same population as generation _i_ :

(14.15)

(14.16)

We see that as long as _μ_ > 3, the square root produces a real number and thus that physical solutions exist (complex or negative _x_ ∗ values are nonphysical). We leave it to your computer explorations to discover how the system continues to double periods as _μ_ is further increased. In all cases, the pattern repeats: one populations bifurcates into two.

## 14.4 Mapping Implementation

It is now time to carry out a more careful investigation of the logistic map along the original path followed by (Feigenbaum, 1979):

  1. Confirm Feigenbaum's observations of the different patterns shown in Figures 14.1 and 14.2 that occur for _μ_ = (0.4, 2.4, 3.2, 3.6, 3.8304) and seed _x_ 0 = 0.75.
  2. Identify the following in your graphs:
    1. a) _Transients:_ Irregular behaviors before reaching a steady state that differ for different seeds.
    2. b) _Asymptotes:_ In some cases, the steady state is reached after only 20 generations, while for larger _μ_ values, hundreds of generations may be needed. These steady-state populations are independent of the seed.
    3. c) _Extinction:_ If the growth rate is too low, _μ_ ≤ 1, the population dies off.
    4. d) _Stable states:_ The stable single-population states attained for _μ_ < 3 should agree with the prediction (14.12).
    5. e) _Multiple cycles:_ Examine the map orbits for a growth parameter _μ_ increasing continuously through 3. Observe how the system continues to double periods as _μ_ increases. To illustrate, in Figure 14.2a with _μ_ = 3.5, we notice a steady state in which the population alternates among four attractors (a _four-cycle_ ).
    6. f) _Intermittency:_ Observe simulations for 3.8264 < _μ_ < 3.8304. Here the system appears stable for a finite number of generations and then jumps all around, only to become stable again.
    7. g) _Chaos: We define chaos as the deterministic behavior of a system displaying no discernible regularity_. This may seem contradictory; if a system is deterministic, it must have step-to-step correlations (which, when added up, mean long-range correlations); but if it is chaotic, the complexity of the behavior may hide the simplicity within. In an operational sense, a chaotic system is one with an extremely high sensitivity to parameters or initial conditions. This sensitivity to even minuscule changes is so high that it is impossible to predict the long-range behavior unless the parameters are known to infinite precision (a physical impossibility).
  3. The system's behavior in the chaotic region is critically dependent on the exact values of _μ_ and _x_ 0. Systems starting out with nearly identical values for _μ_ and _x_ 0 may end up with quite different behaviors. In some cases, the complicated behaviors of nonlinear systems will be _chaotic,_ but this is not the same as being random.2)
    1. a) Compare the long-term behaviors of starting with the two essentially identical seeds _x_ 0 = 0.75 and _x_ ′0 = 0.75(1 + _є_ ), where _є_   2 × 10−14.
    2. b) Repeat the simulation with _x_ 0 = 0.75 and two essentially identical survival parameters, _μ_ = 4.0 and _μ_ ′ = 4.0(1 − _є_ ), where _є_   2 × 10−14. Both simulations should start off the same but eventually diverge.

## 14.5 Bifurcation Diagram (Assessment)

Watching the population change with generation number gives a good idea of the basic dynamics, at least until it gets too complicated to discern patterns. In particular, as the number of bifurcations keeps increasing and the system becomes chaotic, it maybe hard to see a simple underlying structure within the complicated behavior. One way to visualize what is going on is to concentrate on the attractors, that is, those populations that appear to attract the solutions and to which the solutions continuously return (long-term iterates). A plot of these attractors of the logistic map as a function of the growth parameter _μ_ is an elegant way to summarize the results of extensive computer simulations.

A _bifurcation diagram_ for the logistic map is shown in Figure 14.3, with one for a Gaussian map given in Figure 14.4. To generate such a diagram you proceed through all values of _μ_ in steps. For each value of _μ_ , you let the system go through hundreds of iterations to establish that the transients have died out, and then write the values ( _μ, x ∗_) to a file for hundreds of iterations after that. If the system falls into an _n_ -cycle for this _μ_ value, then there should predominantly be _n_ different values written to the file. Next, the value of the initial populations _x_ 0 is changed slightly, and the entire procedure is repeated to ensure that no fixed points are missed. When finished, your program will have stepped through all the values of _μ_ and _x_ 0. Our sample program `Bugs.py` is shown in Listing 14.1.

**Figure 14.3** Bifurcation plot of attractor population _x_ ∗ _vs_. growth rate _μ_ for the logistic map. The inset shows some details of a three-cycle window. (The colors or gray scales indicate the regimes over which we distributed the work on different CPUs when run in parallel.)

**Figure 14.4** Bifurcation plot for the Gaussian map _f_ ( _x_ ) = exp(− _bx_ 2) + _μ_ (courtesy of W. Hager).

**Listing 14.1** **Bugs.py** produces the bifurcation diagram of the logistic map. A proper program requires finer grids, a scan over initial values, and removal of duplicates.

### 14.5.1 Bifurcation Diagram Implementation

The last part of this problem asks you to reproduce Figure 14.3 at various levels of detail. While the best way to make a visualization of this sort would be with visualization software that permits you to vary the intensity of each individual point on the screen, we simply plot individual points and have the density in each region determined by the number of points plotted there. When thinking about plotting many individual points to draw a figure, it is important to keep in mind that your monitor may have approximately 100 pixels per inch and your laser printer 300 dots per inch. This means that if you plot a point at each pixel, you will be plotting ∼ 3000 × 3000   10 million elements. _Beware:_ This can require some time and may choke a printer. In any case, printing at a finer resolution is a waste of time.

### 14.5.2 Visualization Algorithm: Binning

  1. Break up the range 1 ≤ _μ_ ≤ 4 into 1000 steps. These are the "bins" into which we will place the _x_ ∗ values.
  2. In order not to miss any structures in your bifurcation diagram, loop through a range of initial _x_ 0 values as well.
  3. Wait at least 200 generations for the transients to die out, and then print the next several hundred values of ( _μ, x_ ∗) to a file.
  4. Print out your _x_ ∗ values to no more than three or four decimal places. You will not be able to resolve more places than this on your plot, and this restriction will keep your output files smaller, as will removing duplicate entries. You can use formatted output to control the number of decimal places, or you can do via a simple conversion: multiply the _x i_ values by 1000 and then throw away the part to the right of the decimal point:
`Ix[i]= int(1000*x[i])`.
You may then divide by 1000 if you want floating-point numbers.
  5. Plot your file of _x_ ∗ vs. _μ_. Use small symbols for the points and do not connect them.
  6. Enlarge (zoom in on) sections of your plot and notice that a similar bifurcation diagram tends to be contained within each magnified portion (this is called _self-similarity_ ).
  7. Look over the series of bifurcations occurring at

(14.17)

The end of this series is a region of chaotic behavior.
  8. Inspect the way this and other sequences begin and then end in chaos. The changes sometimes occur quickly, and so you may have to make plots over a very small range of _μ_ values to see the structures.
  9. A close examination of Figure 14.3 shows regions where, for a slight increase in _μ_ , a very large number of populations suddenly change to very few populations. Whereas these may appear to be artifacts of the video display, this is a real effect and these regions are called _windows_. Check that at _μ_ = 3.828 427 chaos moves into a three-cycle population.

### 14.5.3 Feigenbaum Constants (Exploration)

Feigenbaum discovered that the sequence of _μ k_ values (14.17) at which bifurcations occur follows a regular pattern (Feigenbaum, 1979). Specifically, the _μ_ values converge geometrically when expressed in terms of the distance between bifurcations _δ_ :

(14.18)

(14.19)

Use your sequence of _μ k_ values to determine the constants in (14.18) and compare them to those found by Feigenbaum:

(14.20)

Amazingly, the value of the _Feigenbaum constant δ_ is universal for all second-order maps.

## 14.6 Logistic Map Random Numbers (Exploration)

Although we have emphasized that chaos, with its short-term predictability, is different from random, with has no predictability, it is nevertheless possible for the logistic map in the chaotic region ( _μ_ ≥ 4) to be used to generate pseudo random numbers (Phatak and Rao, 1995). This is carried out in two steps:

(14.21)

(14.22)

Although successive _x i_'s are correlated, if the population for every sixth generation or so is examined, the correlation is effectively gone and pseudorandom numbers result. To make the sequence more uniform, the trigonometric transformation is then used.

**Exercise** Use the random-number tests discussed in Chapter 4, or an actual Monte Carlo simulation, to test this claim.

## 14.7 Other Maps (Exploration)

Bifurcations and chaos are characteristic properties of nonlinear systems. Yet systems can be nonlinear in a number of ways. The table below lists four maps that generate _x i_ sequences containing bifurcations.

**Name** | **_f(x)_** | **Name** | **_f(x)_**
---|---|---|---
Logistic | _μx_ (1 − _x_ ) | Tent | _μ_ (1 − 2| _x_ − 1/2|)
Ecology | _x_ e _μ_ (1 – _x_ ) | Quartic | _μ_ [1 − (2 _x_ − 1)4]
Gaussian | e− _bx_ 2 \+ _μ_ |  |

The tent map derives its nonlinear dependence from the absolute value operator, while the logistic map is a subclass of the ecology map. Explore the properties of these other maps and note the similarities and differences.

## 14.8 Signals of Chaos: Lyapunov Coefficient and Shannon Entropy

The Lyapunov coefficient or exponent _λ i_ provides an analytic measure of whether a system is chaotic (Wolf _et al.,_ 1985; Ramasubramanian and Sriram, 2000; Williams, 1997). Essentially, the coefficient is a measure of the rate of exponential growth of the solution near an attractor. If the coefficient is positive then the solution moves away from the attractor, which is an indication of chaos, while if the coefficient is negative then the solution moves back toward the attractor, which is an indication of stability. For 1D problems there is only one such coefficient, whereas in general there is a coefficient for each degree of freedom. The essential assumption is that the distance _L_ between neighboring paths _x n_ near an attractor have an _n_ (generation number or time) dependence _L_ ∝ exp( _λt_ ). Consequently, orbits that have _λ_ > 0 diverge and are chaotic; orbits that have _λ_ = 0 remain marginally stable, while orbits with _λ_ < 0 are periodic and stable. Mathematically, the Lyapunov coefficient is defined as

(14.23)

As an example, we calculate the Lyapunov exponent for a general 1D map,

(14.24)

and then apply the result to the logistic map. To determine stability, we examine perturbations about a reference trajectory _x_ 0 by adding a small perturbation and iterating once (Manneville, 1990; Ramasubramanian and Sriram, 2000):

(14.25)

We substitute this into (14.24) and expand _f_ in a Taylor series around _x_ 0:

(14.26)

This is the proof of our earlier statement that a negative derivative indicates stability. To deduce the general result we examine one iteration:

(14.27)

(14.28)

This last relation tells us how trajectories differ on the average after _n_ steps:

(14.29)

We now solve for the Lyapunov number _L_ and take its logarithm to obtain the Lyapunov coefficient:

(14.30)

For the logistic map, we obtain

(14.31)

where the sum is over iterations.

The code `LyapLog.py` in Listing 14.2 computes the Lyapunov exponents for the bifurcation plot of the logistic map. In Figures 14.5 and 14.6, we show its output, and note the sign changes in _λ_ where the system becomes chaotic, and the abrupt changes in slope at bifurcations. (A similar curve is obtained for the fractal dimension of the logistic map as, indeed, the two are proportional.)

**Figure 14.5** Lyapunov coefficient (bottom) and bifurcation values (top) for the logistic map as functions of the growth rate _μ_. Note how the Lyapunov coefficient, whose value is a measure of chaos, changes abruptly at the bifurcations.

**Figure 14.6** Shannon entropy (top) and Lyapunov coefficient (bottom) for the logistic map. Note the close relation between the thermodynamic measure of disorder (entropy) and the nonlinear dynamics measure of chaos (Lyapunov).

_Shannon entropy,_ like the Lyapunov exponent, is used an indicator of chaos. Entropy is a measure of uncertainty (garbled signal) that has proven useful in communication theory (Shannon, 1948; Ott, 2002; Gould _et al.,_ 2006). Imagine that an experiment has _N_ possible outcomes. If the probability of each is _p_ 1, _p_ 2,..., _p N,_ with normalization such that   then the Shannon entropy is defined as

(14.32)

**Listing 14.2** **LyapLog.py** computes Lyapunov coefficient for the bifurcation plot of the logistic map as a function of growth rate. Note the fineness of the _μ_ grid.

If _p i_ ≡ 0, there is no uncertainty and _S_ shannon = 0, as you might expect. If all _N_ outcomes have equal probability, _p i_ ≡ 1/ _N_ , we obtain the expression familiar from statistical mechanics, _S_ Shannon = ln _N_.

The code `Entropy.py` in Listing 14.3 computes the Shannon entropy for the logistic map as a function of the growth parameter _μ_. The results (Figure 14.6, top) are seen to be quite similar to the Lyapunov exponent, again with discontinuities occurring at the bifurcations.

**Listing 14.3** **Entrophy.py** computes the Shannon entropy for the logistic map as a function of growth parameter _μ_.

## 14.9 Coupled Predator–Prey Models

At the beginning of this chapter, we saw complicated behavior arising from a model of bug population dynamics in which we imposed a maximum population. We described that system with a discrete logistic map. Now we study models describing predator–prey population dynamics proposed by the American physical chemist Lotka (Lotka, 1925) and the Italian mathematician Volterra (Volterra, 1926; Gurney and Nisbet, 1998). Though simple, versions of these equations are used to model biological systems and neural networks.

**Problem** Is it possible to use a small number of predators to control a population of pests so that the number of pests remains essentially constant? Include in your considerations the interaction between the populations as well as the competition for food and predation time.

## 14.10 Lotka–Volterra Model

We extend the logistic map to the Lotka–Volterra Model (LVM) which describes two populations coexisting in the same geographical region. Let

(14.33)

In the absence of interactions between the species, we assume that the prey population _p_ breeds at a per-capita rate of _a_. This would lead to exponential growth:

(14.34)

(14.35)

(14.36)

Here we give both the discrete and continuous versions of the model, with the exponential the solution of the continuous version. Yet exponential growth does not occur because the predators _P_ eat more prey if the number of prey increases. The interaction rate between predator and prey requires both to be present, with the simplest assumption being that it is proportional to their joint probability:

(14.37)

where _b_ is a constant. This leads to a prey growth rate including both predation and breeding:

(14.38)

(14.39)

If left to themselves, predators _P_ will also breed and increase their population. Yet predators need animals to eat, and if there are no other populations to prey upon, we assume that they will eat each other (or their young) at a per-capita mortality rate _m_ :

(14.40)

However, if there are prey also to interact with (read "eat") at the rate _bpP,_ the predator population will grow at the rate

(14.41)

where _ _is a constant that measures the efficiency with which predators convert prey interactions into food.

Equations 14.39 and 14.41 are two simultaneous ODEs and are our first model. After placing them in the standard dynamic form, we solve them with the `rk4` algorithm:

(14.42)

A sample code to solve these equations is `PredatorPrey.py` in Listing 14.4.

**Listing 14.4** **PredatorPrey.py** computes population dynamics for a group of interacting predators and prey.

### 14.10.1 Lotka–Volterra Assessment

Results from the code `PredatorPrey.py` are shown in Figure 14.7. In Figure 14.7a, we see that the two populations oscillate out of phase with each other in time; when there are many prey, the predator population eats them and grows; yet then the predators face a decreased food supply and so their population decreases; that in turn permits the prey population to grow, and so forth. In Figure 14.7b, we plot what is called a "phase–space" plot of _P_ ( _t_ ) vs. _p_ ( _t_ ). A closed orbit here indicates a limit cycle that repeats indefinitely. Although increasing the initial number of predators does decrease the maximum number of pests, it is not a satisfactory solution to our _problem,_ as the large variation in the number of pests cannot be called "control."

  1. Explain in words why their is a correlation between extremums in prey and predator populations?
  2. Because predators eat prey, one might expect the existence of a large number of predators to lead to the eating of a large number of prey. Explain why the maxima in predator population and the minima in prey population do not occur at the same time.
  3. Why do the extreme values of the population just repeat with no change in their values?
  4. Explain the meaning of the spirals in the predator–prey phase space diagram.
  5. Explain why the phase–space orbits closed?
  6. What different initial conditions would lead to different phase–space orbits?
  7. Discuss the symmetry and lack of symmetry in the phase–space orbits.

## 14.11 Predator–Prey Chaos

Mathematical analysis tells us that, in addition to nonlinearity, a system must contain a minimum number of degrees of freedom before chaos will occur. For example, for chaos to occur in a predator–prey model, there must be three or more  species present. And so to produce some real chaos, we must extend our previous treatment. Accordingly, we now extend the predator–prey model to include four species, each with population _p i_ competing for the same finite set of resources. The differential equation form of the model (Vano _et al.,_ 2006; Cencini _et al.,_ 2010) extends the Lotka–Volterra model (14.41) to:

**Figure 14.7** (a) The time dependencies of the populations of prey _p_ ( _t_ ) (solid curve) and of predator _P_ ( _t_ ) (dashed curve) fromthe Lotka–Volterra model. (b) A phase–space plot of Prey population _p_ as a function of predator population _P_. The different orbits correspond to different initial conditions.

(14.43)

Here _a i_ is a measure of the growth rate of species _i,_ and _b ij_ is a measure of the extent to which species _j_ consumes the resources otherwise used by species _i._ If we require both _a i_ ≥0 and _b ij_ ≥ 0, then all populations should remain in the range 0 ≤ _p i_ ≤ 1.

Because four species covers a very large parameter space, we suggest that you start your exploration using the same parameters that (Vano _et al.,_ 2006) found produces chaos:

(14.44)

But because chaotic systems are hyper sensitive to the exact parameter values, you may need to modify these slightly.

Note that the self-interactions terms _a ii_ = 1, which is a consequence of measuring the population of each species in units of its individual carrying capacity. We solve (14.43) with initial conditions corresponding to an equilibrium point at

**Figure 14.8** A chaotic attractor for the 4D Lotka–Volterra model projected onto three axes.

which all species coexist:

(14.45)

One way to visualize the solution to (14.43) is to make four plots of _p_ 1( _t_ ), _p_ 2( _t_ ), _p_ 3( _t_ ), and _p_ 4( _t_ ) vs. time. The results are quite interesting, and we leave that as an exercise for you. A more illuminating visualization would be to create a 4D phase–space plot of [ _p_ 1( _t i_), _p_ 2( _t i_), _p_ 3( _t i_), _p_ 4( _t i_)] for _i_ =1, _N,_ where _N_ is the number of time steps used in your numerical solution. Even in the presence of chaos, the geometric structure so created may have a smooth and well-defined shape. Unfortunately, we have no way to show such a plot, and so in its stead we must project the 4D structure onto 2D and 3D axes. We show one such 3D plot in Figure 14.8. This is a classic type of chaotic attractor, with the 3D structure folded over into a nearly 2D structure (the Rossler equation is famous for producing such a structure).

### 14.11.1 Exercises

  1. Solve (14.43) for the suggested parameters and initial conditions, and make plots of _p_ 1–4( _t_ ) as functions of time. Comment on the type of behavior that these plots exhibit.
  2. Construct the 4D phase chaotic attractor formed by the solutions of (14.43) by writing to a file the values [ _p_ 1( _t i_), _p_ 2( _t i_), _p_ 3( _t i_), _p_ 4( _t i_)] where _i_ labels the time step. In order to avoid needlessly long files, you may want to skip a number of time steps before each file output.
    1. a) Plot all possible 2D phase–space plots, that is, plots of _p i vs pj, i_ ≠ _j_ = 1 − 3.
    2. b) Plot all possible 3D phase–space plots, that is, plots of _p i vs pj vs pk_.
_Note:_ you have to adjust the parameters or initial conditions slightly to obtain truly chaotic behavior.

### 14.11.2 LVM with Prey Limit

The initial assumption in the LVM that prey grow without limit in the absence of predators is clearly unrealistic. As with the logistic map, we include a limit on prey numbers that accounts for depletion of the food supply as the prey population grows. Accordingly, we modify the constant growth rate from _a_ to _a_ (1 − _p/K_ ) so that growth vanishes when the population reaches a limit _K_ , the _carrying capacity:_

(14.46)

(14.47)

The behavior of this model with prey limitations is shown in Figure 14.9. We see that both populations exhibit damped oscillations as they approach their equilibrium values, and that as hoped for, the equilibrium populations are independent of the initial conditions. Note how the phase–space plot spirals inward to a single close limit cycle, on which it remains, with little variation in prey number. This is "control," and we may use it to start a chemical-free pest control business.

**Figure 14.9** (a) The Lotka–Volterra model of prey population _p_ ( _t_ ) (solid curve) and predator population _P_ ( _t_ ) (dashed curve) vs. time when a prey population limit is included, (b) Prey population _p_ as a function of predator population _P_.

### 14.11.3 LVM with Perdation Efficiency

An additional unrealistic assumption in the original LVM is that the predators immediately eat all the prey with which they interact. As anyone who has watched a cat hunt a mouse knows, predators spend some time finding prey and also chasing, killing, eating, and digesting it (all together called _handling)_. This extra time decreases the rate _bpP_ at which prey are eliminated. We define the _functional response p a_ as the probability of one predator finding one prey. If a single predator spends time _t_ search searching for prey, then

(14.48)

If we call _t_ h the time a predator spends handling a single prey, then the effective time a predator spends handling a prey is _p a th_. Such being the case, the total time _T_ that a predator spends finding and handling a single prey is

(14.49)

(14.50)

where _p a/T_ is the effective _rate_ of eating prey. We see that as the number of prey _p_ → ∞, the efficiency in eating them → 1. We include this efficiency in (14.46) by modifying the rate _b_ at which a predator eliminates prey to _b_ /(1 + _bpt_ h):

(14.51)

To be more realistic about the predator growth, we also place a limit on the predator carrying capacity but make it proportional to the number of prey:

(14.52)

Solutions for the extended model (14.51) and (14.52) are shown in Figure 14.10. Observe the existence of three dynamic regimes as a function of _b:_

  * small _b_ : no oscillations, no overdamping,
  * medium _b_ : damped oscillations that converge to a stable equilibrium,
  * large _b_ : limit cycle.

The transition from equilibrium to a limit cycle is called a _phase transition._

**Figure 14.10** Lotka–Volterra model with predation efficiency and prey limitations. From left to right: overdamping, _b_ = 0.01; damped oscillations, _b_ = 0.1, and limit cycle, _b_ = 0.3.

We finally have a satisfactory solution to our _problem_. Although the prey population is not eliminated, it can be kept from getting too large and from fluctuating widely. Nonetheless, changes in the parameters can lead to large fluctuations or to nearly vanishing predators.

### 14.11.4 LVM Implementation and Assessment

  1. Write a program to solve (14.51) and (14.52) using the `rk4` algorithm and the following parameter values.  **Model** | **_a_** | **_b_** | **_ _**| **_m_** | **_K_** | **_k_**
---|---|---|---|---|---|---
LVM-I | 0.2 | 0.1 | 1 | 0.1 | 0 | —
LVM-II | 0.2 | 0.1 | 1 | 0.1 | 20 | —
LVM-III | 0.2 | 0.1 | — | 0.1 | 500 | 0.2
  2. For each of the three models, construct
    1. a) a time series for prey and predator populations,
    2. b) phase–space plots of predator vs. prey populations.
  3. _LVM-I_ : Compute the equilibrium values for the prey and predator populations. Do you think that a model in which the cycle amplitude depends on the initial conditions can be realistic? Explain.
  4. _LVM-II_ : Calculate numerical values for the equilibrium values of the prey and predator populations. Make a series of runs for different values of prey carrying capacity _K._ Can you deduce how the equilibrium populations vary with prey carrying capacity?
  5. Make a series of runs for different initial conditions for predator and prey populations. Do the cycle amplitudes depend on the initial conditions?
  6. _LVM-III_ : Make a series of runs for different values of _b_ and reproduce the three regimes present in Figure 14.10.
  7. Calculate the critical value for _b_ corresponding to a phase transition between the stable equilibrium and the limit cycle.

### 14.11.5 Two Predators, One Prey (Exploration)

  1. Another version of the LVM includes the possibility that two populations of predators _P_ 1 and _P_ 2 may "share" the same prey population _p._ Investigate the behavior of a system in which the prey population grows logistically in the absence of predators:

(14.53)

(14.54)

    1. a) Use the following values for the model parameters and initial conditions: _a_ = 0.2, _K_ = 1.7, _b_ 1 = 0.1, _b_ 2 = 0.2, _m_ 1 = _m_ 2 = 0.1, _ _1 = 1.0, _ _2 = 2.0, _p_ (0) = _P_ 2(0) = 1.7, and _P_ 1(0) = 1.0.
    2. b) Determine the time dependence for each population.
    3. c) Vary the characteristics of the second predator and calculate the equilibrium population for the three components.
    4. d) What is your answer to the question, "Can two predators that share the same prey coexist?"
  2. The nonlinear nature of the Lotka–Volterra model can lead to chaos and fractal behavior. Search for complex behaviors by varying the growth rates.

1) Except maybe in Oregon, where storm clouds come to spend their weekends. 2) You may recall from Chapter 4 that a truly random sequence of events does not even have next step predictability, while these chaotic systems do.

# 15
Continuous Nonlinear Dynamics

* * *

In Chapter 14, we developed the logistic map and predator–prey models as a means to understand how biological populations achieve dynamic equilibrium. In this chapter, we explore the driven realistic pendulum, a continuous system chaos that can support chaotic behavior. In Chapter 8, we have already seen how useful our tools are at solving nonlinear equations. Our emphasis now is on exploring chaos and on the usefulness of phase space in displaying the simplicity underlying complex behavior.

## 15.1 Chaotic Pendulum

**Problem** The plane pendulum is a classic subject for physics. However, it has most often been studied assuming small angle displacements, which is a good approximation only for very large grandfather clocks. However, when the pendulum is driven and the displacements get large, the motion becomes too complicated for analytic solution. Your _problem_ is to compute the motion of a driven pendulum with large displacements, to ensure that your calculation is reliable and sensitive, and then to search for the simplicity that may underly complexity.

What we call a _chaotic pendulum_ is just a pendulum with friction and a driving torque (Figure 15.1a), but without the assumption of small deflection angle. Newton's laws of rotational motion tell us that the sum of the gravitational torque − _mgl_ sin _θ_ the frictional torque  , and the external torque _τ_ 0 cos _ωt_ equals the moment of inertia of the pendulum times its angular acceleration (Rasband, 1990):

(15.1)

(15.2)

(15.3)

**Figure 15.1** (a) A pendulum of length / driven through resistive air (dotted arcs) by an external sinusoidal torque (semicircle). The strength of the external torque is given by _f_ and that of air resistance by _a_. (b) A double pendulum with neither air resistance nor a driving force. In both cases there is a gravitational torque.

Equation 15.2 is a second-order time-dependent nonlinear differential equation. Its nonlinearity arises from the sin _θ_ , as opposed to the _θ_ , dependence of the gravitational torque. The parameter _ω_ 0 is the natural frequency of the system arising from the restoring torque, _α_ is a measure of the strength of friction, and _f_ is a measure of the strength of the driving torque. In our standard ODE form, d _y_ / d _t_ = _f_ (Chapter 8), we have two simultaneous first-order equations:

(15.4)

(15.5)

### 15.1.1 Free Pendulum Oscillations

If we ignore friction and external torques, Newton's law (15.2) takes the simple, yet still nonlinear, form

(15.6)

If the displacements are small, we can approximate sin _θ_ by _θ_ and obtain the linear equation of simple harmonic motion with frequency _ω_ 0:

(15.7)

In Chapter 8, we studied how nonlinearities produce anharmonic oscillations, and indeed (15.6) is another good candidate for such studies. As before, we expect solutions of (15.6) for the free realistic pendulum to be periodic, but with a frequency _ω_   _ω_ 0 only for small oscillations. Furthermore, because the restoring torque, _mgl_ sin _θ_   _mgl_ ( _θ_ − _θ_ 3 /3), is less than the _mglθ_ assumed in a harmonic oscillator, realistic pendulums swing slower (have longer periods) as their angular displacements are made larger.

### 15.1.2 Solution as Elliptic Integrals

The analytic solution to the realistic pendulum is a textbook problem (Landau and Lifshitz, 1976; Marion and Thornton, 2003; Scheck, 1994), except that it is hardly a solution and hardly analytic. The "solution" is based on energy being a constant (integral) of the motion. For simplicity, we start the pendulum off at rest at its maximum displacement _θ_ m. Because the initial energy is all potential, we know that the total energy of the system equals its initial potential energy (Figure 15.1),

(15.8)

Yet because _E_ = KE + PE is a constant, we can write for any value of _θ_

(15.9)

(15.10)

Because the motion is still periodic, we have assumed that it takes _T_ /4 for the pendulum to travel from _θ_ = 0 to _θ_ = _θ_ m. The integral in (15.9) can be expressed as an _elliptic integral of the first kind_ (studied in Section 5.16). If you think of an elliptic integral as a generalization of a trigonometric function, then this is a closed-form solution; otherwise, it is an integral needing computation. The series expansion of the period (15.10) is obtained by expanding the denominator and integrating it term by term. It tells us, for example, that an amplitude of 80° leads to a 10% slowdown of the pendulum relative to the small _θ_ result. In contrast, we will determine the period computationally without the need for any expansions.

### 15.1.3 Implementation and Test: Free Pendulum

As a preliminary to the solution of the full equation (15.2), modify your `**rk4**` program to solve (15.6) for the free oscillations of a realistic pendulum.

  1. Start your pendulum at _θ_ = 0 with  . Gradually increase   to increase the importance of nonlinear effects.
  2. Test your program for the linear case (sin _θ_ → _θ_ ) and verify that
    1. a) your solution is harmonic with frequency _ω_ 0 = 2π / _T_ 0, and
    2. b) the frequency of oscillation is independent of the amplitude.
  3. Devise an algorithm to determine the period _T_ of the oscillation by counting the time it takes for three successive passes of the amplitude through _θ_ = 0. (You need _three_ passes because a general oscillation may not be symmetric about the origin.) Test your algorithm for simple harmonic motion where you know _T_ 0.
  4. For the realistic pendulum, observe the change in period as a function of increasing initial energy or displacement. Plot your observations along with (15.10).
  5. Verify that as the initial KE approaches 2 _mgl_ , the motion remains oscillatory but not harmonic (Figure 15.4).
  6. At _E_ = 2 _mgl_ (the _separatrix_ ), the motion changes from oscillatory to rotational ("over the top" or "running"). See how close you can get your solution to the separatrix and to its infinite period.
  7.   Use the applet `**HearData**` (Figure 15.2) to convert your different oscillations to sound and hear the difference between harmonic motion (boring) and anharmonic motion containing overtones (interesting).

**Figure 15.2** The data screen (a) and the output screen (b) of the applet `**HearData**` that converts data into sounds. Columns of [ _t i_, _x_ ( _t i_)] data are pasted into the data window, processed into the graph in the output window, and then converted to sound data that are played by Java.

## 15.2 Visualization: Phase-Space Orbits

The conventional solution to an equation of motion is the position _x_ ( _t_ ) and the velocity _v_ ( _t_ ) as functions of time. Often behaviors that appear complicated as functions of time appear simpler when viewed in an abstract space called _phase space_ , where the ordinate is the velocity _v_ ( _t_ ) and the abscissa is the position _x_ ( _t_ ) (Figure 15.3). As we see from the phase-space figures, the solutions form geometric objects that are easy to recognize. (We provide two applets `**Pend1**` and `**Pend2**` to help the reader make the connections between phase-space shapes and the corresponding physical motion.)

The position and velocity of a free harmonic oscillator are given by the trigonometric functions

(15.11)

When substituted into the total energy, we obtain two important results:

(15.12)

(15.13)

The first equation, being that of an ellipse, proves that the harmonic oscillator follows closed elliptical orbits in phase space, with the size of the ellipse increasing with the system's energy. The second equation proves that the total energy is a constant of the motion. Different initial conditions having the same energy start at different places on the same ellipse and transverse the same orbits.

**Figure 15.3** Three potentials and their behaviors in phase space. The different orbits below the potentials correspond to different energies, as indicated by the limits of maximum displacements within the potentials (dashed lines). (a) A repulsive potential leads to open orbits characteristic of nonperiodic motion. The trajectories cross at the hyperbolic point in the middle, an unstable equilibrium point. (b) The harmonic oscillator leads to symmetric ellipses; the closed orbits indicate periodic behavior, and the symmetric trajectories indicate a symmetric potential. (c) A nonharmonic oscillator. Notice that the ellipse-like trajectories at the bottom are neither ellipses nor symmetric.

**Figure 15.4** (a) Phase-space trajectories for a pendulum including "over the top" motions. (b) The corresponding _θ_ dependence of the potential.

In Figures 15.3–15.7, we show various phase-space structures. _Study these figures and their captions_ and note the following:

  * The orbits of anharmonic oscillations will still be ellipse like, but with angular corners that become more distinct with increasing nonlinearity.
  * Closed trajectories describe periodic oscillations (the same ( _x_ , _v_ ) occur again and again), with a clockwise motion arising from the restoring torque.
  * Open orbits correspond to nonperiodic or "running" motion (a pendulum rotating like a propeller).
  * Regions of space where the potential is repulsive lead to open trajectories in phase space (Figure 15.3).
  * As seen in Figure 15.4a, the separatrix corresponds to the trajectory in phase space that separates open and closed orbits. Motion on the separatrix itself is indeterminant, as the pendulum may balance, or move either way at the maxima of _V_ ( _θ_ ).
  * Friction may cause the energy to decrease with time and the phase space orbit to spiral into a _fixed point_. Yet if the system is driven, it would not remain there.
  * For certain parameters, a closed _limit cycle_ occurs in which the energy pumped in by the external torque exactly balances that lost by friction (Figure 15.5b).
  * Because solutions for different initial conditions are unique, different orbits do not cross. Nonetheless, open orbits join at points of unstable equilibrium ( _hyperbolic points_ in Figure15.3a) where an indeterminacy exists.

### 15.2.1 Chaos in Phase Space

It is easy to solve the nonlinear ODE (15.4) on the computer using our usual techniques. However, it is not so easy to understand the solutions because they are so rich in complexity. The solutions are easier to understand in phase space, particularly if you learn to recognize some characteristic structures there. Actually, there are a number of "tools" that can be used to decide if a system is chaotic in contrast to just complex. Geometric structures in phase space is one of them, and determination of the Lyupanov coefficient (discussed in Section 14.8) is another. Both signal the simplicity lying within the complexity.

**Figure 15.5** (a) Position vs. time for two initial conditions of a chaotic pendulum that end up with the same limit cycle. (b) A phase space plot of position versus velocity for the limit cycle shown in (a) (courtesy of W. Hager).

What may be surprising is that even though the ellipse-like figures in phase space were observed originally for free systems with no friction and no driving torque, similar structures continue to exist for driven systems with friction. The actual trajectories may not remain on a single structure for all times, but they are _attracted_ to them and return to them often. In contrast to periodic motion which produces closed figures in phase-space, random motion appears as a diffuse cloud filling the entire energetically accessible region. Complex or chaotic motion falls someplace in between (Figure 15.6b). If viewed for long times and for many initial conditions, chaotic trajectories in phase space, while resembling the familiar geometric figures, may contain dark or diffuse _bands_ rather than single lines. The continuity of trajectories within bands means that a continuum of solutions are possible and that the system flows continuously among the different trajectories forming the band. The transitions among different phase-space orbits appear chaotic in normal space. The bands are what makes the solutions hypersensitive to the initial conditions as the slightest change in them causes the system to flow to nearby phase-space trajectories.

Pick out the following phase-space structures in your simulations:

**Limit cycles** When a chaotic pendulum is driven by a not-too-large driving torque, it is possible to pick the magnitude for this torque such that after the initial transients die off, the average energy put into the system during one period exactly balances the average energy dissipated by friction during that period (Figure 15.5):

(15.14)

**Figure 15.6** Position vs. time, phase-space plot, and Fourier spectrum for a chaotic pendulum with _ω_ 0 = 1, _a_ = 0.2, _f_ = 0.52, and _ω_ = 0.666 and three different initial conditions. Column (a) displays three dominant cycles, the column (b) only one, while column (c) has multiple cycles. (Examples of chaotic behavior can be seen in Figure 15.7).

This leads to _limit cycles_ that appear as closed ellipse-like figures. (Yet unstable solutions may make sporadic jumps between limit cycles.)

**Predictable attractors** There are orbits, such as fixed points and limit cycles, into which the system settles or returns to often, and that are not particularly sensitive to initial conditions. If your location in phase space is near a predictable attractor, ensuing times will bring you to it.

**Strange attractors** Well-defined, yet complicated, semiperiodic behaviors that appear to be uncorrelated with the motion at an earlier time. They are distinguished from predictable attractors by being fractal (Chapter 16) chaotic, and highly sensitive to the initial conditions (José and Salatan, 1998). Even after millions of oscillations, the motion remains _attracted_ to them.

**Chaotic paths** Regions of phase space that appear as filled-in bands rather than lines. Continuity within the bands implies complicated behaviors, yet still with simple underlying structure.

**Figure 15.7** Some examples of complicated behaviors of a realistic pendulum. From top to bottom in each column of the figure, the first three rows show _θ_ ( _t_ ) behavior, a broadband Fourier spectrum, a phase-space diagram containing regular patterns with dark bands, and a broad Fourier spectrum. These features are characteristic of chaos. In the bottom two rows we see how the behavior changes abruptly after a slight change in the magnitude of the force and that for _f_ = 0.54 there occur the characteristic broad bands of chaos.

**Mode locking** When the magnitude _f_ of the driving torque is larger than that for a limit cycle (15.14), the driving torque can overpower the natural oscillations, resulting in a steady-state motion at the frequency of the driver. This is called _mode locking_. While mode locking can occur for linear or nonlinear systems, for nonlinear systems the driving torque may lock onto the system by exciting an overtone, leading to a rational relation between the driving frequency and the natural frequency:

(15.15)

**Butterfly effects** One of the classic quips about the hypersensitivity of chaotic systems to the initial conditions is that the weather pattern in North America is hard to predict well because it is sensitive to the flapping of butterfly wings in South America. Although this appears to be counterintuitive because we know that systems with essentially identical initial conditions should behave the same, eventually the systems diverge. The applet `**Pend2**` (Figure15.8c and d) lets you compare two simulations of nearly identical initial conditions. As seen in Figure 15.8b and d, the initial conditions for both pendulums differ by only 1 part in 917, and so the initial paths in phase space are the same. Nonetheless, at just the time shown here, the pendulums balance in the vertical position, and then one falls before the other, leading to differing oscillations and differing phase-space plots from this time onward.

### 15.2.2 Assessment in Phase Space

The challenge in understanding simulations of the chaotic pendulum (15.4) is that the 4D parameter space ( _ω_ 0, _α_ , _f_ , _ω_ ) is so immense that only sections of it can be studied systematically. We would expect that sweeping through driving frequency _ω_ should show resonances and beating; sweeping through the frictional force _α_ should show underdamping, critical damping, and overdamping; and sweeping through the driving torque _f_ should show mode locking (for the right values of _ω_ ). All these behaviors can be found in the solution of your differential equation, yet they are mixed together in complex ways.

In this assessment, you should try to reproduce the behaviors shown in the phase-space diagrams in Figure 15.6 and in the applets in Figure 15.8. _Beware_ : Because the system is chaotic, you should expect that your results will be sensitive to the exact values of the initial conditions and to the details of your integration routine. We suggest that you experiment; start with the parameter values we used to produce our plots and then observe the effects of making _very small_ changes in parameters until you obtain different modes of behavior.

  1. Take your solution to the realistic pendulum and include friction. Run it for a variety of initial conditions, including over-the-top ones. Because no energy is fed to the system, you should see spirals in phase space. Note, if you plot points at uniform time steps without connecting them, then the spacing between points gives an indication of the speed of the pendulum.
  2. Next, verify that with no friction, but with a very small driving torque, you obtain a perturbed ellipse in phase space.

**Figure 15.8** (a,b) Output from the applet `**Pend1**` that produces an animation of a chaotic pendulum, along with the corresponding position vs. time and phase-space plots. (b)The resulting Fourier spectrum produced by `**Pend1**`. (c,d) outputs from two runs of the applet `**Pend2**` producing an animation of two chaotic pendula along with the corresponding phase-space plots, and the final output with limit cycles (dark bands).

  3. Set the driving torque's frequency close to the natural frequency _ω_ 0 of the pendulum and search for beats (Figure 15.2b). Note that in addition to the frequency, you may need to adjust the magnitude and phase of the driving torque to avoid an "impedance mismatch" between the pendulum and driver.
  4. Finally, scan the frequency _ω_ of the driving torque and search for nonlinear resonance (it looks like beating).
  5. _Explore chaos_ : Start off with the initial conditions we used in Figure 15.6,

(15.16)

To save time and storage, you may want to use a larger time step for plotting than the one used to solve the differential equations.

  6. Identify which parts of the phase-space plots correspond to transients. (The applets may help you with this, especially if you watch the phase-space features being built up in time.)
  7. Ensure that you have found:
    1. a) a period-3 limit cycle where the pendulum jumps between three major orbits in phase space,
    2. b) a running solution where the pendulum keeps going over the top,
    3. c) chaotic motion in which some paths in the phase space appear as bands.
  8. Look for the "butterfly effect" (Figure 15.8c and d). Start two pendulums off with identical positions but with velocities that differ by 1 part in 1000. Notice that the initial motions are essentially identical but eventually diverge.

## 15.3 Exploration: Bifurcations of Chaotic Pendulums

We have seen that a chaotic system contains a number of dominant frequencies and that the system tends to "jump" from one to another. This means that the dominant frequencies occur sequentially, in contrast to linear systems where they occur simultaneously. We now want to explore this jumping as a computer experiment. If we sample the instantaneous angular velocity   = d _θ_ / d _t_ of our chaotic simulation at various instances in time, we should obtain a series of values for the frequency, with the major Fourier components occurring more often than others.1) These are the frequencies to which the system is _attracted_. That being the case, if we make a scatterplot of the sampled  s for many times at one particular value of the driving force and then change the magnitude of the driving force slightly and sample the frequencies again, the resulting plot may show distinctive patterns of frequencies. That a bifurcation diagram similar to the one for bug populations results is one of the mysteries of life.

In the scatterplot in Figure 15.9, we sample   for the motion of a chaotic pendulum with a vibrating pivot point (in contrast to our usual vibrating external torque). This pendulum is similar to our chaotic one (15.2), but with the driving force depending on sin _θ_ :

(15.17)

**Figure 15.9** A bifurcation diagram for the damped pendulum with a vibrating pivot (see also the similar diagram for a double pendulum, Figure 15.11). The ordinate is | d _θ_ /d _t_ |, the absolute value of the instantaneous angular velocity at the beginning of the period of the driver, and the abscissa is the magnitude of the driving force _f_. Note that the heavy line results from the overlapping of points, not from connecting the points (see enlargement in the inset).

Essentially, the acceleration of the pivot is equivalent to a sinusoidal variation of _g_ or  . Analytic and numeric studies of this system are in the literature (Landau and Lifshitz, 1976; DeJong, 1992; Gould _et al_., 2006). To obtain the bifurcation diagram in Figure 15.9:

  1. Use the initial conditions _θ_ (0) = 1 and  (0) = 1.
  2. Set _α_ = 0.1, _ω_ 0 = 1, and _ω_ = 2, and vary 0 ≤ _f_ ≤ 2.25.
  3. For each value of _f_ , wait 150 periods of the driver before sampling to permit transients to die off. Sample   for 150 times at the instant the driving force passes through zero.
  4. Plot the 150 values of |   | vs. _f_.

## 15.4 Alternate Problem: The Double Pendulum

For those of you who have already studied a chaotic pendulum, an alternative is to study a double pendulum without any small-angle approximation (Figures 15.1b and 15.10, and animation `**DoublePend.mp4**`). A double pendulum has a second pendulum connected to the first, and because each pendulum acts as a driving force for the other, we need not include an external driving torque to produce a chaotic system (there are enough degrees of freedom without it).

The equations of motions for the double pendulum are derived most directly from the Lagrangian formulation of mechanics. The Lagrangian is fairly simple but has the _θ_ 1 and _θ_ 2 motions innately coupled:

(15.18)

**Figure 15.10** Photographs of a double pendulum built by a student in the OSU Physics Department. The upper pendulum consists of two separated shafts so that the lower one can rotate completely around. Both pendula can go over their tops. The first two frames show the pendulum released from rest and then moving quickly. The photography with faster shutter speeds stops the motion in various stages (photograph by R. Landau).

Usually, textbooks approximate these equations for small oscillations, which diminish the nonlinear effects, and results in "slow" and "fast" modes that look much like regular harmonic motions. What is more interesting is the motion that results without any small-angle restrictions, particularly when the pendula have enough initial energy to go over the top (Figure 15.10). In Figure 15.11a, we see several phase-space plots for the lower pendulum with _m_ 1 = 10 _m_ 2. When given enough initial kinetic energy to go over the top, the trajectories are seen to flow between two major attractors as energy is transferred back and forth between the pendula.

In Figure 15.11b is a bifurcation diagram for the double pendulum. This was created by sampling and plotting the instantaneous angular velocity  2 of the lower pendulum at 70 times as the pendulum passed through its equilibrium position. The mass of the upper pendulum (a convenient parameter) was then changed, and the process repeated. The resulting structure is fractal and indicates bifurcations in the number of dominant frequencies in the motion. A plot of the Fourier or wavelet spectrum as a function of mass is expected to show similar characteristic frequencies.

**Figure 15.11** (a) Phase-space trajectories for a double pendulum with _m_ 1 = 10 _m_ 2 and with two dominant attractors. (b) A bifurcation diagram for the double pendulum displaying the instantaneous velocity of the lower pendulum as a function of the mass of the upper pendulum (both plots courtesy of J. Danielson).

## 15.5 Assessment: Fourier/Wavelet Analysis of Chaos

We have seen that a realistic pendulum experiences a restoring torque, _τ g_ ∝ sin _θ_   _θ_ − _θ_ 3 / 3! + _θ_ 5 / 5! +..., that contains nonlinear terms that lead to nonharmonic behavior. In addition, when a realistic pendulum is driven by an external sinusoidal torque, the pendulum may mode-lock with the driver and consequently oscillate at a frequency that is rationally related to the driver's. Consequently, the behavior of the realistic pendulum is expected to be a combination of various periodic behaviors, with discrete jumps between modes.

In this assessment, you should determine the Fourier components present in the pendulum's complicated and chaotic behaviors. You should show that a three-cycle structure, for example, contains three major Fourier components, while a five-cycle structure has five. You should also notice that when the pendulum goes over the top, its spectrum contains a steady-state (DC) component.

  1. Dust off your program for analyzing a _y_ ( _t_ ) into Fourier components.
  2. Apply your analyzer to the solution of the chaotic pendulum for the cases where there are one-, three-, and five-cycle structures in phase space. Deduce the major frequencies contained in these structures. Wait for the transients to die out before conducting your analysis.
  3. Compare your results with the output of the `**Pend1**` applet (Figure 15.8a and b).
  4. Try to deduce a relation among the Fourier components, the natural frequency _ω_ 0, and the driving frequency _ω_.
  5. A classic signal of chaos is a broadband, although not necessarily flat, Fourier spectrum. Examine your system for parameters that give chaotic behavior and verify this statement by plotting the power spectrum in both linear and semilogarithmic plots. (The power spectrum often varies over several orders of magnitude.)

**Wavelet Exploration** We saw in Chapter 13 that a wavelet expansion is more appropriate than a Fourier expansion for a signal containing components that occur for finite periods of time. Because chaotic oscillations are just such signals, repeat the Fourier analysis of this section using wavelets instead of sines and cosines. Can you discern the temporal sequence of the various components?

## 15.6 Exploration: Alternate Phase-Space Plots

Imagine that you have measured the displacement of some system as a function of time. Your measurements appear to indicate characteristic nonlinear behaviors, and you would like to check this by making a phase-space plot but without going to the trouble of measuring the conjugate momenta to plot vs. displacement. Amazingly enough, one may instead plot _x_ ( _t_ \+ _τ_ ) vs. _x_ ( _t_ ) as a function of time to obtain a phase-space plot (Abarbanel _et al_., 1993). Here _τ_ is a _lag time_ and should be chosen as some fraction of a characteristic time for the system under study. While this may not seem like a valid way to make a phase-space plot, recall the forward difference approximation for the derivative,

(15.19)

We see that plotting _x_ ( _t_ \+ _τ_ ) vs. _x_ ( _t_ ) is thus similar to plotting _v_ ( _t_ ) vs. _x_ ( _t_ ).

**Exercise** Create a phase-space plot from the output of your chaotic pendulum by plotting _θ_ ( _t_ \+ _τ_ ) vs. _θ_ ( _t_ ) for a large range of _t_ values. Explore how the graphs change for different values of the lag time _τ_. Compare your results to the conventional phase-space plots you obtained previously.

## 15.7 Further Explorations

  1. The nonlinear behavior in once-common objects such as vacuum tubes and metronomes is described by the _van der Pool Equation_ ,

(15.20)

Verify that the behavior predicted for these systems is _self-limiting_ because the equation contains a limit cycle that is also a predictable attractor. You can think of (15.20) as describing an oscillator with _x_ -dependent damping (the _μ_ term). If _x_ > _x_ 0, friction slows the system down; if _x_ < _x_ 0, friction speeds the system up. Orbits internal to the limit cycle spiral out until they reach the limit cycle; orbits external to it spiral in.

  2. _Duffing Oscillator:_ Another damped, driven nonlinear oscillator is

(15.21)

While similar to the chaotic pendulum, it is easier to find multiple attractors for this oscillator. Perform a phase-space analysis for this equation. (Moon and Li, 1985).

  3. _Lorenz Attractors:_ In 1961, Edward Lorenz was using a simplified atmospheric convection model to predict weather patterns, when, as a shortcut, he entered the decimal 0.506 instead of entering the full 0.506 127 for a parameter in the model (Peitgen _et al_., 1994; Motter and Campbell, 2013). The results for the two numbers were so different that at first he thought it to be a numerical error, but in time he realized that this was a nonlinear system with chaotic behavior. Now we want you to repeat this discovery.
With simplified variables, the equation used by Lorenz are

(15.22)

(15.23)

(15.24)

where _x_ ( _t_ ) is a measure of fluid velocity as a function of time _t_ , _y_ ( _t_ ) and _z_ ( _t_ ) are measures of the temperature distributions in two directions, and _σ_ , _ρ_ , and _β_ are parameters. Note that the _xz_ and _xy_ terms make these equations nonlinear.

    1. a) Modify your ODE solver to handle these three, simultaneous Lorenz equations.
    2. b) To start, use parameter values _σ_ = 10, _β_ = 8/3, and _ρ_ = 28.
    3. c) Make sure to use a small enough step size so that good precision is obtained. You must have confidence that you are seeing chaos and not numerical error.
    4. d) Make plots of _x_ vs. _t_ , _y_ vs. _t_ , and _z_ vs. _t_.
    5. e) The initial behaviors in these plots are called "transients" and are not considered dynamically interesting. Leave off these transients in the plots to follow.
    6. f) Make a "phase-space" plot of _z_ ( _t_ ) vs. _x_ ( _t_ ) (the independent variable _t_ does not appear in such a plot). The distorted, number eight-like figures you obtain (Figure 15.12) are called Lorenz attractors, "attractors" because even chaotic solutions tend to be attracted to them.
    7. g) Make phase-space plots of _y_ ( _t_ ) vs. _x_ ( _t_ ), and _z_ ( _t_ ) vs. _y_ ( _t_ ).
    8. h) Make a 3D plot of _x_ ( _t_ ) vs. _y_ ( _t_ ) vs. _z_ ( _t_ ).
    9. i) The parameters given to you should lead to chaotic solutions. Check this claim by seeing how small a change you can make in a parameter value and still, eventually, obtain different answers.
  4. _A 3D Computer Fly:_ Make _x_ ( _t_ ) vs. _y_ ( _t_ ), _x_ ( _t_ ) vs. _z_ ( _t_ ), and _y_ ( _t_ ) vs. _z_ ( _t_ ) phase-space plots of the equations

(15.25)

Here the parameter _e_ controls the degree of apparent randomness.
  5. _Hénon–Heiles Potential:_ The potential and Hamiltonian

(15.26)

**Figure 15.12** A 3D plot of a Lorenz attractor output from the program `**LorentzAtract.py**` (on the Instructor's disk).

are used to describe three interacting astronomical objects. The potential binds the objects near the origin but releases them if they move far out. The equations of motion follow from the Hamiltonian equations:

(15.27)

    1. a) Numerically solve for the position [ _x_ ( _t_ ), _y_ ( _t_ )] for a particle in the Hénon–Heiles potential.
    2. b) Plot [ _x_ ( _t_ ), _y_ ( _t_ )] for a number of initial conditions. Check that the initial condition _E_ < 1/6 leads to a bounded orbit.
    3. c) Produce a Poincaré section in the ( _y_ , _p y_) plane by plotting ( _y_ , _p y_) each time an orbit passes through _x_ = 0.

1) We refer to this angular velocity as   because we have already used _ω_ for the frequency of the driver and _ω_ 0 for the natural frequency.

# 16
Fractals and Statistical Growth Models

* * *

It is common to notice regular and eye-pleasing natural objects, such as plants and sea shells, that do not have well-defined geometric shapes. When analyzed mathematically with a prescription that normally produces integers, some of these objects are found to have a dimension that is a fractional number, and so they are called "fractals." In this chapter, we implement simple, statistical models that grow fractals. To the extent that these models generate structures that look like those in nature, it is reasonable to assume that the natural processes must be following similar rules arising from the basic physics or biology that creates the objects. Detailed applications of fractals can be found in the literature (Mandelbrot, 1982; Armin and Shlomo, 1991; Sander _et al_., 1994; Peitgen _et al_., 1994).

* * *

## 16.1 Fractional Dimension (Math)

Benoit Mandelbrot, who first studied fractional-dimension figures with supercomputers at IBM Research, gave them the name _fractals_ (Mandelbrot, 1982). Some geometric objects, such as Koch curves, are exact fractals with the same dimension for all their parts. Other objects, such as bifurcation curves, are statistical fractals in which elements of chaos occur and the dimension can be defined only for each part of the object, or on the average.

Consider an abstract object such as the density of charge within an atom. There are an infinite number of ways to measure the "size" of this object. For example, each moment 〈 _r n_〉 is a measure of the size, and there is an infinite number of such moments. Likewise, when we deal with complicated objects, there are different definitions of dimension and each may give a somewhat different value.

Our first definition of the dimension _d_ f, the _Hausdorff–Besicovitch dimension_ , is based on our knowledge that a line has dimension 1, a triangle has dimension 2, and a cube has dimension 3. It seems perfectly reasonable to ask if there is some mathematical formula that agrees with our experience with regular objects, yet can also be used for determining fractional dimensions. For simplicity, let us consider objects that have the same length _L_ on each side, as do equilateral triangles and squares, and that have uniform density. We postulate that the dimension of an object is determined by the dependence of its total mass upon its length:

(16.1)

where the power _d_ f is the _fractal dimension_. As you may verify, this rule works with the 1D, 2D, and 3D regular figures in our experience, so it is a reasonable to try it elsewhere. When we apply (16.1) to fractal objects, we end up with fractional values for _d_ f. Actually, we will find it easier to determine the fractal dimension not from an object's mass, which is _extensive_ (depends on size), but rather from its density, which is _intensive_. The density is defined as mass/length for a linear object, as mass/area for a planar object, and as mass/volume for a solid object. That being the case, for a planar object we hypothesize that

(16.2)

## 16.2 The Sierpiński Gasket (Problem 1)

To generate our first fractal (Figure 16.1), we play a game of chance in which we place dots at points picked randomly within a triangle (Bunde and Havlin, 1991). Here are the rules (which you should try out in the margins now).

  1. Draw an equilateral triangle with vertices and coordinates:

(16.3)

  2. Place a dot at an arbitrary point _P_ = ( _x_ 0, _y_ 0) within this triangle.
  3. Find the next point by selecting randomly the integer 1, 2, or 3:
    1. a) If 1, place a dot halfway between _P_ and vertex 1.
    2. b) If 2, place a dot halfway between _P_ and vertex 2.
    3. c) If 3, place a dot halfway between _P_ and vertex 3.
  4. Repeat the process using the last dot as the new _P_.

Mathematically, the coordinates of successive points are given by the formulas

(16.4)

where _r i_ is a random number between 0 and 1 and where the _integer_ function outputs the closest integer smaller than or equal to the argument. After 15 000 points, you should obtain a collection of dots like those in Figure 16.1a.

### 16.2.1 Sierpiński Implementation

Write a program to produce a Sierpiński gasket. Determine empirically the fractal dimension of your figure. Assume that each dot has mass 1 and that _p_ = _CL_ α. (You can have the computer do the counting by defining an array _box_ of all 0 values and then change a 0 to a 1 when a dot is placed there.)

**Figure 16.1** (a) A statistical fractal Sierpiński gasket containing 10 000 points. Note the self-similarity at different scales. (b) A geometric Sierpiński gasket constructed by successively connecting the midpoints of the sides of each equilateral triangle. The first three steps in the process are labeled as A, B, C.

### 16.2.2 Assessing Fractal Dimension

The topology in Figure 16.1 was first analyzed by the Polish mathematician Sierpiński. Observe that there is the same structure in a small region as there is in the entire figure. In other words, if the figure had infinite resolution, any part of the figure could be scaled up in size and would be similar to the whole. This property is called _self-similarity_.

We construct a nonstatistical form of the Sierpiński gasket by removing an inverted equilateral triangle from the center of all filled equilateral triangles to create the next figure (Figure 16.1b). We then repeat the process ad infinitum, scaling up the triangles so each one has side _r_ = 1 after each step. To see what is unusual about this type of object, we look at how its density (mass/area) changes with size, and then apply (16.2) to determine its fractal dimension. Assume that each triangle has mass _m_ and assign unit density to the single triangle:

(16.5)

Next, for the equilateral triangle with side _L_ = 2, the density is

(16.6)

We see that the extra white space in Figure 16.1b leads to a density that is 3/4 that of the previous stage. For the structure in Figure 16.1c, we obtain

(16.7)

We see that as we continue the construction process, the density of each new structure is 3/4 that of the previous one. Interesting. Yet in (16.2) we derived that

(16.8)

Equation 16.8 implies that a plot of the logarithm of the density _ρ_ vs. the logarithm of the length _L_ for successive structures yields a straight line of slope

(16.9)

As applied to our problem,

(16.10)

As is evident in Figure 16.1, as the gasket grows larger (and consequently more massive), it contains more open space. So despite the fact that its mass approaches infinity as _L_ → ∞, its density approaches zero! And because a 2D figure like a solid triangle has a constant density as its length increases, a 2D figure has a slope equal to 0. Because the Sierpiński gasket has a slope _d_ f − 2   − 0.415 04, it fills space to a lesser extent than a 2D object but more so than a ID object; it is a fractal with dimension 1.6.

## 16.3 Growing Plants (Problem 2)

It seems paradoxical that natural processes subject to chance can produce objects of high regularity and symmetry. For example, it is hard to believe that something as beautiful and graceful as a fern (Figure 16.2a) has random elements in it. Nonetheless, there is a clue here in that much of the fern's beauty arises from the similarity of each part to the whole (self-similarity), with different ferns similar but not identical to each other. These are characteristics of fractals. Your _problem_ is to discover if a simple algorithm including some randomness can draw regular ferns. If the algorithm produces objects that resemble ferns, then presumably you have uncovered mathematics similar to that responsible for the shapes of ferns.

### 16.3.1 Self-Affine Connection (Theory)

In (16.4), which defines mathematically how a Sierpiński gasket is constructed, a _scalingfactor_ of 1/2 is part of the relation of one point to the next. A more general transformation of a point _P_ = ( _x_ , _y_ ) into another point _P_ ′ = ( _x_ ′, _y_ ′) via _scaling_ is

(16.11)

**Figure 16.2** (a) A fractal fern generated by 30 000 iterations of the algorithm (16.14). Enlarging this fern shows that each frond with a similar structure. (b) A fractal tree created with the simple algorithm (16.17).

If the scale factor _s_ > 0, an amplification occurs, whereas if _s_ < 0, a reduction occurs. In our definition (16.4) of the Sierpiński gasket, we also added in a constant _a n_. This is a _translation operation_ , which has the general form

(16.12)

Another operation, not used in the Sierpiński gasket, is a _rotation_ by angle _θ_ :

(16.13)

The entire set of transformations, scalings, rotations, and translations defines an _affine transformation_ (affine denotes a close relation between successive points). The transformation is still considered affine even if it is a more general linear transformation with the coefficients not all related by a single _θ_ (in that case, we can have contractions and reflections). What is important is that the object created with these rules turns out to be self-similar; each step leads to new parts of the object that bear the same relation to the ancestor parts as the ancestors did to theirs. This is what makes the object look similar at all scales.

### 16.3.2 Barnsley's Fern Implementation

We obtain a Barnsley's fern (Barnsley and Hurd, 1992) by extending the dots game to one in which new points are selected using an affine connection with some elements of chance mixed in:

(16.14)

To select a transformation with probability _P_ , we select a uniform random number 0 ≤ _r_ ≤ 1 and perform the transformation if _r_ is in a range proportional to _P_ :

(16.15)

The rules (16.14) and (16.15) can be combined into one:

(16.16)

Although (16.14) makes the basic idea clearer, (16.16) is easier to program, which we do in Listing 16.1.

The starting point in Barnsley's fern (Figure 16.2) is ( _x_ 1, _y_ 1) = (0.5,0.0), and the points are generated by repeated iterations. An important property of this fern is that it is not completely self-similar, as you can see by noting how different the stems and the fronds are. Nevertheless, the stem can be viewed as a compressed copy of a frond, and the fractal obtained with (16.14) is still _self-affine_ , yet with a dimension that varies from part to part in the fern.

Listing 16.1 **Fern3D.py** simulates the growth of ferns in 3D.

### 16.3.3 Self-Affinity in Trees Implementation

Now that you know how to grow ferns, look around and notice the regularity in trees (such as in Figure 16.2b). Can it be that this also arises from a self-affine structure? Write a program, similar to the one for the fern, starting at ( _x_ 1, _y_ 1) = (0.5, 0.0) and iterating the following self-affine transformation:

(16.17)

## 16.4 Ballistic Deposition (Problem 3)

There are a number of physical and manufacturing processes in which particles are deposited on a surface and form a film. Because the particles are evaporated from a hot filament, there is randomness in the emission process yet the produced films turn out to have well-defined, regular structures. Again we suspect fractals. Your _problem_ is to develop a model that simulates this growth process and compare your produced structures to those observed.

### 16.4.1 Random Deposition Algorithm

The idea of simulating random depositions was first reported by Vold (1959), which includes tables of random numbers used to simulate the sedimentation of moist spheres in hydrocarbons. We shall examine a method of simulation that results in the deposition shown in Figure 16.3 (Family and Vicsek, 1985).

Consider particles falling onto and sticking to a horizontal line of length _L_ composed of 200 deposition sites. All particles start from the same height, but to simulate their different velocities, we assume they start at random distances from the left side of the line. The simulation consists of generating uniform random sites between 0 and _L,_ and having a particle stick to the site on which it lands. Because a realistic situation may have columns of aggregates of different heights, the particle may be stopped before it makes it to the line, or it may bounce around until it falls into a hole. We therefore assume that if the column height at which the particle lands is greater than that of both its neighbors, it will add to that height. If the particle lands in a hole, or if there is an adjacent hole, it will fill up the hole. We speedup the simulation by setting the height of the hole equal to the maximum of its neighbors:

**Figure 16.3** A simulation of the ballistic deposition of 20 000 particles onto a substrate of length 200. The vertical height increases in proportion to the length of deposition time, with the top being the final surface.

  1. Choose a random site _r_.
  2. Let the array _h r_ be the height of the column at site _r_.
  3. Make the decision:

(16.18)

Our simulation is `**Fractals/Film.py**` with the essential loop:

The results of this type of simulation show several empty regions scattered throughout the line (Figure 16.3), which is an indication of the statistical nature of the process while the film is growing. Simulations by Fereydoon reproduced the experimental observation that the average height increases linearly with time and produced fractal surfaces. (You will be asked to determine the fractal dimension of a similar surface as an exercise.)

**Exercise** Extend the simulation of random deposition to two dimensions, so rather than making a line of particles we now deposit an entire surface.

## 16.5 Length of British Coastline (Problem 4)

In 1967, Mandelbrot (Mandelbrot, 1967) asked a classic question, "How long is the coast of Britain?" If Britain had the shape of Colorado or Wyoming, both of which have straight-line boundaries, its perimeter would be a curve of dimension 1 with finite length. However, coastlines are geographic not geometric curves, with each portion of the coast sometimes appearing self-similar to the entire coast. If the perimeter of the coast is in fact a fractal, then its length is either infinite or meaningless. Mandelbrot deduced the dimension of the west coast of Britain to be _d_ f = 1.25, which implies infinite length. In your _problem_ we ask you to determine the dimension of the perimeter of one of our fractal simulations.

### 16.5.1 Coastlines as Fractals (Model)

The length of the coastline of an island is the perimeter of that island. While the concept of perimeter is clear for regular geometric figures, some thought is required to give it meaning for an object that may be infinitely self-similar. Let us assume that a map maker has a ruler of length _r_. If she walks along the coastline and counts the number of times _N_ that she must place the ruler down in order to _cover_ the coastline, she will obtain a value for the length _L_ of the coast as _Nr_. Imagine now that the map maker keeps repeating her walk with smaller and smaller rulers. If the coast was a geometric figure or a _rectifiable curve_ , at some point the length _L_ would become essentially independent of _r_ and would approach a constant. Nonetheless, as discovered empirically by Richardson (1961) for natural coastlines such as those of South Africa and Britain, the perimeter appears to be an unusual function of _r_ :

(16.19)

where _M_ and _d_ f are empirical constants. For a geometric figure or for Colorado, _d_ f = 1 and the length approaches a constant as _r_ → 0. Yet for a fractal with _d_ f > 1, the perimeter _L_ → ∞ as _r_ → 0. This means that as a consequence of self-similarity, fractals may be of finite size but have infinite perimeters. Physically, at some point there may be no more details to discern as _r_ → 0 (say, at the quantum or Compton size limit), and so the limit may not be meaningful.

### 16.5.2 Box Counting Algorithm

Consider a line of length _L_ broken up into segments of length _r_ (Figure 16.4a). The number of segments or "boxes" needed to cover the line is related to the size _r_ of the box by

(16.20)

where _C_ is a constant. A proposed definition of fractional dimension is the power of _r_ in this expression as _r_ → 0. In our example, it tells us that the line has dimension _d_ f = 1. If we now ask how many little circles of radius _r_ it would take to _cover_ or fill a circle of area _A_ (Figure 16.4, middle), we will find that

(16.21)

as expected. Likewise, counting the number of little spheres or cubes that can be packed within a large sphere tells us that a sphere has dimension _d_ f = 3. In general, if it takes _N_ little spheres or cubes of side _r_ → 0 to cover some object, then the fractal dimension _d_ f can be deduced as

(16.22)

**Figure 16.4** Examples of the use of "box" counting to determine fractal dimension. (a) In the top "boxes" are circles and the perimeter is being covered. In the bottom an entire figure is being covered. In (b) a "coastline" is being covered by boxes of two different sizes (scales). The fractal dimension can be deduced by recording the number of box of different scale needed to cover the figures.

(16.23)

(16.24)

Here _s_ ∝ 1/ _r_ is called the _scale_ in geography, so _r_ → 0 corresponds to an infinite scale. To illustrate, you may be familiar with the low scale on a map being 10 000 m to a centimeter, while the high scale is 100 m to a centimeter. If we want the map to show small details (sizes), we need a map of high scale.

We will use box counting to determine the dimension of a perimeter, not of an entire figure. Once we have a value for _d_ f, we can determine a value for the length of the perimeter via (16.19). (If you cannot wait to see box counting in action, in the auxiliary online files you will find an applet `**Jfracdim**` that goes through all the steps of box counting before your eyes and even plots the results.)

### 16.5.3 Coastline Implementation and Exercise

Rather than ruin our eyes using a geographic map, we use a mathematical one, specifically, the top portion of Figure 16.3 that may look like a natural coastline. Determine _d_ f by covering this figure, or one you have generated, with a semitransparent piece of graph paper, 1) and counting the number of boxes containing any part of the coastline (Figures 16.4 and 16.5).

**Figure 16.5** Fractal dimensions of a line, box, and coastline determined by box counting. The slope at vanishingly small scale determines the dimension.

  1. Print your coastline graph with the same physical scale ( _aspect ratio_ ) for the vertical and horizontal axes. This is required because the graph paper you will use for box counting has square boxes and so you want your graph to also have the same vertical and horizontal scales. Place a piece of graph paper over your printout and look though the graph paper at your coastline. If you do not have a piece of graph paper available, or if you are unable to obtain a printout with the same aspect ratio for the horizontal and vertical axes, add a series of closely spaced horizontal and vertical lines to your coastline printout and use these lines as your graph paper. (Box counting should still be accurate if both your coastline and your graph paper have the same aspect ratios.)
  2. The vertical height in our printout was 17 cm, and the largest division on our graph paper was 1 cm. This sets the scale of the graph as 1 : 17, or _s_ = 17 for the largest divisions (lowest scale). Measure the vertical height of your fractal, compare it to the size of the biggest boxes on your "piece" of graph paper, and thus determine your lowest scale.
  3. With our largest boxes of 1 cm × 1 cm, we found that the coastline passed through _N_ = 24 large boxes, that is, that 24 large boxes covered the coastline at _s_ = 17. Determine how many of the largest boxes (lowest scale) are needed to cover your coastline.
  4. With our next smaller boxes of 0.5 cm × 0.5 cm, we found that 51 boxes covered the coastline at a scale of _s_ = 34. Determine how many of the midsize boxes (midrange scale) are needed to cover your coastline.
  5. With our smallest boxes of 1 mm × 1 mm, we found that 406 boxes covered the coastline at a scale of _s_ = 170. Determine how many of the smallest boxes (highest scale) are needed to cover your coastline.
  6. Equation 16.24 tells us that as the box sizes get progressively smaller, we have

Clearly, only the relative scales matter because the proportionality constants cancel out in the ratio. A plot of log _N_ vs. log _s_ should yield a straight line. In our example we found a slope of _d_ f = 1.23. Determine the slope and thus the fractal dimension for your coastline. Although only two points are needed to determine the slope, use your lowest scale point as an important check. (Because the fractal dimension is defined as a limit for infinitesimal box sizes, the highest scale points are more significant.)

  7. As given by (16.19), the perimeter of the coastline

(16.25)

If we keep making the boxes smaller and smaller so that we are looking at the coastline at higher and higher scale _and_ if the coastline is self-similar at all levels, then the scale _s_ will keep getting larger and larger with no limits (or at least until we get down to some quantum limits). This means

(16.26)

Does your fractal implies an infinite coastline? Does it make sense that a small island like Britain, which you can walk around, has an infinite perimeter?

## 16.6 Correlated Growth, Forests, Films (Problem 5)

It is an empirical fact that in nature there is increased likelihood that a plant will grow if there is another one nearby (Figure 16.6a). This _correlation_ is also valid for the simulation of surface films, as in the previous algorithm. Your _problem_ is to include correlations in the surface simulation.

### 16.6.1 Correlated Ballistic Deposition Algorithm

A variation of the ballistic deposition algorithm, known as the _correlated ballistic deposition_ , simulates mineral deposition onto substrates on which dendrites form (Tait _et al_., 1990). In Listing 16.2 we extend the previous algorithm to include the likelihood that a freshly deposited particle will attract another particle. We assume that the probability of sticking _P_ depends on the distance _d_ that the added particle is from the last one (Figure 16.6b):

(16.27)

Here _η_ is a parameter and _c_ is a constant that sets the probability scale.2) For our implementation we choose _η_ = −2, which means that there is an inverse square attraction between the particles (decreased probability as they get farther apart).

**Figure 16.6** (a) A view that might be seen in the undergrowth of a forest or after a correlated ballistic deposition. (b) The probability of particle _i_ \+ 1 sticking in one column depends upon the distance _d_ from the previously deposited particle _i_.

As in our study of uncorrelated deposition, a uniform random number in the interval [0, _L_ ] determines the column in which the particle will be deposited. We use the same rules about the heights as before, but now a second random number is used in conjunction with (16.27) to decide if the particle will stick. For instance, if the computed probability is 0.6 and if _r_ < 0.6, the particle will be accepted (sticks), if _r_ > 0.6, the particle will be rejected.

## 16.7 Globular Cluster (Problem 6)

Consider a bunch of grapes on an overhead vine. Your _problem_ is to determine how its tantalizing shape arises. In a flash of divine insight, you realize that these shapes, as well as others such as those of dendrites, colloids, and thin-film structure, appear to arise from an aggregation process that is limited by diffusion.

### 16.7.1 Diffusion-Limited Aggregation Algorithm

A model of diffusion-limited aggregation (DLA) has successfully explained the relation between a cluster's perimeter and mass (Witten and Sander, 1983). We start with a 2D lattice containing a seed particle in the middle, draw a circle around the particle, and place another particle on the circumference of the circle at some random angle. We then release the second particle and have it execute a random walk, much like the one we studied in Chapter 4, but restricted to vertical or horizontal jumps between lattice sites. This is a simulation of a type of _Brownian motion_ related to diffusion. To make the model more realistic, we let the length of each step vary according to a random Gaussian distribution. If at some point during its random walk, the particle encounters another particle within one lattice spacing, they stick together and the walk terminates. If the particle passes outside the circle from which it was released, it is lost forever. The process is repeated as often as desired and results in clusters (Figure 16.7 and applet `**dla**`).

**Figure 16.7** (a) A globular cluster of particles of the type that might occur in a colloid, (b) The applet **Dla2en.html** lets you watch these clusters grow. Here the cluster is at the center of the circle, and random walkers are started at random points around the circle.

**Listing 16.2** **Column.py** simulates correlated ballistic deposition of minerals onto substrates on which dendrites form.

  1. Write a subroutine that generates random numbers with a Gaussian distribution. 3)
  2. Define a 2D lattice of points represented by the array `**grid[400,400]**` with all elements initially zero.
  3. Place the seed at the center of the lattice; that is, set `**grid[199,199]=1**`.
  4. Imagine a circle of radius 180 lattice spacings centered at `**grid[199,199]**`. This is the circle from which we release particles.
  5. Determine the angular position of the new particle on the circle's circumference by generating a uniform random angle between 0 and 2π.
  6. Compute the _x_ and _y_ positions of the new particle on the circle.
  7. Determine whether the particle moves horizontally or vertically by generating a uniform random number 0 < _r_ _xy_ < 1 and applying the rule

(16.28)

  8. Generate a Gaussian-weighted random number in the interval [−∞, ∞]. This is the size of the step, with the sign indicating direction.
  9. We now know the total distance and direction the particle will move. It jumps one lattice spacing at a time until this total distance is covered.
  10. Before a jump, check whether a nearest-neighbor site is occupied:
    1. a) If occupied, the particle stays at its present position and the walk is over.
    2. b) If unoccupied, the particle jumps one lattice spacing.
    3. c) Continue the checking and jumping until the total distance is covered, until the particle sticks, or until it leaves the circle.
  11. Once one random walk is over, another particle can be released and the process repeated. This is how the cluster grows.

Because many particles are lost, you may need to generate hundreds of thousands of particles to form a cluster of several hundred particles.

### 16.7.2 Fractal Analysis of DLA or a Pollock

A cluster generated with the DLA technique is shown in Figure 16.7. We wish to analyze it to see if the structure is a fractal and, if so, to determine its dimension. (As an alternative, you may analyze the fractal nature of the Pollock painting in Figure 16.8, a technique used to determine the authenticity of this sort of art.) As a control, _simultaneously_ analyze a geometric figure, such as a square or circle, whose dimension is known. The analysis is a variation of the one used to determine the length of the coastline of Britain.

  1. If you have not already done so, use the box counting method to determine the fractal dimension of a simple square.
  2. Draw a square of length _L_ , small relative to the size of the cluster, around the seed particle. (Small might be seven lattice spacings to a side.)
  3. Count the number of particles within the square.
  4. Compute the density _ρ_ by dividing the number of particles by the number of sites available in the box (49 in our example).
  5. Repeat the procedure using larger and larger squares.
  6. Stop when the cluster is covered.
  7. The (box counting) fractal dimension _d_ f is estimated from a log–log plot of the density _ρ_ vs. _L_. If the cluster is a fractal, then (16.2) tells us that   and the graph should be a straight line of slope _d_ f − 2.

The graph we generated had a slope of −0.36, which corresponds to a fractal dimension of 1.66. Because random numbers are involved, the graph you generate will be different, but the fractal dimension should be similar. (Actually, the structure is multifractal, and so the dimension also varies with position.)

**Figure 16.8** Number 8 by the American painter Jackson Pollock. (Used with permission, Neuberger Museum, State University of New York.) Some researchers claim that Pollock's paintings exhibit a characteristic fractal structure, while some other researchers question this (Kennedy, 2006). See if you can determine the fractal dimensions within this painting.

## 16.8 Fractals in Bifurcation Plot (Problem 7)

Recollect the project involving the logistics map where we plotted the values of the stable population numbers vs. the growth parameter _μ_. Take one of the bifurcation graphs you produced and determine the fractal dimension of different parts of the graph by using the same technique that was applied to the coastline of Britain.

## 16.9 Fractals from Cellular Automata

* * *

We have already indicated in places how statistical models may lead to fractals. There is a class of statistical models known as _cellular automata_ that produce complex behaviors from very simple systems. Here we study some.

* * *

Cellular automata were developed by von Neumann and Ulam in the early 1940s (von Neumann was also working on the theory behind modern computers then). Though very simple, cellular automata have found applications in many branches of science (Peitgen _et al_., 1994; Sipper, 1996). Their classic definition is (Barnsley and Hurd, 1992):

> _A cellular automaton is a discrete dynamical system in which space, time, and the states of the system are discrete. Each point in a regular spatial lattice, called a cell, can have any one of a finite number of states, and the states of the cells in the lattice are updated according to a local rule. That is, the state of a cell at a given time depends only on its own state one time step previously, and the states of its nearby neighbors at the previous time step. All cells on the lattice are updated synchronously, and so the state of the entice lattice advances in discrete time steps._

A cellular automaton in two dimensions consists of a number of square cells that grow upon each other. A famous one is _Conway's Game of Life,_ which we implement in Listing 16.3. In this, cells with value 1 are alive, while cells with value 0 are dead. Cells grow according to the following rules:

  1. If a cell is alive and if two or three of its eight neighbors are alive, then the cell remains alive.
  2. If a cell is alive and if more than three of its eight neighbors are alive, then the cell dies because of overcrowding.
  3. If a cell is alive and only one of its eight neighbors is alive, then the cell dies of loneliness.
  4. If a cell is dead and more than three of its neighbors are alive, then the cell revives.

A variation on the Game of Life is to include a "rule one out of eight" that a cell will be alive if exactly one of its neighbors is alive, otherwise the cell will remain unchanged.

**Listing 16.3** **Gameoflife.py** is an extension of Conway's Game of Life in which cells always revive if one out of eight neighbors is alive.

Early studies of the statistical mechanics of cellular automata were made by (Wolfram, 1983), who indicated how one can be used to generate a Sierpiński gasket. Because we have already seen that a Sierpiński gasket exhibits fractal geometry (Section 16.2), this represents a microscopic model of how fractals may occur in nature. This model uses eight rules, given graphically at the top of Figure 16.9, to generate new cells from old. We see all possible configurations for three cells in the top row, and the begetted next generation in the row below. At the bottom of Figure 16.9 is a Sierpiński gasket of the type created by the applet `**JCellAut**`. This plays the game and lets you watch and control the growth of the gasket.

**Figure 16.9** The rules for two versions of the Game of Life. The rules, given graphically on the top row, create the gaskets below (output obtained from the applet `**JCellAut**` in the auxiliary files).

## 16.10 Perlin Noise Adds Realism

We have already seen in this chapter how statistical fractals are able to generate objects with a striking resemblance to those in nature. This appearance of realism may be further enhanced by including a type of coherent randomness known as _Perlin noise_. The technique we are about to discuss was developed by Ken Perlin of New York University, who won an Academy Award (an Oscar) in 1997 for it and has continued to improve it (Perlin, 1985). This type of coherent noise has found use in important physics simulations of stochastic media (Tickner, 2004), as well as in video games and motion pictures like _Tron_.

**Figure 16.10** The coordinates used in adding Perlin noise. (a) The rectangular grid used to locate a square in space and a corresponding point within the square. As shown with the arrows, unit vectors _g_ _i_ with random orientation are assigned at each grid point. (b) A point within each square is located by drawing the four _p_ _i_. The _g_ _i_ vectors are the same as on the left.

The inclusion of Perlin noise in a simulation adds both randomness and a type of coherence among points in space that tends to make dense regions denser and sparse regions sparser. This is similar to our correlated ballistic deposition simulations (Section 16.6.1) and related to chaos in its long-range randomness and short-range correlations. We start with some known function of _x_ and _y_ and add noise to it. For this purpose, Perlin used the mapping or _ease_ function (Figure 16.11b)

(16.29)

As a consequence of its S shape, this mapping makes regions close to 0 even closer to 0, while making regions close to 1 even closer (in other words, it increases the tendency to clump, which shows up as higher contrast). We then break space up into a uniform rectangular grid of points (Figure 16.10a), and consider a point ( _x_ , _y_ ) within a square with vertices ( _x_ _0_ , _y_ 0), ( _x_ 1, _y_ 0), ( _x_ 0, _y_ 1), and ( _x_ 1, _y_ 1). We next assign unit gradients vectors _**g**_ 0 to _**g**_ 3 with random orientation at each grid point. A point within each square is located by drawing the four _p i_ vectors (Figurel6.10b):

(16.30)

(16.31)

**Figure 16.11** The mapping used in adding Perlin noise. (a) The numbers _s_ , _t_ , _u_ , and _v_ are represented by perpendiculars to the four vertices, with lengths proportional to their values. (b) The function 3p2 − 2p3 is used as a map of the noise at a point like ( _x_ , _y_ ) to others close by.

Next the scalar products of the **_p_** ′s and the **_g_** ′s are formed:

(16.32)

As shown in Figure 16.11a, the numbers _s_ , _t_ , _u_ , and _v_ are assigned to the four vertices of the square and represented there by lines perpendicular to the square with lengths proportional to the values of _s_ , _t_ , _u_ , and _v_ (which can be positive or negative).

The actual mapping proceeds via a number of steps (Figure 16.12):

  1. Transform the point ( _x_ , _y_ ) to ( _s_ x, _s_ y),

(16.33)

  2. Assign the lengths _s_ , _t_ , _u_ , and _v_ to the vertices in the mapped square.
  3. Obtain the height _a_ (Figure 16.12) via linear interpolation between _s_ and _t_.
  4. Obtain the height _b_ via linear interpolation between _u_ and _v_.
  5. Obtain _s y_ as a linear interpolation between _a_ and _b_.
  6. The vector _c_ so obtained is now the two components of the noise at ( _x_ , _y_ ).

### 16.10.1 Ray Tracing Algorithms

Ray tracing is a technique that renders an image of a scene by simulating the way rays of light travel (Pov-Ray, 2013). To avoid tracing rays that do not contribute to the final image, ray-tracing programs start at the viewer, trace rays backward onto the scene, and then back again onto the light sources. You can vary the location of the viewer and light sources and the properties of the objects being viewed, as well as atmospheric conditions such as fog, haze, and fire.

As an example of what can be carried out, in Figure 16.14a we show the output from the ray-tracing program `**Islands.pov**` in Listing 16.4, using as input the coherent random noise as shown in Figure 16.13b. The program options we used are given in Listing 16.4 and are seen to include commands to color the islands, to include waves, and to give textures to the sky and the sea. Pov-Ray also allows the possibility of using Perlin noise to give textures to the objects to be created. For example, the stone cup on the right in Figure 16.14 has a marblelike texture produced by Perlin noise.

**Figure 16.12** Perlin noise mapping, (a) The point (x,y) is mapped to point _(s x,xy)_. (b) Using (16.33). Then three linear interpolations are performed to find _c,_ the noise at (x, y).

**Figure 16.13** After the addition of Perlin noise, the random scatterplot in (a) becomes the clusters on (b).

**Listing 16.4** **Islands.pov** in the Codes/Animations/Fractals directory gives the Pov-Ray ray-tracing commands needed to convert the coherent noise random plot of Figure 16.13 into the mountain like image as shown in Figure 16.14a.

**Figure 16.14** (a) The output from the Pov-Ray ray-tracing program that took as input the 2D coherent random noise plot in Figure 16.13 and added height and fog. (b) An image of a surface of revolution produced by Pov-Ray in which the marble-like texture is created by Perlin noise.

## 16.11 Exercises

  1. Figure 16.9 gives the rules (at top) and the results (below) for two versions of the Game of Life. These results were produced by the applet `**JCellAut**`. Write a Python program that runs the same games. Check that you obtain the same results for the same initial conditions.
  2. Recall how box counting is used to determine the fractal dimension of an object. Imagine that the result of some experiment or simulation is an interesting geometric figure.
    1. a) What might be the physical/theoretical importance of determining that this object is a fractal?
    2. b) What might be the importance of determining its fractal dimension?
    3. c) Why is it important to use _more_ than two sizes of boxes?

1) Yes, we are suggesting a painfully analog technique based on the theory that trauma leaves a lasting impression. If you prefer, you can store your output as a matrix of 1 and 0 values and let the computer do the counting, but this will take more of your time! 2) The absolute probability, of course, must be less than one, but it is nice to choose _c_ so that the relative probabilities produce a graph with easily seen variations. 3) We indicated how to do this in Section 5.22.1.

# 17
Thermodynamic Simulations and Feynman Path Integrals

* * *

We start this chapter by describing how magnetic materials can be simulated by applying the Metropolis algorithm to the Ising model. This extends the Monte Carlo techniques studied in Chapter 4 to include now thermodynamics. Not only do thermodynamic simulations have important practical applications, but they also give us insight into what is "dynamic" in thermodynamics. Toward the middle of the chapter, we describe a recent Monte Carlo algorithm known as Wang–Landau sampling that has shown itself to be far more efficient than the 60-plus-year-old Metropolis algorithm. Wang–Landau sampling is an active subject in present research, and it is nice to see it fitting well into an elementary textbook. We end the chapter by applying the Metropolis algorithm to Feynman's path integral formulation of quantum mechanics (Feynman and Hibbs, 1965). The theory, while the most advanced to be found in this book, forms the basis for field-theoretic computations of quantum chromodynamics, some of the most fundamental and most time-consuming computations in existence. Basic discussions can be found in (Mannheim, 1983; MacKeown, 1985; MacKeown and Newman, 1987), with a recent review in (Potvin, 1993).

* * *

## 17.1 Magnets via Metropolis Algorithm

Ferromagnets contain finite-size _domains_ in which the spins of all the atoms point in the same direction. When an external magnetic field is applied to these materials, the different domains align and the materials become "magnetized." Yet as the temperature is raised, the total magnetism decreases, and at the Curie temperature the system goes through a _phase transition_ beyond which all magnetization vanishes. Your _problem_ is to explain the thermal behavior of ferromagnets.

## 17.2 An Ising Chain (Model)

As our model, we consider _N_ magnetic dipoles fixed in place on the links of a linear chain (Figure 17.1). (It is a straightforward generalization to handle 2D and 3D lattices.) Because the particles are fixed, their positions and momenta are not dynamic variables, and we need worry only about their spins. We assume that the particle at site _i_ has spin _s_ _i_ , which is either up or down:

(17.1)

Each configuration of the _N_ particles is described by a quantum state vector

(17.2)

Because the spin of each particle can assume any one of the _two_ values, there are 2 _N_ different possible states for the _N_ particles in the system. Because fixed particles cannot be interchanged, we do not need to concern ourselves with the symmetry of the wave function.

The energy of the system arises from the interaction of the spins with each other and with the external magnetic field _B_. We know from quantum mechanics that an electron's spin and magnetic moment are proportional to each other, so a magnetic _dipole–dipole_ interaction is equivalent to a _spin–spin_ interaction. We assume that each dipole interacts with the external magnetic field and with its nearest neighbor through the potential:

(17.3)

Here the constant _J_ is called the _exchange energy_ and is a measure of the strength of the spin–spin interaction. The constant g is the gyromagnetic ratio, that is, the proportionality constant between a particle's angular momentum and magnetic moment. The constant _μ_ b = e /(2 _m_ e _c_ ) is the Bohr magneton, the basic measure for magnetic moments.

**Figure 17.1** The 1D lattice of _N_ spins used in the Ising model of magnetism. The interaction energy between nearest-neighbor pairs _E_ = ± _J_ is shown for aligned and opposing spins.

Even for small numbers of particles, the 2 _N_ possible spin configurations gets to be very large (220 > 106), and it is expensive for the computer to examine them all. Realistic samples with ~ 1023 particles are beyond imagination. Consequently, statistical approaches are usually assumed, even for moderate values of _N_. Just how large _N_ must be for this to be accurate is one of the things we want you to explore with your simulations.

The energy of this system in state _α_ _k_ is the expectation value of the sum of the potential _V_ over the spins of the particles:

(17.4)

An apparent paradox in the Ising model occurs when we turn off the external magnetic field and thereby eliminate a preferred direction in space. This means that the average magnetization should vanish despite the fact that the lowest energy state would have all spins aligned. The answer to this paradox is that the system with _B_ = 0 is unstable; even if all the spins are aligned, there is nothing to stop their spontaneous reversal. The instabilities are a type of Bloch-wall transitions in which regions of different spin orientations change size. Indeed, natural magnetic materials have multiple domains with all the spins aligned, but with different domains pointing in different directions.

For simplicity, we assume _B_ = 0, which means that there are just spin–spin interactions. However, be cognizant of the fact that this means there is no preferred direction in space, and so you may have to be careful how you calculate observables when averaging over domains. For example, you may need to take an absolute value of the total spin when calculating the magnetization, that is, to calculate 〈| **Σ** _i_ _s_ _i_ |〉 rather than 〈 **Σ** _i_ _s_ _i_ 〉.

The equilibrium alignment of the spins depends critically on the sign of the exchange energy _J_. If _J_ > 0, the lowest energy state will tend to have neighboring spins aligned. If the temperature is low enough, the ground state will be a _ferromagnet_ with all the spins aligned. Yet if _J_ < 0, the lowest energy state will tend to have neighbors with opposite spins. If the temperature is low enough, the ground state will be a _antiferromagnet_ with alternating spins.

The simple 1D Ising model has its limitations. Although the model is accurate in describing a system in thermal equilibrium, it is not accurate in describing the _approach_ to thermal equilibrium. Also, we have postulated that only one spin is flipped at a time, while real magnetic materials tend to flip many spins at a time. Other limitations are straightforward to improve, for example, the addition of longer range interactions than just nearest neighbors, the motion of the centers, higher multiplicity spin states, and extension to two and three dimensions.

A fascinating aspect of magnetic materials is the existence of a critical temperature, the _Curie temperature,_ above which the gross magnetization essentially vanishes. Below the Curie temperature the system is in a quantum state with macroscopic order; above the Curie temperature there is only short-range order extending over atomic dimensions. Although the 1D Ising model predicts realistic temperature dependences for the thermodynamic quantities, the model is too simple to support a phase transition. However, the 2D and 3D Ising models do support the Curie temperature phase transition (Yang, 1952).

## 17.3 Statistical Mechanics (Theory)

Statistical mechanics starts with the elementary interactions among a system's particles and constructs the macroscopic thermodynamic properties such as specific heats. The essential assumption is that all configurations of the system consistent with the constraints are possible. In some simulations, such as the molecular dynamics ones in Chapter 18, the problem is set up such that the _energy_ of the system is fixed. The states of this type of system are described by what is called a _microcanonical ensemble_. In contrast, for the thermodynamic simulations we study in this chapter, the temperature, volume, and number of particles remain fixed, and so we have what is called a _canonical ensemble_.

When we say that an object is _at_ temperature _T_ , we mean that the object's atoms are in thermodynamic equilibrium with an average kinetic energy proportional to _T_. Although this may be an equilibrium state, it is also a dynamic one in which the object's energy fluctuates as it exchanges energy with its environment (it is thermodynamics after all). Indeed, one of the most illuminating aspects of the simulation to follow its visualization of the continual and random interchange of energy that occurs at equilibrium.

The energy _E α _j__ of state _α j_ in a canonical ensemble is not constant but is distributed with probabilities _P_ ( _α _j__ ) given by the Boltzmann distribution:

(17.5)

Here _k_ is Boltzmann's constant, _T_ the temperature, and _Z_ ( _T_ ) the partition function, a weighted sum over the individual _states_ or _configurations_ of the system. Another formulation, such as the Wang–Landau algorithm discussed in Section 17.5, sums over the _energies_ of the states of the system and includes a density-of-states factor _g_ ( _E_ _i_ ) to account for degenerate states with the same energy. While the present sum over states is a simpler way to express the problem (one less function), we shall see that the sum over energies is more efficient numerically. In fact, we are even able to ignore the partition function _Z_ ( _T_ ) because it cancels out in our forming the _ratio_ of probabilities.

### 17.3.1 Analytic Solution

For very large numbers of particles, the thermodynamic properties of the 1D Ising model can be solved analytically and yields (Plischke and Bergersen, 1994)

(17.6)

(17.7)

The analytic results for the specific heat per particle and the magnetization are

(17.8)

(17.9)

The _2D Ising model_ has an analytic solution, but it is not easy to derive it (Yang, 1952; Huang, 1987). Whereas the internal energy and heat capacity are expressed in terms of elliptic integrals, the spontaneous magnetization per particle has the simple form

(17.10)

(17.11)

where the temperature is measured in units of the Curie temperature _T_ _c_.

## 17.4 Metropolis Algorithm

In trying to devise an algorithm that simulates thermal equilibrium, it is important to understand that the Boltzmann distribution (17.5) does not require a system to remain always in the state of lowest energy, but says that it is less likely for the system to be found in a higher energy state than in a lower energy one. Of course, as _T_ → 0 only the lowest energy state will be populated. For finite temperatures we expect the energy to fluctuate by approximately _k_ B _T_ about the equilibrium value.

In their simulation of neutron transmission through matter, Metropolis, Rosenbluth, Teller, and Teller (Metropolis _et al._ , 1953) invented an algorithm to improve the Monte Carlo calculation of averages. This _Metropolis algorithm_ is now a cornerstone of computational physics because the sequence of configurations it produces (a _Markov chain_ ) accurately simulates the fluctuations that occur during thermal equilibrium. The algorithm randomly changes the individual spins such that, on the average, the probability of a configuration occurring follows a Boltzmann distribution. (We do not find the proof illuminating.)

The Metropolis algorithm is a combination of the variance reduction technique discussed in Section 5.19 and the von Neumann rejection technique discussed in Section 5.21. There we showed how to make Monte Carlo integration more efficient by sampling random points predominantly where the integrand is large and how to generate random points with an arbitrary probability distribution. Now we would like to have spins flip randomly, have a system that can reach any energy in a finite number of steps ( _ergodic_ sampling), and have a distribution of energies described by a Boltzmann distribution, yet have systems that equilibrate quickly enough to compute in reasonable times.

The Metropolis algorithm is implemented via a number of steps. We start with a fixed temperature and an initial spin configuration, and apply the algorithm until a thermal equilibrium is reached (equilibration). Continued application of the algorithm generates the statistical fluctuations about equilibrium from which we deduce the thermodynamic quantities such as the magnetization _M_ ( _T_ ). Then the temperature is changed, and the whole process is repeated in order to deduce the _T_ dependence of the thermodynamic quantities. The accuracy of the deduced temperature dependences provides convincing evidence of the validity of the algorithm. Because the 2 _N_ possible configurations of _N_ particles can be a very large number, the amount of computer time needed can be very long. Typically, a small number of iterations   10 _N_ is adequate for equilibration.

The explicit steps of the Metropolis algorithm are:

  1. Start with an arbitrary spin configuration _α_ _k_ = { _s_ 1, _s_ 2,..., _s_ _N_ }.
  2. Generate a trial configuration _α_ _k_ +1 by
    1. a) picking a particle _i_ randomly and
    2. b) flipping its spin.1)
  3. Calculate the energy _E_ _α_ tr of the trial configuration.
  4. If _E_ _α_ tr ≤ _E_ _α_ _k_ , accept the trial by setting _α_ _k_ +1 = _α_ tr.
  5. If _E_ _α_ tr > _E_ _α_ _k_ , accept with relative probability  :
    1. a) Choose a uniform random number 0 ≤ _r_ _i_ ≤ 1.
    2. b)

The heart of this algorithm is its generation of a random spin configuration _α_ _j_ , (17.2) with probability

(17.12)

The technique is a variation of von Neumann rejection (stone throwing) in which a random _trial_ configuration is either accepted or rejected depending upon the value of the Boltzmann factor. Explicitly, the ratio of probabilities for a trial configuration of energy _E_ t to that of an initial configuration of energy _E_ i is

(17.13)

If the trial configuration has a lower energy (Δ _E_ ≤ 0), the relative probability will be greater than 1 and we will accept the trial configuration as the new initial configuration without further ado. However, if the trial configuration has a higher energy (Δ _E_ > 0), we will not reject it out of hand, but instead accept it with relative probability  . To accept a configuration with a probability, we pick a uniform random number between 0 and 1, and if the probability is greater than this number, we accept the trial configuration; if the probability is smaller than the chosen random number, we reject it. (You can remember which way this goes by letting _E_ _α_ tr → ∞, in which case _P_ → 0 and nothing is accepted.) When the trial configuration is rejected, the next configuration is identical to the preceding one.

How do you start? One possibility is to start with random values of the spins (a "hot" start). Another possibility (Figure 17.2) is a "cold" start in which you start with all spins parallel ( _J_ > 0) or antiparallel ( _J_ < 0). In general, one tries to remove the importance of the starting configuration by letting the calculation run a while (  10 _N_ rearrangements) before calculating the equilibrium thermodynamic quantities. You should get similar results for hot, cold, or arbitrary starts, and by taking their average you remove some of the statistical fluctuations.

**Figure 17.2** An Ising model simulation on a 1D lattice of 100 initially aligned spins (on the left). Up spins are indicated by circles, and down spins by blank spaces. Although the system starts with all up spins (a "cold" start), the system is seen to form domains of up and down spins as time progresses.

### 17.4.1 Metropolis Algorithm Implementation

  1. Write a program that implements the Metropolis algorithm, that is, that produces a new configuration _α_ _k_ +1 from the present configuration _α_ _k_. (Alternatively, use the program `**IsingViz.py**` shown in Listing 17.1.)
  2. Make the key data structure in your program an array `**s[N]**` containing the values of the spins _s_ _i_. For debugging, print out + and − to give the spin at each lattice point and examine the pattern for different trial numbers.
  3. The value for the exchange energy _J_ fixes the energy scale. Keep it fixed at _J_ = 1. (You may also wish to study antiferromagnets with _J_ = −1, but first examine ferromagnets whose domains are easier to understand.)
  4. The thermal energy _k_ B _T_ is in units of Joule and is the independent variable. Use _k_ B _T_ = 1 for debugging.
  5. Use periodic boundary conditions on your chain to minimize end effects. This means that the chain is a circle with the first and last spins adjacent to each other.
  6. Try _N_   20 for debugging, and larger values for production runs.
  7. Use the printout to check that the system equilibrates for
    1. a) a totally ordered initial configuration (cold start); your simulation should resemble Figure 17.2.
    2. b) a random initial configuration (hot start).

**Listing 17.1** **IsingViz.py** implements the Metropolis algorithm for a 1D Ising chain.

### 17.4.2 Equilibration, Thermodynamic Properties (Assessment)

  1. Watch a chain of _N_ atoms attain thermal equilibrium when in contact with a heat bath. At high temperatures, or for small numbers of atoms, you should see large fluctuations, while at lower temperatures you should see smaller fluctuations.
  2. Look for evidence of instabilities in which there is a spontaneous flipping of a large number of spins. This becomes more likely for larger _k_ B _T_ values.
  3. Note how at thermal equilibrium the system is still quite dynamic, with spins flipping all the time. It is this energy exchange that determines the thermodynamic properties.
  4. You may well find that simulations at small _k_ B _T_ (say, _k_ B _T_   0.1 for _N_ = 200) are slow to equilibrate. Higher _k_ B _T_ values equilibrate faster yet have larger fluctuations.
  5. Observe the formation of domains and the effect they have on the total energy. Regardless of the direction of spin within a domain, the atom–atom interactions are attractive and so contribute negative amounts to the energy of the system when aligned. However, the ↑↓ or ↓↑ If interactions between domains contribute positive energy. Therefore, you should expect a more negative energy at lower temperatures where there are larger and fewer domains.
  6. Make a graph of average domain size vs. temperature.

**Thermodynamic Properties** For a given spin configuration _α_ _j_ , the energy and magnetization are given by

(17.14)

The internal energy _U_ ( _T_ ) is just the average value of the energy,

(17.15)

where the average is taken over a system in equilibrium. At high temperatures, we expect a random assortment of spins and so a vanishing magnetization. At low temperatures when all the spins are aligned, we expect _M_ to approach _N_ /2. Although the specific heat can be computed from the elementary definition,

(17.16)

the numerical differentiation may be inaccurate because _U_ has statistical fluctuations. A better way to calculate the specific heat is to first calculate the fluctuations in energy occurring during _M_ trials and then determine the specific heat from the fluctuations:

(17.17)

(17.18)

  1. Extend your program to calculate the internal energy _U_ and the magnetization _M_ for the chain. Do not recalculate entire sums when only one spin changes.
  2. Make sure to wait for your system to equilibrate before you calculate thermodynamic quantities. (You can check that _U_ is fluctuating about its average.) Your results should resemble Figure 17.3.
  3. Reduce statistical fluctuations by running the simulation a number of times with different seeds and taking the average of the results.

**Figure 17.3** Simulation results from a 1D Ising model of 100 spins. (a) Energy and specific heat as functions of temperature; (b) magnetization as a function of temperature.

  4. The simulations you run for small _N_ may be realistic but may not agree with statistical mechanics, which assumes _N_   ∞ (you may assume that _N_   2000 is close to infinity). Check that agreement with the analytic results for the thermodynamic limit is better for large _N_ than small _N_.
  5. Check that the simulated thermodynamic quantities are independent of initial conditions (within statistical uncertainties). In practice, your cold and hot start results should agree.
  6. Make a plot of the internal energy _U_ as a function of _k_ B _T_ and compare it to the analytic result (17.7).
  7. Make a plot of the magnetization _M_ as a function of _k_ B _T_ and compare it to the analytic result. Does this agree with how you expect a heated magnet to behave?
  8. Compute the energy fluctuations _U_ 2 (17.17) and the specific heat C (17.18). Compare the simulated specific heat to the analytic result (17.8).

### 17.4.3 Beyond Nearest Neighbors, 1D (Exploration)

  * Extend the model so that the spin–spin interaction (17.3) extends to next-nearest neighbors as well as nearest neighbors. For the ferromagnetic case this should lead to more binding and less fluctuation because we have increased the couplings among spins, and thus increased the thermal inertia.
  * Extend the model so that the ferromagnetic spin–spin interaction (17.3) extends to nearest neighbors in two dimensions, and for the truly ambitious, three dimensions. Continue using periodic boundary conditions and keep the number of particles small, at least to start with (Gould _et al._ , 2006).

  1. Form a square lattice and place   spins on each side.
  2. Examine the mean energy and magnetization as the system equilibrates.

**Figure 17.4** The energy, specific heat, and magnetization as a function of temperature from a 2D Ising model simulation with 40 000 spins. Evidence of a phase transition at the Curie temperature _kT_ =  2.5 is seen in all three functions. The values of _C_ and _E_ have been scaled to fit on the same plot as _M_ (courtesy of J.Wetzel).

  3. Is the temperature dependence of the average energy qualitatively different from that of the 1D model?
  4. Make a print out of the spin configuration for small _N_ , and identify domains.
  5. Once your system appears to be behaving properly, calculate the heat capacity and magnetization of the 2D Ising model with the same technique used for the 1D model. Use a total number of particles of 100 ≤ _N_ ≤ 2000.
  6. Look for a phase transition from ordered to unordered configurations by examining the heat capacity and magnetization as functions of temperature. The former should diverge, while the latter should vanish at the phase transition (Figure 17.4).

**Exercise** Three fixed spin-1/2 particles interact with each other at temperature _T_ = 1/ _k_ b such that the energy of the system is

(17.19)

The system starts in the configuration ↑↓↑. Do a simulation by hand that uses the Metropolis algorithm and the series of random numbers 0.5, 0.1, 0.9, 0.3 to determine the results of just two thermal fluctuations of these three spins.

## 17.5 Magnets via Wang–Landau Sampling

We have used a Boltzmann distribution to simulate the thermal properties of an Ising model. We described the probabilities for explicit spin _states α_ with energy _E α_ for a system at temperature _T_ , and summed over various configurations. An equivalent formulation describes the probability that the system will have the explicit _energy E_ at temperature _T_ :

(17.20)

Here _k_ B is Boltzmann's constant, _T_ is the temperature, _g_ ( _E_ i) is the number of states of energy _E_ i ( _i_ = 1,..., _M_ ), _Z_ ( _T_ ) is the partition function, and the sum is still over all _M_ states of the system, but nowwith states of the same energy entering just once owing to _g_ ( _E_ i) accounting for their degeneracy. Because we again apply the theory to the Ising model with its discrete spin states, the energy assumes only discrete values. If the physical system had an energy that varied continuously, then the number of states in the interval _E_ → _E_ \+ d _E_ would be given by _g_ ( _E_ ) d _E_ and _g_ ( _E_ ) would be called the _density of states_. As a matter of convenience, we call _g_ ( _E_ i) the density of states even when dealing with discrete systems, although the term "degeneracy factor" may be more precise.

Even as the Metropolis algorithm has been providing excellent service for more than 60 years, recent literature shows increasing use of Wang–Landau sampling (WLS)2) (Landau and Wang, 2001; STP, 2011). Because WLS determines the density of states and the associated partition function, it is not a direct substitute for the Metropolis algorithm and its simulation of thermal fluctuations. However, we will see that WLS provides an equivalent simulation for the Ising model.

The advantages of WLS is that it requires much shorter simulation times than the Metropolis algorithm and provides a direct determination of _g_ ( _E_ i). For these reasons, it has shown itself to be particularly useful for first-order phase transitions where systems spend long times trapped in metastable states, as well as in areas as diverse as spin systems, fluids, liquid crystals, polymers, and proteins. The time required for a simulation becomes crucial when large systems are modeled. Even a spin lattice as small as 8 × 8 has 264   1.84 × 1019 configurations, which makes visiting them all too expensive.

In our discussion of the Ising model, we ignored the partition function when applying the Metropolis algorithm. Now we focus on the partition function _Z_ ( _T_ ) and the density-of-states function _g_ ( _E_ ). Because _g_ ( _E_ ) is a function of energy but not temperature, once it has been deduced, _Z(T)_ and all thermodynamic quantities can be calculated from it without having to repeat the simulation for each temperature. For example, the internal energy and the entropy are calculated directly as

(17.21)

(17.22)

The density of states _g_ ( _E_ i) will be determined by taking the equivalent of a random walk in _energy_ space. We flip a randomly chosen spin, record the energy of the new configuration, and then keep flipping more spins to change the energy. The table _H_ ( _E_ i) of the number of times each energy _E_ i is attained is called the energy _histogram_ (Figure 17.5b). If the walk were continued for a very long time, the histogram _H_ ( _E_ i) would converge to the density of states _g_ ( _E_ i). Yet with 1019−1030 steps required even for small systems, this direct approach is unrealistically inefficient because the walk would rarely ever get away from the most probable energies.

**Figure 17.5** Wang–Landau sampling used in the Ising model 2D Ising model on an 8 × 8 lattice. (a) Logarithm of the density of states log _g_ ( _E_ ) ∝ _S_ vs. the energy per particle. (b) The histogram _H_ ( _E_ ) showing the number of states visited as a function of the energy per particle. The aim of WLS is to make this function flat.

Clever idea number 1 behind the Wang–Landau algorithm is to explore more of the energy space by increasing the likelihood of walking into less probable configurations. This is carried out by increasing the acceptance of less likely configurations while simultaneously decreasing the acceptance of more likely ones. In other words, we want to accept more states for which the density _g_ ( _E_ i) is small and reject more states for which _g_ ( _E_ i) is large (fret not these words, the equations are simple). To accomplish this trick, we accept a new energy _E_ i with a probability inversely proportional to the (initially unknown) density of states,

(17.23)

and then build up a histogram of visited states via a random walk.

The problem with clever idea number 1 is that _g_ ( _E_ i) is unknown. WLS's clever idea 2 is to determine the unknown _g_ ( _E_ i) simultaneously with the construction of the random walk. This is accomplished by improving the value of _g_ ( _E_ i) via the multiplication _g_ ( _E_ i) → _f g_ ( _E_ i), where _f_ > 1 is an empirical factor. When this works, the resulting histogram _H_ ( _E_ i) becomes "flatter" because making the small _g_ ( _E_ i) values larger makes it more likely to reach states with small _g_ ( _E_ i) values. As the histogram gets flatter, we keep decreasing the multiplicative factor _f_ until it is satisfactory close to 1. At that point, we have a flat histogram and a determination of _g_ ( _E_ i).

At this point you may be asking yourself, "Why does a flat histogram mean that we have determined _g_ ( _E_ i)?" Flat means that all energies are visited equally, in contrast to the peaked histogram that would be obtained normally without the 1/ _g_ ( _E_ i) weighting factor. Thus, if by including this weighting factor we produce a flat histogram, then we have perfectly counteracted the actual peaking in _g_ ( _E_ i), which means that we have arrived at the correct _g_ ( _E_ i).

## 17.6 Wang–Landau Algorithm

The steps in WLS are similar to those in the Metropolis algorithm, but now with use of the density-of-states function _g_ ( _E_ i) rather than a Boltzmann factor:

  1. Start with an arbitrary spin configuration   and with arbitrary values for the density of states _g_ ( _E_ i) = 1, _i_ = 1,... , _M_ , where _M_ = 2 _N_ is the number of states of the system.
  2. Generate a trial configuration _α_ _k_ +1 by
    1. a) picking a particle _i_ randomly and
    2. b) flipping _i_ 's spin.
  3. Calculate the energy _E_ _α_ tr of the trial configuration.
  4. If _g_ ( _E_ _α_ tr) ≤ _g_ ( _E_ _α_ _k_ ), accept the trial, that is, set _α_ _k_ +1 = _α_ tr.
  5. If _g_ ( _E_ _α_ tr) > _g_ ( _E_ _α_ _k_ ), accept the trial with probability _P_ = _g_ ( _E_ _α_ _k_ )/( _g_ ( _E_ _α_ tr):
    1. a) Choose a uniform random number 0 ≤ _r_ _i_ ≤ 1.
    2. b)

This acceptance rule can be expressed succinctly as

(17.24)

which manifestly always accepts low-density (improbable) states.

  6. One we have a new state, we modify the current density of states _g_ ( _E_ i) via the multiplicative factor _f_ :

(17.25)

and add 1 to the bin in the histogram corresponding to the new energy:

(17.26)

  7. The value of the multiplier _f_ is empirical. We start with Euler's number _f_ = e = 2.718 28, which appears to strike a good balance between very large numbers of small steps (small _f_ ) and too rapid a set of jumps through energy space (large _f_ ). Because the entropy _S_ = _k_ B ln _g_ ( _E_ i) → _k_ B[ln _g_ ( _E_ i) + ln _f_ ], (17.25) corresponds to a uniform increase by _k_ B in entropy.
  8. Even with reasonable values for _f_ , the repeated multiplications in (17.25) lead to exponential growth in the magnitude of _g_. This may cause floating-point overflows and a concordant loss of information (in the end, the magnitude of _g_ ( _E_ i) does not matter because the function is normalized). These overflows are avoided by working with logarithms of the function values, in which case the update of the density of states (17.25) becomes

(17.27)

  9. The difficulty with storing ln _g_ ( _E_ i) is that we need the ratio of _g_ ( _E_ i) values to calculate the probability in (17.24). This is circumvented by employing the identity _x_ ≡ exp(ln _x_ ) to express the ratio as

(17.28)

In turn, _g_ ( _E_ _k_ ) = _f_ × _g_ ( _E_ _k_ ) is modified to ln _g_ ( _E_ _k_ ) → ln _g_ ( _E_ _k_ ) + ln _f_.

  10. The random walk in _E_ i continues until a flat histogram of visited energy values is obtained. The flatness of the histogram is tested regularly (every 10 000 iterations), and the walk is terminated once the histogram is sufficiently flat. The value of _f_ is then reduced so that the next walk provides a better approximation to _g_ ( _E_ i). Flatness is measured by comparing the variance in _H_ ( _E_ i) to its average. Although 90–95% flatness can be achieved for small problems like ours, we demand only 80% (Figure 17.5):

(17.29)

  11. Keep the generated _g_ ( _E_ i) and reset the histogram values _h_ ( _E_ i) to zero.
  12. The walks are terminated and new ones initiated until no significant correction to the density of states is obtained. This is measured by requiring the multiplicative factor _f_   1 within some level of tolerance, for example, _f_ ≤ 1 + 10−8. If the algorithm is successful, the histogram should be flat (Figure 17.5) within the bounds set by (17.29).
  13. The final step in the simulation is normalization of the deduced density of states _g_ ( _E_ i). For the Ising model with _N_ up or down spins, a normalization condition follows from knowledge of the total number of states (STP, 2011):

(17.30)

Because the sum in (17.30) is most affected by those values of energy where _g_ ( _E_ i) is large, it may not be precise for the low- _E_ i densities that contribute little to the sum. Accordingly, a more precise normalization, at least if your simulation has performed a good job in occupying all energy states, is to require that there are just two grounds states with energies _E_ = −2 _N_ (one with all spins up and one with all spins down):

(17.31)

In either case, it is good practice to normalize _g_ ( _E_ i) with one condition and then use the other as a check.

### 17.6.1 WLS Ising Model Implementation

We assume an Ising model with spin–spin interactions between nearest neighbors located in an _L_ × _L_ lattice (Figure 17.6). To keep the notation simple, we set _J_ = 1 so that

(17.32)

where ↔ indicates nearest neighbors. Rather than recalculate the energy each time a spin is flipped, only the difference in energy is computed. For example, for eight spins in a 1D array,

(17.33)

where the 0–7 interaction arises because we assume periodic boundary conditions. If spin 5 is flipped, the new energy is

(17.34)

and the difference in energy is

(17.35)

This is cheaper to compute than calculating and then subtracting two energies.

When we advance to two dimensions with the 8 × 8 lattice in Figure 17.6, the change in energy when a spin _σ i, j_ at site ( _i_ , _j_ ) flips is

(17.36)

which can assume the values −8, −4, 0, 4, and 8. There are two states of minimum energy _E_ = −2 _N_ for a 2D system with _N_ spins, and they are ones with all spins pointing in the same direction (either up or down). The maximum energy is 2 _N_ , and it corresponds to alternating spin directions on neighboring sites. Each spin flip on the lattice changes the energy by four units between these limits, and so the values of the energies are

(17.37)

These energies can be stored in a uniform 1D array via the simple mapping

(17.38)

Listing 17.2 displays our implementation of Wang–Landau sampling to calculate the density of states and internal energy _U_ ( _T_ ) (17.21). We used it to obtain the entropy _S_ ( _T_ ) and the energy histogram _H_ ( _E_ i) illustrated in Figure 17.5. Other thermodynamic functions can be obtained by replacing the _E_ in (17.21) with the appropriate variable. The results look like those in Figure 17.4.

**Figure 17.6** The numbering scheme used in our WLS implementation of the 2D Ising model with an 8 × 8 lattice of spins.

A problem that may be encountered when calculating these variables is that the sums in (17.21) can become large enough to cause overflows, although the ratio would not. You work around that by factoring out a common large factor, for example,

(17.39)

where _λ_ is the largest value of ln _g_ ( _E_ i) − _E_ i/ _k_ B _T_ at each temperature. The factor e _λ_ does not actually need to be included in the calculation of the variable because it is present in both the numerator and denominator and so eventually cancels out.

**Listing 17.2** **WangLandau.py** simulates the 2D Ising model using Wang–Landau sampling to compute the density of states and from that the internal energy.

### 17.6.2 WLS Ising Model Assessment

Repeat the assessment conducted in Section 17.4.2 for the thermodynamic properties of the Ising model using WLS in place of the Metropolis algorithm.

## 17.7 Feynman Path Integral Quantum Mechanics

**Problem** As is well known, a classical particle attached to a linear spring undergoes simple harmonic motion with a position as a function of time given by _x_ ( _t_ ) = _A_ sin( _ω_ 0 _t_ \+ _ϕ_ ). Your _problem_ is to take this classical space–time trajectory _x_ ( _t_ ) and use it to generate the quantum wave function _ψ_ ( _x_ , _t_ ) for a particle bound in a harmonic oscillator potential.

## 17.8 Feynman's Space–Time Propagation (Theory)

Feynman was looking for a formulation of quantum mechanics that gave a more direct connection to classical mechanics than does Schrödinger theory, and that made the statistical nature of quantum mechanics evident from the start. He followed a suggestion by Dirac that Hamilton's principle of least action, which can be used to derive classical dynamics, may also be the _ _→ 0 limit of a quantum least-action principle. Seeing that Hamilton's principle deals with the paths of particles through space–time, Feynman postulated that the quantum-mechanical wave function describing the propagation of a free particle from the space–time point _a_ = ( _x_ _a_ , _t_ _a_ ) to the point _b_ = ( _x_ _b_ , _t_ _b_ ) can expressed as (Feynman and Hibbs, 1965; Mannheim, 1983)

(17.40)

where _G_ is _Green's function or propagator_

(17.41)

Equation 17.40 is a form of Huygens's wavelet principle in which each point on the wavefront _ψ_ ( _x_ _a_ , _t_ _a_ ) emits a spherical wavelet _G_ ( _b_ ; _a_ ) that propagates forward in space and time. It states that the new wavefront _ψ_ ( _x_ _b_ , _t_ _b_ ) is created by summation over and interference with all the other wavelets.

Feynman imagined that another way of interpreting (17.40) is as a form of Hamilton's principle in which the probability amplitude, that is the wave function _ψ_ , for a particle to be at _B_ equals the sum over all _paths_ through space–time originating at time _A_ and ending at _B_ (Figure 17.7). This view incorporates the statistical nature of quantum mechanics by having different probabilities for travel along the different paths. All paths are possible, but some are more likely than others. (When you realize that Schrödinger theory solves for wave functions and considers paths a classical concept, you can appreciate how different it is from Feynman's view.) The values for the probabilities of the paths derive from _Hamilton's classical principle of least action:_

> _The most general motion of a physical particle moving along the classical trajectory  from time _t_ _a_ to _t_ _b_ is along a path such that the action   is an extremum:_
>
> (17.42)
>
> _with the paths constrained to pass through the endpoints:_
>
>

**Figure 17.7** In the Feynman path-integral formulation of quantum mechanics a collection of paths connect the initial space–time point _A_ to the final point _B_. The solid line is the trajectory followed by a classical particle, while the dashed lines are additional paths sampled by a quantum particle. A classical particle somehow "knows" ahead of time that travel along the classical trajectory minimizes the action _S_.

This formulation of classical mechanics, which is based on the calculus of variations, is equivalent to Newton's differential equations if the action _S_ is taken as the line integral of the Lagrangian along the path:

(17.43)

Here _T_ is the kinetic energy, _V_ is the potential energy,  , and square brackets indicate a _functional_3) of the function _x_ ( _t_ ) and  .

Feynman observed that the classical action for a free ( _V_ = 0) particle,

(17.44)

is related to the free-particle propagator (17.41) by

(17.45)

**Figure 17.8** (a) The probability distribution for the harmonic oscillator ground state as determined with a path-integral calculation (the classical result has maxima at the two turning points). (b) A space–time quantum path resulting from applying the Metropolis algorithm.

This is the much sought-after connection between quantum mechanics and Hamilton's principle. Feynman went on to postulate a reformulation of quantum mechanics that incorporated its statistical aspects by expressing _G_ ( _b_ , _a_ ) as the weighted sum over all _paths_ connecting _a_ to _b_ ,

(17.46)

Here the classical action _S_ (17.43) is evaluated along different paths (Figure 17.7), and the exponential of the action is summed over paths. The sum (17.46) is called _a path integral_ because it sums over actions _S_ [ _b_ , _a_ ], each of which is an integral (on the computer an integral and a sum are the same anyway).

The essential connection between classical and quantum mechanics is the realization that in units of _ _  10−34 J s, the action is a very large number, _S_ / _ _≥ 10 20, and so even though all paths enter into the sum (17.46), the main contributions come from those paths adjacent to the classical trajectory  . In fact, because _S_ is an extremum for the classical trajectory, it is a constant to first order in the variation of paths, and so nearby paths have phases that vary smoothly and relatively slowly. In contrast, paths far from the classical trajectory are weighted by a rapidly oscillating exp(i _S_ / _ _), and when many are included they tend to cancel each other out. In the classical limit, _ _→ 0, only the single classical trajectory contributes and(17.46) becomes Hamilton's principle of least action. In Figure 17.8, we show an example of an actual trajectory used in path-integral calculations.

### 17.8.1 Bound-State Wave Function (Theory)

Although you may be thinking that you have already seen enough expressions for Green's function, there is yet another one we need for our computation. Let us assume that the Hamiltonian operator   supports a spectrum of eigenfunctions,

(17.47)

each labeled by the index _n_. Because   is Hermitian, its wave functions form a complete orthonormal set in which we may expand a general solution:

(17.48)

where the value for the expansion coefficients _c_ _n_ follows from the orthonormality of the _ψ_ _n_ 's. If we substitute this _c_ _n_ back into the wave function expansion (17.48), we obtain the identity

(17.49)

Comparison with (17.40) yields the eigenfunction expansion for _G_ :

(17.50)

We relate this to the bound-state wave function (recall that our _problem_ is to calculate that) by (1) requiring all paths to start and end at the space position _x_ 0 = _x_ , (2) by taking _t_ 0 = 0, and (3) by making an analytic continuation of (17.50) to negative imaginary time (permissable for analytic functions):

(17.51)

(17.52)

The limit here corresponds to long imaginary times _τ_ , after which the parts of _ψ_ with higher energies decay more quickly, leaving only the ground state _ψ_ 0.

Equation 17.52 provides a closed-form solution for the ground-state wave function directly in terms of the propagator _G_. Although we will soon describe how to compute this equation, look now at Figure 17.8 showing some results of a computation. Although we start with a probability distribution that peaks near the classical turning points at the edges of the well, after a large number of iterations we end up with a distribution that resembles the expected Gaussian, which indicates that the formulation appears to be working. On the right, we see a trajectory that has been generated via statistical variations about the classical trajectory _x_ ( _t_ ) = _A_ sin( _ω_ 0 _t_ \+ _φ_ ).

### 17.8.2 Lattice Path Integration (Algorithm)

Because both time and space need to be integrated over when evaluating a path integral, our simulation starts with a lattice of discrete space–time points. We visualize a particle's trajectory as a series of straight lines connecting one time to the next (Figure 17.9). We divide the time between points A and B into _N_ equal steps of size _ε_ , and label them with the index _j_ :

**Figure 17.9** (a) A path through a space–time lattice that starts and ends at _x_ = _x_ _a_ = _x_ _b_. The action is an integral over this path, while the _path integral_ is a sum of integrals over all paths. The dotted path BD is a transposed replica of path AC. (b) The dashed path joins the initial and final times in two equal time steps; the solid curve uses _N_ steps each of size _ε_. The position of the curve at time _t_ _j_ defines the position _x_ _j_.

(17.53)

Although it is more precise to use the actual positions _x_ ( _t_ _j_ ) of the trajectory at the times _t_ _j_ to determine the _x_ _j_ 's (as in Figure 17.9), in practice we discretize space uniformly and have the links end at the nearest regular points. Once we have a lattice, it is easy to evaluate derivatives or integrals on a link4):

(17.54)

(17.55)

where we have assumed that the Lagrangian is constant over each link.

Lattice path integration is based on the _composition theorem_ for propagators:

(17.56)

For a free particle, this yields

(17.57)

where we have added the actions because line integrals combine as _S_ [ _b_ , _j_ ] + _S_ [ _j_ , _a_ ] = _S_ [ _b_ , _a_ ]. For the _N_ -linked path in Figure 17.9, (17.56) becomes

(17.58)

where _S_ _j_ is the value of the action for link _j_. At this point the integral over the _single_ path shown in Figure 17.9 has become an _N_ -term sum that becomes an infinite sum as the time step _ε_ approaches zero.

To summarize, Feynman's path-integral postulate (17.46) means that we sum over all paths connecting A to B to obtain the Green's function _G_ ( _b_ , _a_ ). This means that we must sum not only over the links in one path but _also_ over all the different paths in order to produce the variation in paths required by Hamilton's principle. The sum is constrained such that paths must pass through A and B and cannot double back on themselves (causality requires that particles move only forward in time). This is the essence of _path integration_. Because we are integrating over functions as well as along paths, the technique is also known as _functional integration_.

The propagator (17.46) is the sum over all paths connecting A to B, with each path weighted by the exponential of the action along that path, explicitly:

(17.59)

(17.60)

where   is the average value of the Lagrangian on link _j_ at time _t_ = _jε_. The computation is made simpler by assuming that the potential _V_ ( _x_ ) is independent of velocity and does not depend on other _x_ values (local potential). Next we observe that _G_ is evaluated with a negative imaginary time in expression (17.52) for the ground-state wave function. Accordingly, we evaluate the Lagrangian with _t_ = −i _τ_ :

(17.61)

(17.62)

We see that the reversal of the sign of the kinetic energy in _L_ means that _L_ now equals the negative of the Hamiltonian evaluated at a real positive time _t_ = _τ_ :

(17.63)

(17.64)

In this way, we rewrite the _t_ -path integral of _L_ as a _τ_ -path integral of _H_ , and so express the action and Green's function in terms of the Hamiltonian:

(17.65)

(17.66)

where the line integral of _H_ is over an entire trajectory. Next we express the path integral in terms of the average energy of the particle on each link, _E_ _j_ = _T_ _j_ \+ _V_ _j_ , and then sum over links to obtain the summed energy _ _5) :

(17.67)

(17.68)

In (17.68), we have approximated each path link as a _straight line_ , used Euler's derivative rule to obtain the velocity, and evaluated the potential at the midpoint of each link. We now substitute this _G_ into our solution (17.52) for the ground-state wave function in which the initial and final points in space are the same:

(17.69)

(17.70)

The similarity of these expressions to thermodynamics, even with a partition function _Z_ , is no accident; by making the time parameter of quantum mechanics imaginary, we have converted the time-dependent Schrödinger equation to the heat diffusion equation

(17.71)

It is not surprising then that the sum over paths in Green's function has each path weighted by the Boltzmann factor _P_ = e− _ε _ usually associated with thermodynamics. We make the connection complete by identifying the temperature with the inverse time step:

(17.72)

Consequently, the _ε_ → 0 limit, which makes time continuous, is a "high-temperature" limit. The _τ_ → ∞ limit, which is required to project the ground-state wave function, means that we must integrate over a path that is long in imaginary time, that is, long compared to a typical time _ _/ _ΔE_. Just as our simulation of the Ising model required us to wait a long time for the system to equilibrate, so the present simulation requires us to wait a long time so that all but the ground-state wave function has decayed. Alas, this is the solution to our _problem_ of finding the ground-state wave function.

To summarize, we have expressed Green's function as a path integral that requires integration of the Hamiltonian along paths and a summation over all the paths (17.69). We evaluate this path integral as the sum over all the trajectories in a space–time lattice. Each trial path occurs with a probability based on its action, and we use the Metropolis algorithm to include statistical fluctuation in the links, as if they are in thermal equilibrium. This is similar to our work with the Ising model, however now, rather than reject or accept a _flip in spin_ based on the change in energy, we reject or accept a _change in a link_ based on the change in energy. The more iterations we let the algorithm run for, the more time the deduced wave function has to equilibrate to the ground state.

In general, Monte Carlo Green's function techniques work best if we start with a good guess at the correct answer, and then have the algorithm calculate variations on our guess. For the present problem this means that if we start with a path in space–time close to the classical trajectory, the algorithm may be expected to do a good job at simulating the quantum fluctuations about the classical trajectory. However, it does not appear to be good at finding the classical trajectory from arbitrary locations in space–time. We suspect that the latter arises from _δS_ / _ _being so large that the weighting factor exp( _δS/ _ ) fluctuates wildly (essentially averaging out to zero) and so loses its sensitivity.

#### 17.8.2.1 A Time-Saving Trick

As we have formulated the computation, we pick a value of _x_ and perform an expensive computation of line integrals over all space and time to obtain | _ψ_ 0( _x_ )|2 at one _x_. To obtain the wave function at another _x_ , the entire simulation must be repeated from scratch. Rather than go through all that trouble again and again, we will compute the entire _x_ dependence of the wave function in one fell swoop. The trick is to insert a delta function into the probability integral (17.69), thereby fixing the initial position to be _x_ 0, and then to integrate over all values for _x_ 0:

(17.73)

(17.74)

This equation expresses the wave function as an average of a delta function over all paths, a procedure that might appear totally inappropriate for numerical computation because there is tremendous error in representing a singular function on a finite-word-length computer. Yet when we simulate the sum over all paths with (17.74), there will always be some _x_ value for which the integral is nonzero, and we need to accumulate only the solution for various (discrete) _x_ values to determine | _ψ_ 0( _x_ )|2 for all _x_.

To understand how this works in practice, consider path AB in Figure 17.9 for which we have just calculated the summed energy. We form a new path by having one point on the chain jump to point C (which changes two links). If we replicate section AC and use it as the extension AD to form the top path, we see that the path CBD has the same summed energy (action) as path ACB, and in this way can be used to determine | _ψ_ ( _x_ ′ _j_ )|2. That being the case, once the system is equilibrated, we determine new values of the wave function at new locations _x_ ′ _j_ by flipping links to new values and calculating new actions. The more frequently some _x_ _j_ is accepted, the greater the wave function at that point.

### 17.8.3 Lattice Implementation

The program QMC.py in Listing 17.3 evaluates the integral (17.46) by finding the average of the integrand _δ_ ( _x_ 0 − _x_ ) with paths distributed according to the weighting function exp[− _ε _ ( _x_ 0, _x_ 1,... , _x_ _N_ )]. The physics enters via (17.76), the calculation of the summed energy _ _( _x_ 0, _x_ 1,... , _x_ _N_ ). We evaluate the action integral for the harmonic oscillator potential

(17.75)

and for a particle of mass _m_ = 1. Using a convenient set of natural units, we measure lengths in   and times in 1/ _ω_ = 1. Correspondingly, the oscillator has a period _T_ = 2π. Figure 17.8 shows results from an application of the Metropolis algorithm. In this computation, we started with an initial path close to the classical trajectory and then examined a half million variations about this path. All paths are constrained to begin and end at _x_ = 1 (which turns out to be somewhat less than the maximum amplitude of the classical oscillation). When the time difference _t b_ − _t a_ equals a short time like _2T_ , the system has not had enough time to equilibrate to its ground state and the wave function looks like the probability distribution of an excited state (nearly classical with the probability highest for the particle to be near its turning points where its velocity vanishes). However, when _t b_ − _t a_ equals the longer time 20 _T_ , the system has had enough time to decay to its ground state and the wave function looks like the expected Gaussian distribution. In either case (Figure 17.8a), the trajectory through space–time fluctuates about the classical trajectory. This fluctuation is a consequence of the Metropolis algorithm occasionally going uphill in its search; if you modify the program so that searches go only downhill, the space–time trajectory will be a very smooth trigonometric function (the classical trajectory), but the wave function, which is a measure of the fluctuations about the classical trajectory, will vanish! The explicit steps of the calculation are (MacKeown, 1985; MacKeown and Newman, 1987):

  1. Construct a grid of _N_ time steps of length _ε_ (Figure 17.9). Start at _t_ = 0 and extend to time _τ_ = _Nε_ [this means _N_ time intervals and ( _N_ \+ 1) lattice points in time]. Note that time always increases monotonically along a path.
  2. Construct a grid of _M_ space points separated by steps of size _δ_. Use a range of _x_ values several time larger than the characteristic size or range of the potential being used and start with _M_   _N_.
  3. When calculating the wave function, any _x_ or _t_ value falling between lattice points should be assigned to the closest lattice point.
  4. Associate a position _x_ _j_ with each time _τ_ _j_ , subject to the boundary conditions that the initial and final positions always remain the same, _x_ _N_ = _x_ 0 = _x_.
  5. Choose a path of straight-line links connecting the lattice points corresponding to the classical trajectory. Observe that the _x_ values for the links of the path may have values that increase, decrease, or remain unchanged (in contrast to time, which always increases).
  6. Evaluate the energy _ _by summing the kinetic and potential energies for each link of the path starting at _j_ = 0:

(17.76)

  7. Begin a sequence of repetitive steps in which a random position _x_ _j_ associated with time _t_ _j_ is changed to the position _x_ ′ _j_ (point _C_ in Figure 17.9). This changes _two_ links in the path.
  8. For the coordinate that is changed, use the Metropolis algorithm to weigh the change with the Boltzmann factor.
  9. For each lattice point, establish a running sum that represents the value of the wave function squared at that point.
  10. After each single-link change (or decision not to change), increase the running sum for the new _x_ value by 1. After a sufficiently long running time, the sum divided by the number of steps is the simulated value for | _ψ_ ( _x_ _j_ )|2 at each lattice point _x_ _j_.
  11. Repeat the entire link-changing simulation starting with a different seed. The average wave function from a number of intermediate-length runs is better than that from one very long run.

**Listing 17.3** **QMC.py** determines the ground-state probability via a Feynman path integration using the Metropolis algorithm to simulate variations about the classical trajectory

### 17.8.4 Assessment and Exploration

  1. Plot some of the actual space–time paths used in the simulation along with the classical trajectory.
  2. For a more continuous picture of the wave function, make the _x_ lattice spacing smaller; for a more precise value of the wave function at any particular lattice site, sample more points (run longer) and use a smaller time step _ε_.
  3. Because there are no sign changes in a ground-state wave function, you can ignore the phase, assume  , and then estimate the energy via

(17.77)

where the space derivative is evaluated numerically.

  4. Explore the effect of making _ _larger and thus permitting greater fluctuations around the classical trajectory. Do this by decreasing the value of the exponent in the Boltzmann factor. Determine if this makes the calculation more or less robust in its ability to find the classical trajectory.
  5. Test your _ψ_ for the gravitational potential (see quantum bouncer below):

(17.78)

## 17.9 Exploration: Quantum Bouncer's Paths

Another problem for which the classical trajectory is well known is that of a _quantum bouncer_.6) Here we have a particle dropped in a uniform gravitational field, hitting a hard floor, and then bouncing. When treated quantum mechanically, quantized levels for the particle result (Gibbs, 1975; Goodings and Szeredi, 1992; Whineray, 1992; Banacloche, 1999; Vallée, 2000). In 2002, an experiment to discern this gravitational effect at the quantum level was performed by Nesvizhevsky _et al_. (2002) and described by Shaw (1992). It consisted of dropping ultracold neutrons from a height of 14 μm unto a neutron mirror and watching them bounce. It found a neutron ground state at 1.4 peV.

We start by determining the analytic solution to this problem for stationary states and then generalize it to include time dependence. The time-independent Schrödinger equation for a particle in a uniform gravitation potential is

(17.79)

(17.80)

The boundary condition (17.80) is a consequence of the hard floor at _x_ = 0. A change of variables converts (17.79) to a dimensionless form

(17.81)

(17.82)

This equation has an analytic solution in terms of Airy functions Ai( _z_ ):

(17.83)

where _N_ n is a normalization constant and _z_ E is the scaled value of the energy. The boundary condition _ψ_ (0) = 0 implies that

(17.84)

which means that the allowed energies of the system are discrete and correspond to the zeros _z n_ of the Airy functions (Press _et al_., 1994) at negative argument. To simplify the calculation, we take _ _= 1, _g_ = 2, and _m_ = 1/2, which leads to _z_ = _x_ and _z_ E = _E_.

The time-dependent solution for the quantum bouncer is constructed by forming the infinite sum over all the discrete eigenstates, each with a time dependence appropriate to its energy:

(17.85)

where the _C n_'s are constants.

Figure 17.10 shows the results of solving for the quantum bouncer's ground-state probability | _ψ_ 0( _z_ )|2 using Feynman's path integration. The time increment d _t_ and the total time _t_ were selected by trial and error in such a way as to make

**Figure 17.10** The analytic and quantum Monte Carlo solution for the quantum bouncer. The continuous line is the Airy function squared and the dashed line | _ψ_ 0( _z_ )|2 after a million trajectories.

| _ψ_ (0)|2   0 (the boundary condition). To account for the fact that the potential is infinite for negative _x_ values, we selected trajectories that have positive _x_ values over all their links. This incorporates the fact that the particle can never penetrate the floor. Our program is given in Listing 17.4, and it yields the results in Figure 17.10 after using 106 trajectories and a time step _ε_ = d _τ_ = 0.05. Both results were normalized via a trapezoid integration. As can be seen, the agreement between the analytic result and the path integration is satisfactory.

**Listing 17.4** **QMCbouncer.py** uses Feynman path integration to compute the path of a quantum particle in a gravitational field.

  1) Large-scale, practical computations make a full sweep in which every spin is updated once, and then use this as the new trial configuration. This is found to be efficient and useful in removing some autocorrelations. 2) We thank Oscar A. Restrepo of the Universidad de Antioquia for letting us use some of his material here. 3) A _functional_ is a number whose value depends on the complete behavior of some function and not just on its behavior at one point. For example, the derivative _f_ ′( _x_ ) depends on the value of _f_ at _x_ , yet the integral   depends on the entire function, and is therefore a functional of _f_. 4) Although Euler's rule has a large error, it is often use in lattice calculations because of its simplicity. However, if the Lagrangian contains second derivatives, you should use the more precise central-difference method to avoid singularities. 5) In some cases, such as for an infinite square well, this can cause problems if the trial link causes the energy to be infinite. In that case, one can modify the algorithm to use the potential at the beginning of a link. 6) Oscar A. Restrepo assisted in the preparation of this section.

# 18
Molecular Dynamics Simulations

* * *

You may have seen in introductory chemistry or physics classes that the ideal gas law can be derived from first principles when the interactions of the molecules with each other are ignored, and only reflections off the walls of the surrounding box are considered. We extend that model so that we can solve for the motion of every molecule in a box interacting with every other molecule in the box (but not with the walls). While our example is a simple one, molecular dynamics is of key importance in many fields, including material science and biology.

Your _problem_ is to determine whether a collection of argon molecules placed in a box will form an ordered structure at low temperature.

## 18.1 Molecular Dynamics (Theory)

_Molecular dynamics_ (MD) is a powerful simulation technique for studying the physical and chemical properties of solids, liquids, amorphous materials, and biological molecules. Although we know that quantum mechanics is the proper theory for molecular interactions, MD uses Newton's laws as the basis of the technique and focuses on bulk properties, which do not depend much on small- _r_ behaviors. In 1985, Car and Parrinello showed how MD can be extended to include quantum mechanics by applying density functional theory to calculate the force (Car and Parrinello, 1985). This technique, known as _quantum MD,_ is an active area of research but is beyond the realm of this chapter.1) For those with more interest in the subject, there are full texts (Allan and Tildesley, 1987; Rapaport, 1995; Hockney, 1988) on MD and good discussions (Gould _et al.,_ 2006; Thijssen, 1999; Fosdick _et al_., 1996), as well as primers (Ercolessi, 1997) and codes, (Nelson _et al.,_ 1996; Refson, 2000; Anderson _et al.,_ 2008) available online.

MD's solution of Newton's laws is conceptually simple, yet when applied to a very large number of particles becomes the "high school physics problem from hell." Some approximations must be made in order not to have to solve the 1023–1025 equations of motion describing a realistic sample, but instead to limit the problem to ~ 106 particles for protein simulations, and ~ 108 particles for materials simulations. If we have some success, then it is a good bet that the model will improve if we incorporate more particles or more quantum mechanics, something that becomes easier as computing power continues to increase.

In a number of ways, MD simulations are similar to the thermal Monte Carlo simulations we studied in Chapter 17. Both typically involve a large number _N_ of interacting particles that start out in some set configuration and then equilibrate into some dynamic state on the computer. However, in MD we have what statistical mechanics calls a _microcanonical ensemble_ in which the energy _E_ and volume _V_ of the _N_ particles are fixed. We then use Newton's laws to generate the dynamics of the system. In contrast, Monte Carlo simulations do not start with first principles, but instead, incorporate an element of chance and have the system in contact with a heat bath at a fixed temperature rather than keeping the energy _E_ fixed. This is called a _canonical ensemble._

Because a system of molecules in an MD simulation is dynamic, the velocities and positions of the molecules change continuously. After the simulation has run long enough to stabilize, we will compute time averages of the dynamic quantities in order to deduce the thermodynamic properties. We apply Newton's laws with the assumption that the net force on each molecule is the sum of the two-body forces with all other ( _N_ – 1) molecules:

(18.1)

(18.2)

In writing these equations, we have ignored the fact that the force between argon atoms really arises from the particle–particle interactions of the 18 electrons and the nucleus that constitute each atom (Figure 18.1). Although it may be possible to ignore this internal structure when deducing the long-range properties of inert elements, it matters for systems such as polyatomic molecules that display rotational, vibrational, and electronic degrees of freedom as the temperature is raised.2)

The force on molecule _i_ derives from the sum of molecule–molecule potentials:

(18.3)

(18.4)

(18.5)

**Figure 18.1** The molecule–molecule effective interaction arises from the many-body interaction of the electrons and nucleus in one molecule (circle) with the electrons and nucleus in another molecule (other circle). Note, the size of the nucleus at the center of each molecule is highly exaggerated, and real electrons have no size.

Here   is the distance between the centers of molecules _i_ and _j_ , and the limits on the sums are such that no interaction is counted twice. Because we have assumed a _conservative_ potential, the total energy of the system, that is, the potential plus kinetic energies summed over all particles, should be conserved over time. Nonetheless, in a practical computation we "cut the potential off" (assume _u_ ( _r ij_) = 0) when the molecules are far apart. Because the derivative of the potential produces an infinite force at this cutoff point, energy will no longer be precisely conserved. Yet because the cutoff radius is large, the cutoff occurs only when the forces are minuscule, and so the violation of energy conservation should be small relative to approximation and round-off errors.

In a first-principles calculation, the potential between any two argon atoms arises from the sum over approximately 1000 electron–electron and electron–nucleus Coulomb interactions. A more practical calculation would derive an effective potential based on a form of many-body theory, such as Hartree–Fock or density functional theory. Our approach is simpler yet. We use the Lennard–Jones potential,

(18.6)

(18.7)

Here the parameter _ _governs the strength of the interaction, and _σ_ determines the length scale. Both are deduced by fits to data, which is why this potential is called a "phenomenological" potential.

Some typical values for the parameters and scales for the variables are given in Table 18.1. In order to make the program simpler and to avoid under- and overflows, it is helpful to measure all variables in the natural units formed by these constants. The interparticle potential and force then take the forms

(18.8)

**Table 18.1** Parameter and scales for the Lennard-Jones potential.

**Figure 18.2** The Lennard–Jones effective potential used in many MD simulations. Note the sign change at _r_ = 1 and the minimum at _r_   1.1225 (natural units). Also note that because the _r_ -axis does not extend to _r_ = 0, the infinitely high central repulsion is not shown.

The Lennard–Jones potential is seen in Figure 18.2 to be the sum of a long-range attractive interaction ∝ 1/ _r_ 6 and a short-range repulsive one ∝ 1/ _r_ 12. The change from repulsion to attraction occurs at _r_ = _σ_. The minimum of the potential occurs at _r_ = 21/6 _σ_ = 1.1225σ, which would be the atom–atom spacing in a solid bound by this potential. The repulsive 1/ _r_ 12 term in the Lennard–Jones potential (18.6) arises when the electron clouds from two atoms overlap, in which case the Coulomb interaction and the Pauli exclusion principle force the electrons apart. The 1/ _r_ 12 term dominates at short distances and makes atoms behave like hard spheres. The precise value of 12 is not of theoretical significance (although it being large is) and was probably chosen because it is 2 × 6.

The 1/ _r_ 6 term that dominates at large distances models the weak _van der Waals_ induced dipole–dipole attraction between two molecules. The attraction arises from fluctuations in which at some instant in time a molecule on the right tends to be more positive on the left side, like a dipole ⇐. This in turn attracts the negative charge in a molecule on its left, thereby inducing a dipole ⇐ in it. As long as the molecules stay close to each other, the polarities continue to fluctuate in synchronization ⇐⇐ so that the attraction is maintained. The resultant dipole–dipole attraction behaves like 1/ _r_ 6, and although much weaker than a Coulomb force, it is responsible for the binding of neutral, inert elements, such as argon for which the Coulomb force vanishes.

### 18.1.1 Connection to Thermodynamic Variables

We assume that the number of particles is large enough to use statistical mechanics to relate the results of our simulation to the thermodynamic quantities. The simulation is valid for any number of particles, but the use of statistics requires large numbers. The equipartition theorem tells us that, on the average, for molecules in thermal equilibrium at temperature _T,_ each degree of freedom has an energy _k_ B _T_ /2 associated with it, where _k_ B = 1.38 × 10–23 J/K is Boltzmann's constant. A simulation provides the kinetic energy of translation3):

(18.9)

The time average of KE (three degrees of freedom) is related to temperature by

(18.10)

The system's pressure _P_ is determined by a version of the _Virial theorem,_

(18.11)

where the Virial _w_ is a weighted average of the forces. Note that because ideal gases have no intermolecular forces, their Virial vanishes and we have the ideal gas law. The pressure is thus

(18.12)

where _p_ = _N/V_ is the density of the particles.

### 18.1.2 Setting Initial Velocities

Although we start the system off with a velocity distribution characteristic of a definite temperature, this is not a true temperature of the system because the system is not in equilibrium initially, and there will a redistribution of energy between KE and PE (Thijssen, 1999). Note that this initial randomization is the only place where chance enters into our MD simulation, and it is there to speed the simulation along. Indeed, in Figure 18.3 we show results of simulations in which the molecules are initially at rest and equally spaced. Once started, the time evolution is determined by Newton's laws, in contrast to Monte Carlo simulations which are inherently stochastic. We produce a Gaussian (Maxwellian) velocity distribution with the methods discussed in Chapter 4. In our sample code, we take the average   of uniform random numbers   to produce a Gaussian distribution with mean  . We then subtract this mean value to obtain a distribution about 0.

**Figure 18.3** (a) Two frames from an animation showing the results of a 1D MD simulation that starts with uniformly spaced atoms. Note the unequal spacing resulting from an image atom moving in from the left after an atom left from the right. (b) Two frames from the animation of a 2D MD simulation showing the initial and an equilibrated state. Note how the atoms start off in a simple cubic arrangement but then equilibrate to a face-centered-cubic lattice. In both the cases, the atoms remain confined as a result of the interatomic forces.

### 18.1.3 Periodic Boundary Conditions and Potential Cutoff

It is easy to believe that a simulation of 1023 molecules should predict bulk properties well, but with typical MD simulations employing only 103–106 particles, one must be clever to make less seem like more. Furthermore, because computers are finite, the molecules in the simulation are constrained to lie within a finite box, which inevitably introduces artificial _surface effects_ arising from the walls. Surface effects are particularly significant when the number of particles is small because then a large fraction of the molecules reside near the walls. For example, if 1000 particles are arranged in a 10 × 10 × 10 × 10 cube, there are 103 – 83 = 488 particles one unit from the surface, that is, 49% of the molecules, while for 106 particles this fraction falls to 6%.

The imposition of _periodic boundary conditions_ (PBCs) strives to minimize the shortcomings of both the small numbers of particles and of artificial boundaries. Although we limit our simulation to an _L x_ × _L y_ × _L z_ box, we imagine this box being replicated to infinity in all directions (Figure 18.4). Accordingly, after each time-integration step we examine the position of each particle and check if it has left the simulation region. If it has, then we bring an _image_ of the particle back through the opposite boundary (Figure 18.4):

(18.13)

Consequently, each box looks the same and has continuous properties at the edges. As shown by the one-headed arrows in Figure 18.4, if a particle exits the simulation volume, its image enters from the other side, and so balance is maintained.

In principle, a molecule interacts with all others molecules and their images, so despite the fact that there is a finite number of atoms in the interaction volume, there is an effective infinite number of interactions (Ercolessi, 1997). Nonetheless, because the Lennard–Jones potential falls off so rapidly for large _r_ ,   _V_ (1.13σ)/200, far-off molecules do not contribute significantly to the motion of a molecule, and we pick a value   beyond which we ignore the effect of the potential:

**Figure 18.4** The infinite space generated by imposing periodic boundary conditions on the particles within the simulation volume (shaded box). The two-headed arrows indicate how a particle interacts with the nearest version of another particle, be that within the simulation volume or an image. The vertical arrows indicate how the image of particle 4 enters when the actual particle 4 exits.

(18.14)

Accordingly, if the simulation region is large enough for  , an atom interacts with only the _nearest image_ of another atom.

As already indicated, a shortcoming with the cutoff potential (18.14) is that because the derivative d _u_ /d _r_ is singular at _r_ = _r_ cut, the potential is no longer conservative and thus energy conservation is no longer ensured. However, because the forces are already very small at _r_ cut, the violation will also be very small.

## 18.2 Verlet and Velocity–Verlet Algorithms

A realistic MD simulation may require integration of the 3D equations of motion for 1010 time steps for each of 103–106 particles. Although we could use our standard `rk4` ODE solver for this, time is saved by using a simple rule embedded in the program. The Verlet algorithm uses the central-difference approximation (Chapter 5) for the second derivative to advance the solutions by a single time step _h_ for all _N_ particles simultaneously:

(18.15)

(18.16)

where we have set _m_ = 1. (Improved algorithms may vary the time step depending upon the speed of the particle.) Note that although the atom–atom force does not have an explicit time dependence, we include a _t_ dependence in it as a way of indicating its dependence upon the atoms' positions at a particular time. Because this is really an implicit time dependence, energy remains conserved.

Part of the efficiency of the Verlet algorithm (18.16) is that it solves for the position of each particle without requiring a separate solution for the particle's velocity. However, once we have deduced the position for various times, we can use the central-difference approximation for the first derivative of _r i_ to obtain the velocity:

(18.17)

Finally, note that because the Verlet algorithm needs _r_ from two previous steps, it is not self-starting and so we start it with the forward difference

(18.18)

**Velocity–Verlet Algorithm** Another version of the Verlet algorithm, which we recommend because of its increased stability, uses a forward-difference approximation for the derivative to advance _both_ the position and velocity simultaneously:

(18.19)

(18.20)

(18.21)

Although this algorithm appears to be of lower order than (18.16), the use of updated positions when calculating velocities, and the subsequent use of these velocities, make both algorithms of similar precision.

Of interest is that (18.21) approximates the average force during a time step as  . Updating the velocity is a little tricky because we need the force at time _t_ \+ _h_ , which depends on the particle positions at _t_ \+ _h_. Consequently, we must update all the particle positions and forces to _t_ \+ _h_ before we update any velocities, while saving the forces at the earlier time for use in (18.21). As soon as the positions are updated, we impose periodic boundary conditions to establish that we have not lost any particles, and then we calculate the forces.

## 18.3 1D Implementation and Exercise

In the supplementary materials to this book, you will find a number of 2D animations (movies) of solutions to the MD equations. Some frames from these animations are shown in Figure 18.3. We recommend that you look at the movies in order to better visualize what the particles do during an MD simulation. In particular, these simulations use a potential and temperature that should lead to a solid or liquid system, and so you should see the particles binding together.

**Listing 18.1** **MD.py** performs a 2D MD simulation with a small number of rather large time steps for just a few particles. To be realistic the user should change the parameters and the number of random numbers added to form the Gaussian distribution.

The program `MD.py` in Listings 18.1 implements an MD simulation in 1D using the velocity-Verlet algorithm. Use it as a model and do the following:

  1. Establish that you can run and visualize the 1D simulation.
  2. Place the particles initially at the sites of a simple cubic lattice. The equilibrium configuration for a Lennard–Jones system at low temperature is a face-centered-cubic, and if your simulation is running properly, then the particles should migrate from SC to FCC. An FCC lattice has four quarters of a particle per unit cell, so an _L_ 3 box with a lattice constant _L/N_ contains (parts of) 4 _N_ 3 = 32, 108, 256,... particles.
  3. To save computing time, assign initial particle velocities corresponding to a fixed-temperature Maxwellian distribution.
  4. Print the code and indicate on it which integration algorithm is used, where the periodic boundary conditions are imposed, where the nearest image interaction is evaluated, and where the potential is cut off.
  5. A typical time step is  , which in our natural units equals 0.004. You probably will need to make 104–105 such steps to equilibrate, which corresponds to a total time of only 10–9 s (a lot can happen to a speedy molecule in 10–9 s). Choose the _largest_ time step that provides stability and gives results similar to Figure 18.5.
  6. The PE and KE change with time as the system equilibrates. Even after that, there will be fluctuations because this is a dynamic system. Evaluate the time-averaged energies for an equilibrated system.
  7. Compare the final temperature of your system to the initial temperature. Change the initial temperature and look for a simple relation between it and the final temperature (Figure 18.6).

## 18.4 Analysis

  1. Modify your program so that it outputs the coordinates and velocities of a few particles throughout the simulation. Note that you do not need as many time steps to follow a trajectory as you do to compute it, and so you may want to use the _mod_ operator `%100` for output.
    1. a) Start your assessment with a 1D simulation at zero temperature. The particles should remain in place without vibration. Increase the temperature and note how the particles begin to move about and interact.
    2. b) Try starting off all your particles at the minima in the Lennard–Jones potential. The particles should remain bound within the potential until you raise the temperature.
    3. c) Repeat the simulations for a 2D system. The trajectories should resemble billiard ball-like collisions.
    4. d) Create an animation of the time-dependent locations of several particles.
    5. e) Calculate and plot the root-mean-square displacement of molecules as a function of temperature:

(18.22)

where the average is over all the particles in the box. Determine the approximate time dependence of _R_ rms.
    6. f) Test your system for time-reversal invariance. Stop it at a fixed time, reverse all velocities, and see if the system retraces its trajectories back to the initial configuration after this same fixed time.

**Figure 18.5** The kinetic, potential, and total energy as a function of time or number of steps for a 2D MD simulation with 36 particles (a), and 300 particles (b), both with an initial temperature of 150 K. The potential energy is negative, the kinetic energy is positive, and the total energy is seen to be conserved (flat).

  2. _Hand Computation_ We wish to make an MD simulation _by hand_ of the positions of particles 1 and 2 that are in a 1D box of side 8. For an origin located at the center of the box, the particles are initially at rest and at locations  . The particles are subject to the force

**Figure 18.6** (a) The temperature after equilibration as a function of initial kinetic energy for a 2D MD simulation with 36 particles. The two are nearly proportional. (b) The pressure vs. temperature for a simulation with several hundred particles. An ideal gas (noninteracting particles) would yield a straight line (courtesy of J. Wetzel).

**Figure 18.7** A simulation of a projectile shot into a group of particles. The energy introduced by the projectile is seen to lead to evaporation of the particles (courtesy of J. Wetzel).

(18.23)

Use a simple algorithm to determine the positions of the particles up until the time they leave the box. Make sure to apply periodic boundary conditions. _Hint:_ Because the configuration is symmetric, you know the location of particle 2 by symmetry and do not need to solve for it. We suggest the Verlet algorithm (no velocities) with a forward-difference algorithm to initialize it. To speed things along, use a time step of  .
  3. _Diffusion_ It is well known that light molecules diffuse more quickly than heavier ones. See if you can simulate diffusion with your MD simulation using a Lennard–Jones potential and periodic boundary conditions (Satoh, 2011).
    1. a) Generalize the velocity-Verlet algorithm so that it can be used for molecules of different masses.
    2. b) Modify the simulation code so that it can be used for five heavy molecules of mass _M_ = 10 and five light molecules of mass _m_ = 1.
    3. c) Start with the molecules placed randomly near the center of the square simulation region.
    4. d) Assign random initial velocities to the molecules.
    5. e) Run the simulation several times and verify visually that the lighter molecules tend to diffuse more quickly than the heavier ones.
    6. f) For each ensemble of molecules, calculate the RMS velocity at regular instances of time, and then plot the RMS velocities as functions of time. Do the lighter particles have a greater RMS velocity?
  4. As shown in Figure 18.7, simulate the impact of a projectile with a block of material.

1) We thank Satoru S. Kano for pointing this out to us. 2) We thank Saturo Kano for clarifying this point. 3) Unless the temperature is very high, argon atoms, being inert spheres, have no rotational energy.

# 19
PDE Review and Electrostatics via Finite Differences and Electrostatics via Finite Differences

* * *

This chapter is the first of several dealing with partial differential equations (PDEs); several because PDEs are more complex than ODEs, and because each type of PDE requires its own algorithm. We start the chapter with a discussion of PDEs in general, and the requirements for a unique solution of each type to exist. Then we get down to business and examine the simple, but powerful, _finite-differences_ method for solving Poisson's and Laplace's equations on a lattice in space. Chapter 23 covers the more complicated, but ultimately more efficient, _finite elements_ method for solving the same equations.

* * *

## 19.1 PDE Generalities

Physical quantities such as temperature and pressure vary continuously in both space and time. Such being our world, the function or _field_  ( _x, y, z, t_ ) used to describe these quantities must contain independent space and time variations. As time evolves, the changes in  ( _x, y, z, t_ ) at any one position affect the field at neighboring points. This means that the dynamic equations describing the dependence of   on four independent space–time variables must be written in terms of partial derivatives, and therefore the equations must be _partial differential equations_ (PDEs), in contrast to ordinary differential equations (ODEs).

The most general form for a two-independent variable PDE is

(19.1)

where _A, B, C,_ and _F_ are arbitrary functions of the variables _x_ and _y._ In Table 19.1, we define the classes of PDEs by the value of the discriminant _d = AC − B_ 2 in row two (Arfken and Weber, 2001), and give examples in rows three and four.

We usually think of an elliptic equation as containing the second-order derivatives of all the variables, with all having the same sign when placed on the same side of the equal sign; a parabolic equation as containing a first-order derivative in one variable and a second-order derivative in the other; and a hyperbolic equation as containing second-order derivatives of all the variables, with opposite signs when placed on the same side of the equal sign.

**Table 19.1** Three categories of PDE based on the value of their discriminant _d._

Elliptic | Parabolic | Hyperbolic
---|---|---
  |   |
  |   |
Poisson's | Heat | Wave

After solving enough problems, one often develops some physical intuition as to whether one has sufficient _boundary conditions_ for there to exist a unique solution for a given physical situation (this, of course, is in addition to requisite _initial conditions_ ). Table 19.2 gives the requisite boundary conditions for a unique solution to exist for each type of PDE. For instance, a string tied at both ends and a heated bar placed in an infinite heat bath are physical situations for which the boundary conditions are adequate. If the boundary condition is the value of the solution on a surrounding closed surface, we have a _Dirichlet boundary condition._ If the boundary condition is the value of the normal derivative on the surrounding surface, we have a _Neumann boundary condition._ If the value of both the solution and its derivative are specified on a closed boundary, we have a _Cauchy boundary condition._ Although having an adequate boundary condition is necessary for a unique solution, having too many boundary conditions, for instance, both Neumann and Dirichlet, may be an overspecification for which no solution exists.1)

Solving PDEs numerically differs from solving ODEs in a number of ways. First, because we are able to write all ODEs in a standard form

(19.2)

with _t_ the single independent variable, we are able to use a standard algorithm such as `**rk4**` to solve all such equations. Yet because PDEs have several independent variables, for example, _ρ_ ( _x, y, z, t_ ), we would have to apply (19.2) simultaneously and independently to each variable, which would be very complicated. Second, because there are more equations to solve with PDEs than with ODEs, we need more information than just the two _initial conditions_  . In addition, because each PDE often has its own particular set of boundary conditions, we have to develop a special algorithm for each particular problem.

## 19.2 Electrostatic Potentials

Your _problem_ is to find the electric potential for all points _inside_ the charge-free square shown in Figure 19.1. The bottom and sides of the region are made up of wires that are "grounded" (kept at 0V). The top wire is connected to a voltage source that keeps it at a constant 100V.

### 19.2.1 Laplace's Elliptic PDE (Theory)

We consider the entire square in Figure 19.1 as our boundary with the voltages prescribed upon it. If we imagine infinitesimal insulators placed at the top corners of the box, then we will have a closed boundary. Because the values of the potential are given on all sides, we have Neumann conditions on the boundary and, according to Table 19.2, a unique and stable solution.

**Table 19.2** The relation between boundary conditions and uniqueness for PDEs.

**Figure 19.1** (a) The shaded region of space within a square in which we determine the electric potential by solving Laplace's equation. There is a wire at the top kept at a constant 100 V and a grounded wire (dashed) at the sides and bottom. (b) The computed electric potential as a function of _x_ and _y_. The projections onto the shaded _xy_ plane are equipotential (contour) lines.

It is known from classical electrodynamics that the electric potential  ( _x_ ) arising from static charges satisfies Poisson's PDE (Jackson, 1988):

(19.3)

where _ρ_ ( _x_ ) is the charge density. In charge-free regions of space, that is, regions where _ρ_ ( _x_ ) = 0, the potential satisfies _Laplace's equation:_

(19.4)

Both these equations are elliptic PDEs of a form that occurs in various applications. We solve them in 2D rectangular coordinates:

(19.5)

(19.6)

In both cases, we see that the potential depends simultaneously on _x_ and _y._ For Laplace's equation, the charges, which are the source of the field, enter indirectly by specifying the potential values in some region of space; for Poisson's equation they enter directly.

## 19.3 Fourier Series Solution of a PDE

For the simple geometry of Figure 19.1, an analytic solution of Laplace's equation (19.5) exists in the form of an infinite series. If we assume that the solution is the product of independent functions of _x_ and _y_ and substitute the product into (19.5), we obtain

(19.7)

Because _X_ ( _x_ ) is a function of only _x_ , and _Y_ ( _y_ ) is a function of only _y,_ the derivatives in (19.7) are _ordinary_ as opposed to _partial_ derivatives. Because _X_ ( _x_ ) and _Y_ ( _y_ ) are assumed to be independent, the only way (19.7) can be valid for _all_ values of _x_ and _y_ is for each term in (19.7) to be equal to a constant:

(19.8)

(19.9)

We shall see that this choice of sign for the constant matches the boundary conditions and gives us periodic behavior in _x._ The other choice of sign would give periodic behavior in _y,_ and that would not work with these boundary conditions.

The solutions for _X_ ( _x_ ) are periodic, and those for _Y_ ( _y_ ) are exponential:

(19.10)

The _x_ = 0 boundary condition  ( _x_ = 0, _y_ ) = 0 can be met only if _B_ = 0. The _x_ = _L_ boundary condition  ( _x_ = _L, y_ ) = 0 can be met only for

(19.11)

Such being the case, for each value of _n_ there is the solution

(19.12)

For each value of _k n_, _Y_ ( _y_ ) must satisfy the _y_ boundary condition  ( _x_ , 0) = 0, which requires _D = −C_ :

(19.13)

Because we are solving linear equations, the principle of linear superposition holds, which means that the most general solution is the sum of the products:

(19.14)

The _E n_ values are arbitrary constants and are fixed by requiring the solution to satisfy the remaining boundary condition at _y = L_ ,  ( _x, y = L_ ) = 100 V:

(19.15)

We determine the constants _E_ _n_ by projection: multiply both sides of the equation by sin _mπ/Lx,_ with _m_ an integer, and integrate from 0 to _L:_

(19.16)

The integral on the LHS is nonzero only for _n = m,_ which yields

(19.17)

Finally, we obtain an infinite series (analytic solution) for the potential at any point ( _x, y_ ):

(19.18)

### 19.3.1 Polynomial Expansion as an Algorithm

If we try to use (19.18) as an algorithm, we must terminate the sum at some point. Yet in practice the convergence of the series is so painfully slow that many terms are needed for good accuracy, and so the round-off error may become a problem. In addition, the sinh functions in (19.18) overflow for large _n,_ which can be avoided somewhat by expressing the quotient of the two sinh functions in terms of exponentials and then taking a large _n_ limit:

(19.19)

A third problem with the "analytic" solution is that a Fourier series converges only in the _mean square_ (Figure 19.2). This means that it converges to the _average_ of the left- and right-hand limits in the regions where the solution is discontinuous (Kreyszig, 1998), such as in the corners of the box. Explicitly, what you see in Figure 19.2 is a phenomenon known as the _Gibbs overshoot_ that occurs when a Fourier series with a finite number of terms is used to represent a discontinuous function. Rather than fall off abruptly, the series develops large oscillations that tend to overshoot the function at the corner. To obtain a smooth solution, we had to sum 40 000 terms, where, in contrast, the numerical solution required only several hundred steps.

**Figure 19.2** The analytic (Fourier series) solution of Laplace's equation summing 21 terms. Gibbs overshoot leads to the oscillations near _x_ = 0, and persist even if a large number of terms is summed.

## 19.4 Finite-Difference Algorithm

To solve our 2D PDE numerically, we divide space up into a lattice (Figure 19.3) and solve for _U_ at each site on the lattice. Because we will express derivatives in terms of the finite differences in the values of _U_ at the lattice sites, this is called a _finite-difference_ method. A numerically more efficient method, but with more complicated set up, is the _finite-element_ method (FEM) that solves the PDE for small geometric elements and then matches the solution over the elements. We discuss FEM in Chapter 23.

To derive the finite-difference algorithm for the numeric solution of (19.5), we follow the same path taken in Section 5.1 to derive the forward-difference algorithm for differentiation. We start by adding the two Taylor expansions of the potential to the right and left of ( _x, y_ ) and the two for above and below ( _x, y_ ):

(19.20)

(19.21)

(19.22)

(19.23)

**Figure 19.3** The algorithm for Laplace's equation in which the potential at the point ( _x, y_ ) = ( _i, j_ )Δ equals the average of the potential values at the four nearest-neighbor points. The nodes with white centers correspond to fixed values of the potential along the boundaries.

All odd terms cancel when we add these equations in pairs, and we obtain a central-difference approximation for the second partial derivative good to order Δ4:

(19.24)

(19.25)

Substitution of both these approximations in Poisson's equation (19.6) leads us to a finite-difference form of the PDE:

(19.26)

(19.27)

We take the _x_ and _y_ grids to be of equal spacings Δ _x_ = Δ _y_ = Δ, and thus obtain a simple form for the equation

(19.28)

The reader will notice that this equation shows a relation among the solutions at five points in space. When _U_ ( _x, y_ ) is evaluated for the _N x x_ values on the lattice and for the _N y y_ values, we obtain a set of _N x_ × _N y_ simultaneous linear algebraic equations for `**U[i,j]**` to solve. One approach is to solve these equations explicitly as a (big) matrix problem. This is attractive, as it is a direct solution, but it requires a great deal of memory and accounting. The approach we use follows from the algebraic solution of (19.28) for _U_ ( _x, y_ ):

(19.29)

where we would omit the _ρ_ ( _x_ ) term for Laplace's equation. In terms of discrete locations on our lattice, the _x_ and _y_ variables are

(19.30)

where we have placed our lattice in a square of side _L._ The finite-difference algorithm (19.29) becomes

(19.31)

This equation says that when we have a proper solution, it will be the average of the potential at the four nearest neighbors (Figure 19.3) plus a contribution from the local charge density. As an algorithm, (19.31) does not provide a direct solution to Poisson's equation, but rather must be repeated many times to converge upon the solution. We start with an initial guess for the potential, improve it by sweeping through all space taking the average over nearest neighbors at each node, and keep repeating the process until the solution no longer changes to some level of precision or until failure is evident. When converged, the initial guess is said to have _relaxed_ into the solution.

A reasonable question with this simple an approach is, "Does it always converge, and if so, does it converge fast enough to be useful?" In some sense, the answer to the first question is not an issue; if the method does not converge, then we will know it; otherwise we have ended up with a solution and the path we followed to get there is no body's business! The answer to the question of speed is that relaxation methods may converge slowly (although still faster than a Fourier series), yet we will show you two clever tricks to accelerate the convergence.

At this point, it is important to remember that our algorithm arose from expressing the Laplacian ∇2 in rectangular coordinates. While this does not restrict us from solving problems with circular symmetry, there may be geometries where it is better to develop an algorithm based on expressing the Laplacian in cylindrical or spherical coordinates in order to have grids that fit the geometry better.

### 19.4.1 Relaxation and Over-relaxation

There are a number of ways in which algorithm (19.31) can be iterated so as to convert the boundary conditions to a solution. Its most basic form is the _Jacobi method_ and is one in which the potential values are not changed until an entire sweep of applying (19.31) at each point is completed. This maintains the symmetry of the initial guess and boundary conditions. A rather obvious improvement on the Jacobi method utilizes the updated guesses for the potential in (19.31) as soon as they are available. As a case in point, if the sweep starts in the upper left-hand corner of Figure 19.3, then the leftmost `**U([ −1, j]**` and topmost `**U[i,j−l]**` values of the potential used will be from the present generation of guesses, while the other two values of the potential will be from the previous generation: (Gauss–Seidel method)

(19.32)

This technique, known as the _Gauss–Seidel method,_ usually leads to accelerated convergence, which in turn leads to less round-off error. It also uses less memory as there is no need to store two generations of guesses. However, it does distort the symmetry of the boundary conditions, which one hopes is insignificant when convergence is reached.

A less obvious improvement in the relaxation technique, known as _successive over-relaxation_ (SOR), starts by writing algorithm (19.31) in a form that determines the new values of the potential _U_ (new) as the old values _U_ (old) plus a correction or residual _r_ :

(19.33)

While the Gauss–Seidel technique may still be used to incorporate the updated values in _U_ (old) to determine _r_ , we rewrite the algorithm here in the general form

(19.34)

The successive over-relaxation technique (Press _et al.,_ 1994; Garcia, 2000) proposes that if convergence is obtained by adding _r_ to _U_ , then more rapid convergence might be obtained by adding more or less of _r_ :

(19.35)

where _ω_ is a parameter that amplifies or reduces the residual. The nonaccelerated relaxation algorithm (19.32) is obtained with _ω_ = 1, accelerated convergence (over-relaxation) is obtained with _ω_ ≥ 1, and underrelaxation is obtained with _ω_ < 1. Values of   often work well, yet _ω_ > 2 may lead to numerical instabilities. Although a detailed analysis of the algorithm is needed to predict the optimal value for _ω_ , we suggest that you explore different values for _ω_ to see which one works best for your particular problem.

### 19.4.2 Lattice PDE Implementation

In Listing 19.1, we present the code `**LaplaceLine.py**` that solves the square-wire problem (Figure 19.1). Here we have kept the code simple by setting the length of the box _L_ = _N_ max Δ = 100 and by taking Δ = 1:

(19.36)

We run algorithm (10.19) for a fixed 1000 iterations. A better code would vary Δ and the dimensions and would quit iterating once the solution converges to some tolerance. Study, compile, and execute the basic code.

**Listing 19.1** `**LaplaceLine.py**` solves Laplace's equation via relaxation. Various parameters should be adjusted for an accurate solution.

## 19.5 Assessment via Surface Plot

After executing `**LaplaceLine.py**`, you should see a surface plot like Figure 19.1. Study this file in order to understand how to make surface plots with Matplotlib in Python. It is important to visualize your output as a surface plot to establish the reasonableness of the solution.

## 19.6 Alternate Capacitor Problems

* * *

We give you (or your instructor) a choice now. You can carry out the assessment using our wire-plus-grounded-box problem, or you can replace that problem with a more interesting one involving a realistic capacitor or nonplanar capacitors. We now describe the capacitor problem and then move on to the assessment and exploration.

* * *

Elementary textbooks solve the capacitor problem for the uniform field confined between two infinite plates. The field in a finite capacitor varies near the edges (edge effects) and extends beyond the edges of the capacitor (fringe fields). We model the realistic capacitor in a grounded box (Figure 19.4) as two plates (wires) of finite length. Write your simulation such that it is convenient to vary the grid spacing Δ and the geometry of the box and plate. We pose three versions of this problem, each displaying somewhat different physics. In each case, the boundary condition _U_ = 0 on the surrounding box must be imposed for all iterations in order to obtain a unique solution.

**Figure 19.4** (a) A simple model of a parallel-plate capacitor within a box. A realistic model would have the plates close together, in order to condense the field, and the enclosing grounded box so large that it has no effect on the field near the capacitor. (b) A numerical solution for the electric potential for this geometry. The projection on the _xy_ plane gives the equipotential lines.

  1. For the simplest version, assume that the plates are very thin sheets of conductors, with the top plate maintained at 100 V and the bottom at −100 V. Because the plates are conductors, they must be equipotential surfaces, and a battery can maintain them at constant voltages. Write or modify the given program to solve Laplace's equation such that the plates have fixed voltages.
  2. For the next version of this problem, assume that the plates are composed of a line of dielectric material with uniform charge densities _ρ_ on the top and − _ρ_ on the bottom. Solve Poisson's equation (19.3) in the region including the plates, and Laplace's equation elsewhere. Experiment until you find a numerical value for _ρ_ that gives a potential similar to that shown in Figure 19.6 for plates with fixed voltages.
  3. For the final version of this problem investigate how the charges on a capacitor with finite-thickness conducting plates (Figure 19.5) distribute themselves. Because the plates are conductors, they are still equipotential surfaces at 100 and −100 V, only now you should make them have a thickness of at least 2Δ (so we can see the difference between the potential near the top and the bottom surfaces of the plates). Such being the case, we solve Laplace's equation (19.4) much as before to determine _U_ ( _x, y_ ). Once we have _U_ ( _x, y_ ), we substitute it into Poisson's equation (19.3) and determine how the charge density distributes itself along the top and bottom surfaces of the plates. _Hint:_ Because the electric field is no longer uniform, we know that the charge distribution also will no longer be uniform. In addition, because the electric field now extends beyond the ends of the capacitor and because field lines begin and end on charge, some charge may end up on the edges and outer surfaces of the plates (Figure 19.4).

**Figure 19.5** A guess as to how charge may rearrange itself on finite conducting plates.

**Figure 19.6** (a) A visualization of the computed electric potential for a capacitor with finite width plates. (b) A visualization of the charge distribution along one plate determined by evaluating ∇2 _U_ ( _x, y_ ) (courtesy of J. Wetzel). Note the "lightening rod" effect of charge accumulating at corners and points.

  4. The numerical solution to our PDE can be applied to arbitrary boundary conditions. Two boundary conditions to explore are triangular and sinusoidal:

(19.37)

  5. _Square conductors:_ You have designed a piece of equipment consisting of a small metal box at 100 V within a larger grounded one (Figure 19.7). You find that sparking occurs between the boxes, which means that the electric field is too large. You need to determine where the field is greatest so that you can change the geometry and eliminate the sparking. Modify the program to satisfy these boundary conditions and to determine the field between the boxes. Gauss's law tells us that the field vanishes within the inner box because it contains no charge. Plot the potential and equipotential surfaces and sketch in the electric field lines. Deduce where the electric field is most intense and try redesigning the equipment to reduce the field.
  6. _Cracked cylindrical capacitor:_ You have designed the cylindrical capacitor containing a long outer cylinder surrounding a thin inner cylinder (Figure 19.7b). The cylinders have a small crack in them in order to connect them to the battery that maintains the inner cylinder at −100 V and outer cylinder at 100 V. Determine how this small crack affects the field configuration. In order for a unique solution to exist for this problem, place both cylinders within a large grounded box. Note that because our algorithm is based on expansion of the Laplacian in rectangular coordinates, you cannot just convert it to a radial and angle grid.

**Figure 19.7** (a) The geometry of a capacitor formed by placing two long, square cylinders within each other. (b) The geometry of a capacitor formed by placing two long, circular cylinders within each other. The cylinders are cracked on the side so that wires can enter the region.

## 19.7 Implementation and Assessment

  1. Write or modify the program to find the electric potential for a capacitor within a grounded box. Use the labeling scheme as shown in Figure 19.4a.
  2. To start, have your program undertake 1000 iterations and then quit. During debugging, examine how the potential changes in some key locations as you iterate toward a solution.
  3. Repeat the process for different step sizes Δ and draw conclusions regarding the stability and accuracy of the solution.
  4. Once your program produces reasonable solutions, modify it so that it stops iterating after convergence is reached, or if the number of iterations becomes too large. Rather than trying to discern small changes in highly compressed surface plots, use a numerical measure of precision, for example,

(19.38)

which samples the solution along the diagonal. Remember, this is a simple algorithm and so may require many iterations for high precision. You should be able to obtain changes in the trace that are less than 1 part in 104. (The `**break**` command or a `**while**` loop is useful for this type of test.)
  5. Equation 19.35 expresses the _successive over-relaxation_ technique in which convergence is accelerated by using a judicious choice of _ω._ Determine by trial and error a best value of _ω._ This should let you double the speed of the algorithm.
  6. Now that your code is accurate, modify it to simulate a more realistic capacitor in which the plate separation is approximately 1/10 of the plate length. You should find the field more condensed and more uniform between the plates.
  7. If you are working with the wire-in-the-box problem, compare your numerical solution to the analytic one (19.18). Do not be surprised if you need to sum thousands of terms before the analytic solution converges!

## 19.8 Electric Field Visualization (Exploration)

Plot the equipotential surfaces on a separate 2D plot. Start with a crude, hand-drawn sketch of the electric field by drawing curves orthogonal to the equipotential lines, beginning and ending on the boundaries (where the charges lie). The regions of high density are regions of high electric field. Physics tells us that the electric field _E_ is the negative gradient of the potential:

(19.39)

where   is a unit vector in the _i_ direction. While at first it may seem that some work is involved in determining these derivatives, once you have a solution for _U_ ( _x, y_ ) on a grid, it is simple to use the central-difference approximation for the derivative to determine the field, for example:

(19.40)

Once you have a data file representing such a vector field, it can be visualized by plotting arrows of varying lengths and directions, or with just lines (Figure 19.8). In Section 1.5.6, we have shown how to do this with Mayavi.

**Figure 19.8** (a) Computed equipotential surfaces and electric field lines for a realistic capacitor. (b) Equipotential surfaces and electric field lines mapped onto the surface for a 3D capacitor constructed from two tori.

## 19.9 Review Exercise

You are given a simple Laplace-type equation

(19.41)

where _x_ and _y_ are the Cartesian spatial coordinates and _ρ_ ( _x, y_ ) is the charge density in space.

  1. Develop a simple algorithm that will permit you to solve for the potential _u_ between two square conductors kept at fixed _u,_ with a charge density _ρ_ between them.
  2. Make a simple sketch that shows with arrows how your algorithm works.
  3. Make sure to specify how you start and terminate the algorithm.
  4. _Thinking outside the box_  : Find the electric potential for all points _outside_ the charge-free square shown in Figure 19.1. Is your solution unique?

1) Although conclusions drawn for exact PDEs may differ from those drawn for the finite-difference equations, we use for our algorithms, they are usually the same. In fact, Morse and Feshbach (Morse and Feshbach, 1953) use the finite-difference form to derive the relations between boundary conditions and uniqueness for each type of equation shown in Table 19.2 (Jackson, 1988).

# 20
Heat Flow via Time Stepping

* * *

As the present now And the first one now

Will later be past Will later be last

The order is For the times they are a-changin'.

Rapidly fadin'

_Bob Dylan_

_This chapter examines the heat equation and develops the leapfrog method for solving it on a space–time lattice. We also develop an improved Crank–Nicolson method that determines the solution over all of space in a single step. Time stepping is simple, yet important, and we will see it again when we attack various wave equations._

## 20.1 Heat Flow via Time-Stepping (Leapfrog)

**Problem** You are given an aluminum bar of length _L_ = 1 m and width _w_ aligned along the _x_ -axis (Figure 20.1). It is insulated along its length but not at its ends. Initially the bar is at a uniform temperature of 100 °C, and then both ends are placed in contact with ice water at 0°C. Heat flows out of the noninsulated ends only. Your _problem_ is to determine how the temperature will vary as we move along the length of the bar at later times.

**Figure 20.1** A metallic bar insulated along its length with its ends in contact with ice. The bar is displayed in dark gray and the insulation is of lighter gray.

## 20.2 The Parabolic Heat Equation (Theory)

A basic fact of nature is that heat flows from hot to cold, that is, from regions of high temperature to regions of low temperature. We give these words mathematical expression by stating that the rate of heat flow **H** through a material is proportional to the gradient of the temperature _T_ across the material:

(20.1)

where _K_ is the thermal conductivity of the material. The total amount of heat _Q_ ( _t_ ) in the material at any one time is proportional to the integral of the temperature over the material's volume:

(20.2)

where _C_ is the specific heat of the material and _ρ_ is its density. Because energy is conserved, the rate of decrease in _Q_ with time must equal the amount of heat flowing out of the material. After this energy balance is struck and the divergence theorem applied, there results the _heat equation_

(20.3)

The heat equation (20.3) is a parabolic PDE with space and time as independent variables. The specification of this problem implies that there is no temperature variation in directions perpendicular to the bar ( _y_ and _z_ ), and so we have only one spatial coordinate in the Laplacian:

(20.4)

As given, the initial temperature of the bar and the boundary conditions are

(20.5)

### 20.2.1 Solution: Analytic Expansion

Analogous to Laplace's equation, the analytic solution starts with the assumption that the solution separates into the product of functions of space and time:

(20.6)

When (20.6) is substituted into the heat equation (20.4) and the resulting equation is divided by _X_ ( _x_ ) ( _t_ ), two noncoupled ODEs result:

(20.7)

where _k_ is a constant still to be determined. The boundary condition that the temperature equals zero at _x_ = 0 requires a sine function for _X_ :

(20.8)

The boundary condition that the temperature equals zero at _x_ = _L_ requires the sine function to vanish there:

(20.9)

To avoid blow up, the time function must be a decaying exponential with _k_ in the exponent:

(20.10)

where _n_ can be any integer and _A n_ is an arbitrary constant. Because (20.4) is a linear equation, the most general solution is a linear superposition of _X n_( _x_ ) _T n_( _t_ ) products for all values of _n_ :

(20.11)

The coefficients _A n_ are determined by the initial condition that at time _t_ = 0 the entire bar has temperature _T_ = 100 K:

(20.12)

Projecting the sine functions determines _A n_ = 4 _T_ 0/ _n_ π for _n_ odd, and so

(20.13)

### 20.2.2 Solution: Time Stepping

As we did with Laplace's equation, the numerical solution is based on converting the differential equation to a finite-difference ("difference") equation. We discretize space and time on a lattice (Figure 20.2) and solve for solutions on the lattice sites. The sites along the top with white centers correspond to the known values of the temperature for the initial time, while the sites with white centers along the sides correspond to the fixed temperature along the boundaries. If we _also_ knew the temperature for times along the bottom row, then we could use a relaxation algorithm as we did for Laplace's equation. However, with only the top and side rows known, we shall end up with an algorithm that steps forward in time one row at a time, as in the children's game _leapfrog._

**Figure 20.2** The algorithm for the heat equation in which the temperature at the location _x_ = _i_ Δ _x_ and time _t_ = ( _j_ \+ 1)Δ _t_ is computed from the temperature values at three points of an earlier time. The nodes with white centers correspond to known initial and boundary conditions (the boundaries are placed artificially close for illustrative purposes).

As is often the case with PDEs, the algorithm is customized for the equation being solved and for the constraints imposed by the particular set of initial and boundary conditions. With only one row of times to start with, we use a forward-difference approximation for the time derivative of the temperature:

(20.14)

Because we know the spatial variation of the temperature along the entire top row and the left and right sides, we are less constrained with the space derivative as with the time derivative. Consequently, as we did with the Laplace equation, we use the more accurate central-difference approximation for the space derivative:

(20.15)

Substitution of these approximations into (20.4) yields the heat difference equation

(20.16)

We reorder (20.16) into a form in which _T_ can be stepped forward in _t_ :

(20.17)

where _x_ = _i_ Δ _x_ and _t_ = _j_ Δ _t_. This algorithm is _explicit_ because it provides a solution in terms of known values of the temperature. If we tried to solve for the temperature at all lattice sites in Figure 20.2 simultaneously, then we would have an _implicit_ algorithm that requires us to solve equations involving unknown values of the temperature. We see that the temperature at space–time point ( _i_ , _j_ \+ 1) is computed from the three temperature values at an earlier time _j_ and at adjacent space values _i_ ± 1, _i_. We start the solution at the top row, moving it forward in time for as long as we want and keeping the temperature along the ends fixed at 0 K (Figure 20.2).

### 20.2.3 von Neumann Stability Assessment

When we solve a PDE by converting it to a difference equation, we hope that the solution of the latter is a good approximation to the solution of the former. If the difference-equation solution diverges, then we know we have a bad approximation, but if it converges, then we may feel confident that we have a good approximation to the PDE. The _von Neumann stability analysis_ is based on the assumption that eigenmodes of the difference equation can be expressed as

(20.18)

where _x_ = _m_ Δ _x_ and _t_ = _j_ Δ _t_ , and   is the imaginary number. The constant _k_ in (20.18) is an unknown wave vector (2π/ _λ_ ), and  ( _k_ ) is an unknown complex function. View (20.18) as a basis function that oscillates in space (the exponential) with an amplitude or _amplification factor_  ( _k_ ) _j_ that increases by a power of   for each time step. If the general solution to the difference equation can be expanded in terms of these eigenmodes, then the general solution will be stable if the eigenmodes are stable. Clearly, for an eigenmode to be stable, the amplitude   cannot grow in time _j_ , which means | ( _k_ )| < 1 for all values of the parameter _k_ (Press _et al_., 1994; Ancona, 2002).

Application of a stability analysis is more straightforward than it might appear. We just substitute expression (20.18) into the difference equation (20.17):

(20.19)

After canceling some common factors, it is easy to solve for  :

(20.20)

In order for | ( _k_ )| < 1 for all possible _k_ values, we must have

(20.21)

This equation tells us that if we make the time step Δ _t_ smaller, we will always improve the stability, as we would expect. But if we decrease the space step Δ _x_ without a simultaneous quadratic _increase_ in the time step, we will worsen the stability. The lack of space–time symmetry arises from our use of stepping in time, but not in space.

In general, you should perform a stability analysis for every PDE you have to solve, although it can get complicated (Press _et al_., 1994). Yet even if you do not, the lesson here is that you may have to try different _combinations_ of Δ _x_ and Δ _t_ variations until a stable, reasonable solution is obtained. You may expect, nonetheless, that there are choices for Δ _x_ and Δ _t_ for which the numerical solution fails and that simply decreasing an individual Δ _x_ or Δ _t_ , in the hope that this will increase precision, may not improve the solution.

**Listing 20.1** **EqHeat.py** solves the 1D space heat equation on a lattice by leapfrogging (time-stepping) the initial conditions forward in time. You will need to adjust the parameters to obtain a solution like those in the figures.

### 20.2.4 Heat Equation Implementation

Recall that we want to solve for the temperature distribution within an aluminum bar of length _L_ = 1 m subject to the boundary and initial conditions

(20.22)

The thermal conductivity, specific heat, and density for Al are

(20.23)

  1. Write or modify `**EqHeat.py**` in Listing 20.1 to solve the heat equation.
  2. Define a 2D array `**T[101,2]**` for the temperature as a function of space and time. The first index is for the 100 space divisions of the bar, and the second index is for present and past times (because you may have to make thousands of time steps, you save memory by saving only two times).
  3. For time _t_ = 0 ( _j_ = 1), initialize `**T**` so that all points on the bar except the ends are at 100 K. Set the temperatures of the ends to 0 K.
  4. Apply (20.14) to obtain the temperature at the next time step.
  5. Assign the present-time values of the temperature to the past values: `**T[i,1] = T[i,2], i = 1,...,101**`.
  6. Start with 50 time steps. Once you are confident the program is running properly, use thousands of steps to see the bar cool smoothly with time. For approximately every 500 time steps, print the time and temperature along the bar.

## 20.3 Assessment and Visualization

  1. Check that your program gives a temperature distribution that varies smoothly along the bar and agrees with the boundary conditions, as in Figure 20.3.

**Figure 20.3** A numerical calculation of the temperature vs. position and vs. time, with isotherm contours projected onto the horizontal plane.

  2. Check that your program gives a temperature distribution that varies smoothly with time and reaches equilibrium. You may have to vary the time and space steps to obtain stable solutions.
  3. Compare the analytic and numeric solutions (and the wall times needed to compute them). If the solutions differ, suspect the one that does not appear smooth and continuous.
  4. Make a surface plot of temperature vs. position vs. time.
  5. Plot the _isotherms_ (contours of constant temperature).
  6. _Stability test:_ Verify the stability condition (20.21) by observing how the temperature distribution diverges if _η_ > 1/4.
  7. _Material dependence:_ Repeat the calculation for iron. Note that the stability condition requires you to change the size of the time step.
  8. _Initial sinusoidal distribution_ sin(π _x_ / _L_ ) _:_ Compare to the analytic solution,

(20.24)

  9. _Two bars in contact:_ Two identical bars 0.25 m long are placed in contact along one of their ends with their other ends kept at 0 K. One is kept in a heat bath at 100 K, and the other at 50 K. Determine how the temperature varies with time and location (Figure 20.4).
  10. _Radiating bar (Newton's cooling):_ Imagine now that instead of being insulated along its length, a bar is in contact with an environment at a temperature _T_ e. Newton's law of cooling (radiation) says that the rate of temperature change as a result of radiation is

(20.25)

where _h_ is a positive constant. This leads to the modified heat equation

(20.26)

Modify the algorithm to include Newton's cooling and compare the cooling of a radiating bar with that of the insulated bar.

## 20.4 Improved Heat Flow: Crank–Nicolson Method

The Crank–Nicolson method (Crank and Nicolson, 1946) provides a higher degree of precision for the heat equation (20.3) than the simple leapfrog method we have just discussed. This method calculates the time derivative with a central-difference approximation, in contrast to the forward-difference approximation used previously. In order to avoid introducing error for the initial time step where only a single time value is known, the method uses a _split time step_ ,1) so that time is advanced from time _t_ to _t_ \+ Δ _t_ /2:

**Figure 20.4** Temperature vs. position and time when two bars at differing temperatures are placed in contact at _t_ = 0. The projected contours show the isotherms.

(20.27)

Yes, we know that this looks just like the forward-difference approximation for the derivative at time _t_ \+ Δ _t_ , for which it would be a bad approximation; regardless, it is a better approximation for the derivative at time _t_ \+ Δ _t_ /2, although it makes the computation more complicated. Likewise, in (20.14) we gave the central-difference approximation for the second space derivative for time _t_. For _t_ = _t_ \+ Δ _t_ /2, that becomes

(20.28)

In terms of these expressions, the heat difference equation is

(20.29)

We group together terms involving the same temperature to obtain an equation with future times on the LHS and present times on the RHS:

(20.30)

This equation represents an _implicit_ scheme for the temperature _T i,j_, where the word "implicit" means that we must solve simultaneous equations to obtain the full solution for all space. In contrast, an _explicit_ scheme requires iteration to arrive at the solution. It is possible to solve (20.30) simultaneously for all unknown temperatures (1 ≤ _i_ ≤ _N_ ) at times _j_ and _j_ \+ 1. We start with the initial temperature distribution throughout all of space, the boundary conditions at the ends of the bar for all times, and the approximate values from the first derivative:

(20.31)

We rearrange (20.30) so that we can use these known values of _T_ to step the _j_ = 0 solution forward in time by expressing (20.30) as a set of simultaneous linear equations (in the matrix form):

(20.32)

Observe that the _T_ s on the RHS are all at the present time _j_ for various positions, and at future time _j_ \+ 1 for the two ends (whose _T_ s are known for all times via the boundary conditions). We start the algorithm with the _T_ _i,j_ =0 values of the initial conditions, then solve a matrix equation to obtain _T_ _i,j_ =1. With that we know all the terms on the RHS of the equations ( _j_ = 1 throughout the bar and _j_ = 2 at the ends) and so can repeat the solution of the matrix equations to obtain the temperature throughout the bar for _j_ = 2. So again, we time-step forward, only now we solve matrix equations at each step. That gives us the spatial solution at all locations directly.

Not only is the Crank–Nicolson method more precise than the low-order time-stepping method, but it also is stable for all values of Δ _t_ and Δ _x_. To prove that, we apply the von Neumann stability analysis discussed in Section 20.2.3 to the Crank–Nicolson algorithm by substituting (20.17) into (20.30). This determines an amplification factor

(20.33)

Because sin2() is positive definite, this proves that | | ≤ 1 for all Δ _t_ , Δ _x_ , and _k_.

### 20.4.1 Solution of Tridiagonal Matrix Equations

The Crank–Nicolson equations (20.32) are in the standard [ **A** ] ** _x_** = **_b_** form for linear equations, and so we can use our previous methods to solve them. Nonetheless, because the coefficient matrix [ **A** ] is tridiagonal (zero elements except for the main diagonal and two diagonals on either side of it),

(20.34)

a more robust and faster solution exists that makes this implicit method as fast as an explicit one. Because tridiagonal systems occur frequently, we now outline the specialized technique for solving them (Press _et al_., 1994). If we store the matrix elements _a_ _i_ , _j_ using both subscripts, then we will need _N_ 2 locations for elements and _N_ 2 operations to access them. However, for a tridiagonal matrix, we need to store only the vectors { _d i_} _i_ =1, _N_ , { _c i_} _i_ =1, _N_ , and { _a i_} _i_ =1, _N_ , along, above, and below the diagonals. The single subscripts on _a i_, _d i_, and _c i_ reduce the processing from _N_ 2 to (3 _N_ − 2) elements.

We solve the matrix equation by manipulating the individual equations until the coefficient matrix is _upper triangular_ with all the elements of the main diagonal equal to 1. We start by dividing the first equation by _d_ 1, then subtract _a_ 2 times the first equation,

(20.35)

and then dividing the second equation by the second diagonal element,

(20.36)

Assuming that we can repeat these steps without ever dividing by zero, the system of equations will be reduced to upper triangular form,

(20.37)

where _h_ 1 = _c_ 1/ _d_ 1 and _p_ 1 = _b_ 1/ _d_ 1. We then recur for the others elements:

(20.38)

Finally, back substitution leads to the explicit solution for the unknowns:

(20.39)

In Listing 20.2, we give the program `**HeatCNTridiag.py**` that solves the heat equation using the Crank–Nicolson algorithm via a triadiagonal reduction.

**Listing 20.2** **HeatCNTridiag.py** is the complete program for solution of the heat equation in one space dimension and time via the Crank–Nicolson method. The resulting matrix equations are solved via a technique specialized to tridiagonal matrices.

### 20.4.2 Crank–Nicolson Implementation, Assessment

  1. Write a program using the Crank–Nicolson method to solve for the heat flow in the metal bar of Section 20.1 for at least 100 time steps.
  2. Solve the linear system of equations (20.32) using either Matplotlib or the special tridiagonal algorithm.
  3. Check the stability of your solution by choosing different values for the time and space steps.
  4. Construct a contoured surface plot of temperature vs. position and vs. time.
  5. Compare the implicit and explicit algorithms used in this chapter for relative precision and speed. You may assume that a stable answer that uses very small time steps is accurate.

1) In Section 22.2.1 we develop a different split-time algorithm for the solution of the Schrödinger equation. There the real and imaginary parts of the wave function are computed at times that differ by Δ _t_ /2.

# 21
Wave Equations I: Strings and Membranes

* * *

In this chapter and in the next, we explore the numerical solution of several PDEs that yield waves as solutions. If you have skipped the discussion of the heat equation in Chapter 20, then this chapter will be the first example of how initial conditions are propagated forward in time with a time-stepping or leapfrog algorithm. An important aspect of this chapter is its demonstration that once you have a working algorithm for solving a wave equation, you can include considerably more physics than is possible with analytic treatments. First, we deal with a number of aspects of 1D waves on a string, and then with 2D waves on a membrane. In the next chapter, we look at quantum wave packets and E&M waves. In Chapter 24, we look at shock and solitary waves.

## 21.1 A Vibrating String

**Problem** Recall the demonstration from elementary physics in which a string tied down at both ends is plucked "gently" at one location and a pulse is observed to travel along the string. Likewise, if the string has one end free and you shake it just right, a standing-wave pattern is set up in which the nodes remain in place and the antinodes move just up and down. Your _problem_ is to develop an accurate model for wave propagation on a string, and to see if you can set up traveling- and standing-wave patterns.1)

## 21.2 The Hyperbolic Wave Equation (Theory)

Consider a string of length _L_ tied down at both ends (Figure 21.1a). The string has a constant density _ρ_ per unit length, a constant tension _T,_ no frictional forces acting on it, and a tension that is so high that we may ignore sagging as a result of gravity. We assume that the displacement of the string from its rest position _y_ ( _x, t_ ) is in the vertical direction only and that it is a function of the horizontal location along the string _x_ and the time _t._

**Figure 21.1** (a) A stretched string of length _L_ tied down at both ends and under high enough tension so that we can ignore gravity. The vertical disturbance of the string from its equilibrium position is _y_ ( _x, t_ ). (b) A differential element of the string showing how the string's displacement leads to the restoring force.

To obtain a simple linear equation of motion (nonlinear wave equations are discussed in Chapters 24 and , we assume that the string's relative displacement _y_ ( _x, t_ )/ _L_ and slope _∂y/∂x_ are small. We isolate an infinitesimal section Δ _x_ of the string (Figure 21.1b) and see that the difference in the vertical components of the tension at either end of the string produces the restoring force that accelerates this section of the string in the vertical direction. By applying Newton's laws to this section, we obtain the familiar wave equation:

(21.1)

(21.2)

(21.3)

(21.4)

where we have assumed that _θ_ is small enough for sin  . The existence of two independent variables _x_ and _t_ makes this a PDE. The constant _c_ is the velocity with which a disturbance travels along the wave, and is seen to decrease for a denser string and increase for a tighter one. Note that this signal velocity _c_ is _not_ the same as the velocity of a string element  .

The initial condition for our problem is that the string is plucked gently and released. We assume that the "pluck" places the string in a triangular shape with the center of triangle 8/10 of the way down the string and with a height of 1:

(21.5)

Because (21.4) is second order in time, a second initial condition (beyond initial displacement) is needed to determine the solution. We interpret the "gentleness" of the pluck to mean that the string is released from rest:

(21.6)

The boundary conditions have both ends of the string tied down for all times:

(21.7)

### 21.2.1 Solution via Normal-Mode Expansion

The analytic solution to (21.4) is obtained via the familiar separation-of-variables technique. We assume that the solution is the product of a function of space and a function of time:

(21.8)

We substitute (21.8) into (21.4), divide by _y_ ( _x, t_ ), and are left with an equation that has a solution only if there are solutions to the two ODEs:

(21.9)

The angular frequency _ω_ and the wave vector _k_ are determined by demanding that the solutions satisfy the boundary conditions. Specifically, the string being attached at both ends demands

(21.10)

(21.11)

The time solution is

(21.12)

where the frequency of this _n_ th _normal mode_ is also fixed. In fact, it is the single frequency of oscillation that defines a normal mode. The _initial condition_ (21.5) of zero velocity, _∂y/∂t_ ( _t_ = 0) = 0, requires the _C n_ values in (21.12) to be zero. Putting the pieces together, the normal-mode solutions are

(21.13)

Because the wave equation (21.4) is linear in _y,_ the principle of linear superposition holds and the most general solution for waves on a string with fixed ends can be written as the sum of normal modes:

(21.14)

(Yet we will lose linear superposition once we include nonlinear terms in the wave equation.) The Fourier coefficient _B n_ is determined by the second initial condition (21.5), which describes how the wave is plucked:

(21.15)

We multiply both sides by sin _mk_ 0 _x_ , substitute the value of _y_ ( _x,_ 0) from (21.5), and integrate from 0 to _l_ to obtain

(21.16)

You will be asked to compare the Fourier series (21.14) to our numerical solution. While it is in the nature of the approximation that the precision of the numerical solution depends on the choice of step sizes, it is also revealing to realize that the precision of the so-called analytic solution depends on summing an infinite number of terms, which can be carried out only approximately.

### 21.2.2 Algorithm: Time Stepping

As with Laplace's equation and the heat equation, we look for a solution _y_ ( _x, t_ ) only for discrete values of the independent variables _x_ and _t_ on a grid (Figure 21.2):

(21.17)

(21.18)

In contrast to Laplace's equation where the grid was in two space dimensions, the grid in Figure 21.2 is in both space and time. That being the case, moving across a row corresponds to increasing _x_ values along the string for a fixed time, while moving down a column corresponds to increasing time steps for a fixed position. Although the grid in Figure 21.2 may be square, we cannot use a relaxation technique like we did for the solution of Laplace's equation because we do not know the solution on all four sides. The boundary conditions determine the solution along the right- and left-sides, while the initial time condition determines the solution along the top.

As with the Laplace equation, we use the central-difference approximation to _discretize_ the wave equation into a difference equation. First, we express the second derivatives in terms of finite differences:

(21.19)

Substituting (21.19) in the wave equation (21.4) yields the difference equation

(21.20)

**Figure 21.2** The solutions of the wave equation for four earlier space–time points are used to obtain the solution at the present time. The boundary and initial conditions are indicated by the white-centered dots.

Note that this equation contains three time values: _j_ \+ 1 = the future, _j_ = the present, and _j_ – 1 = the past. Consequently, we rearrange it into a form that permits us to predict the future solution from the present and past solutions:

(21.21)

Here _c_ ′ is a combination of numerical parameters with the dimension of velocity whose size relative to c determines the stability of the algorithm. The algorithm (21.21) propagates the wave from the two earlier times, _j_ and _j_ – 1, and from three nearby positions, _i_ – 1, _i,_ and _i_ \+ 1, to a later time _j_ \+ 1 and a single space position _i_ (Figure 21.2).

As you have seen in our discussion of the heat equation, a leapfrog method is quite different from a relaxation technique. We start with the solution along the topmost row and then move down one step at a time. If we write the solution for present times to a file, then we need to store only three time values on the computer, which saves memory. In fact, because the time steps must be quite small to obtain high precision, you may want to store the solution only for every fifth or tenth time.

Initializing the recurrence relation is a bit tricky because it requires displacements from two earlier times, whereas the initial conditions are for only one time. Nonetheless, the rest condition (21.5) when combined with the _central-difference_ approximation lets us extrapolate to negative time:

(21.22)

Here we take the initial time as _j_ = 1, and so _j_ = 0 corresponds to _t_ = –Δ _t_. Substituting this relation into (21.21) yields for the initial step

(21.23)

Equation 21.23 uses the solution throughout all space at the initial time _t_ = 0 to propagate (leapfrog) it forward to a time Δ _t_. Subsequent time steps use (21.21) and are continued for as long as you like.

As is also true with the heat equation, the success of the numerical method depends on the relative sizes of the time and space steps. If we apply a von Neumann stability analysis to this problem by substituting  , as we did in Section 20.2.3, a complicated equation results. Nonetheless, Press _et al._ (1994) shows that the difference-equation solution will be stable for the general class of transport equations if (Courant _et al.,_ 1928)

(21.24)

Equation 21.24 means that the solution gets better with smaller _time_ steps but gets worse for smaller space steps (unless you simultaneously make the time step smaller). Having different sensitivities to the time and space steps may appear surprising because the wave equation (21.4) is symmetric in _x_ and _t_ ; yet the symmetry is broken by the nonsymmetric initial and boundary conditions.

**Exercise** Figure out a procedure for solving for the wave equation for all times in just one step. Estimate how much memory would be required.

**Exercise** Try to figure out a procedure for solving for the wave motion with a relaxation technique. What would you take as your initial guess, and how would you know when the procedure has converged?

### 21.2.3 Wave Equation Implementation

The program `**EqStringAnimate.py**` in Listing 21.1 solves the wave equation for a string of length _L_ = 1 m with its ends fixed and with the gently plucked initial conditions. Note that our use of _L_ = 1 violates our assumption that   but makes it easy to display the results; you should try _L_ = 1000 to be realistic. The values of density and tension are entered as constants, _ρ_ = 0.01 kg/m and _T_ = 40 N, with the space grid set at 101 points, corresponding to Δ = 0.01 cm.

**Listing 21.1** **EqStringAnimate.py** solves the wave equation via time stepping for a string of length _L_ = 1 m with its ends fixed and with the gently plucked initial conditions. You will need to modify this code to include new physics.

### 21.2.4 Assessment, Exploration

  1. Solve the wave equation and make a surface plot of displacement vs. time and position.
  2. Explore a number of space and time step combinations. In particular, try steps that satisfy and that do not satisfy the Courant condition (21.24). Does your exploration confirm the stability condition?
  3. Compare the analytic and numeric solutions, summing at least 200 terms in the analytic solution.
  4. Use the plotted time dependence to estimate the peak's propagation velocity _c_. Compare the deduced _c_ to (21.4).
  5. Our solution of the wave equation for a plucked string leads to the formation of a wave packet that corresponds to the sum of multiple normal modes of the string. On the right in Figure 21.3 we show the motion resulting from the string initially placed in a single normal mode (standing wave),

(21.25)

**Figure 21.3** The vertical displacement as a function of position _x_ and time _t_ for a string initially placed in a standing mode when friction is included. Notice how the wave moves up and down with time (courtesy of J. Wiren).

but with friction (to be discussed soon) included.
Modify your program to incorporate this initial condition and see if a normal mode results.
  6. Observe the motion of the wave for initial conditions corresponding to the sum of two adjacent normal modes. Does beating occur?
  7. When a string is plucked near its end, a pulse reflects off the ends and bounces back and forth. Change the initial conditions of the model program to one corresponding to a string plucked exactly in its middle and see if a traveling or a standing wave results.
  8. Figure 21.4 shows the wave packets that result as a function of time for initial conditions corresponding to the double pluck. Verify that initial conditions of the form

(21.26)

lead to this type of a repeating pattern. In particular, observe whether the pulses move or just oscillate up and down.

**Figure 21.4** The vertical displacement as a function of position and time of a string initially plucked simultaneously at two points, as shown by arrows. Note that each initial peak breaks up into waves traveling to the right and to the left. The traveling waves invert on reflection from the fixed ends. As a consequence of these inversions, the _t_   12 wave is an inverted _t_ = 0 wave.

## 21.3 Strings with Friction (Extension)

* * *

The string problem we have investigated so far can be handled by either a numerical or an analytic technique. We now wish to extend the theory to include some more realistic physics. These extensions have only numerical solutions.

Plucked strings do not vibrate forever because there is friction in the real world. Consider again the element of a string between _x_ and _x_ \+ d _x_ (Figure 21.1b) but now imagine that this element is moving in a viscous fluid such as air. An approximate model for the frictional force has it pointing in a direction opposite the vertical velocity of the string and proportional to that velocity, as well as proportional to the length of the string element:

(21.27)

where _κ_ is a constant that is proportional to the viscosity of the medium in which the string is vibrating. Including this force in the equation of motion changes the wave equation to

(21.28)

In Figure 21.3, we show the resulting motion of a string plucked in the middle when friction is included. Observe how the initial pluck breaks up into waves traveling to the right and to the left that are reflected and inverted by the fixed ends. Because those parts of the wave with the higher velocity experience greater friction, the peak tends to be smoothed out the most as time progresses.

**Exercise** Generalize the algorithm used to solve the wave equation now to include friction and check if the wave's behavior seems physical (damps in time). Start with _T_ = 40 N and _ρ_ = 10 kg/m, and pick a value of _κ_ large enough to cause a noticeable effect but not so large as to stop the oscillations. As a check, reverse the sign of _κ_ and see if the wave grows in time (which would eventually violate our assumption of small oscillations).

## 21.4 Strings with Variable Tension and Density

We have derived the propagation velocity for waves on a string as  . This says that waves move slower in regions of high density and faster in regions of high tension. If the density of the string varies, for instance, by having the ends thicker in order to support the weight of the middle, then _c_ will no longer be a constant and our wave equation will need to be extended. In addition, if the density increases, so will the tension because it takes greater tension to accelerate a greater mass. If gravity acts, then we will also expect the tension at the ends of the string to be higher than in the middle because the ends must support the entire weight of the string.

To derive the equation for wave motion with variable density and tension, consider again the element of a string (Figure 21.1b) used in our derivation of the wave equation. If we do not assume the tension _T_ is constant, then Newton's second law gives

(21.29)

(21.30)

(21.31)

If _ρ_ ( _x_ ) and _T_ ( _x_ ) are known functions, then these equations can be solved with just a small modification of our algorithm.

In Section 21.4.1, we will solve for the tension in a string as a result of gravity. Readers interested in an _alternate easier problem_ that still shows the new physics may assume that the density and tension are proportional:

(21.32)

While we would expect the tension to be greater in regions of higher density (more mass to move and support), being proportional is clearly just an approximation. Substitution of these relations into (21.31) yields the new wave equation:

(21.33)

Here _c_ is a constant that would be the wave velocity if _α_ = 0. This equation is similar to the wave equation with friction; only now the first derivative is with respect to _x_ and not _t._ The corresponding difference equation follows from using central-difference approximations for the derivatives:

(21.34)

### 21.4.1 Waves on Catenary

Up until this point we have been ignoring the effect of gravity upon our string's shape and tension. This is a good approximation if there is very little sag in the string, as might happen if the tension is very high and the string is light. Even if there is some sag, our solution for _y_ ( _x_ , _t_ ) could still be used as the disturbance about the equilibrium shape. However, if the string is massive, say, like a chain or heavy cable, then the sag in the middle caused by gravity could be quite large (Figure 21.5), and the resulting variation in shape and tension needs to be incorporated into the wave equation. Because the tension is no longer uniform, waves travel faster near the ends of the string, which are under greater tension because they must support the entire weight of the string.

### 21.4.2 Derivation of Catenary Shape

Consider a string of uniform density _ρ_ acted upon by gravity. To avoid confusion with our use of _y_ ( _x_ ) to describe a disturbance on a string, we call _u_ ( _x_ ) the equilibrium shape of the string (Figure 21.5). The statics problem we need to solve is to determine the shape _u_ ( _x_ ) and the tension _T_ ( _x_ ). The inset in Figure 21.5 is a free-body diagram of the midpoint of the string and shows that the weight _W_ of this section of arc length _s_ is balanced by the vertical component of the tension _T._ The horizonal tension _T_ 0 is balanced by the horizontal component of _T_ :

(21.35)

**Figure 21.5** (a) A uniform string suspended from its ends in a gravitational field assumes a catenary shape. (b) A force diagram of a section of the catenary at its lowest point. Because the ends of the string must support the entire weight of the string, the tension now varies along the string.

(21.36)

The trick is to convert (21.36) to a differential equation that we can solve. We do that by replacing the slope tan _θ_ by the derivative d _u_ /d _x_ , and taking the derivative with respect to _x_ :

(21.37)

Yet because  , we have our differential equation

(21.38)

(21.39)

where _D_ is a combination of constants with the dimension of length. Equation 21.38 is the equation for the _catenary_ and has the solution (Becker, 1954)

(21.40)

Here we have chosen the _x_ -axis to lie a distance _D_ below the bottom of the catenary (Figure 21.5) so that _x_ = 0 is at the center of the string where _y_ = _D_ and _T_ = _T_ 0. Equation 21.37 tells us the arc length _s_ = _D_ d _u_ /d _x_ , so we can solve for _s_ ( _x_ ) and for the tension _T_ ( _x_ ) via (21.35):

(21.41)

It is this variation in tension that causes the wave velocity to change for different positions on the string.

### 21.4.3 Catenary and Frictional Wave Exercises

We have given you the program `**EqStringAnimate.py**` (Listing 21.1) that solves the wave equation. Modify it to produce waves on a catenary including friction, or for the assumed density and tension given by (21.32) with _α_ = 0.5, _T_ 0 = 40 N, and _ρ_ 0 = 0.01 kg/m. (The instructor's site contains the programs `**CatFtiction.py**` and `**Catstring.py**` that do this.)

  1. Look for some interesting cases and create surface plots of the results.
  2. Describe in words how the waves dampen and how a wave's velocity appears to change.
  3. _Normal modes:_ Search for normal-mode solutions of the variable-tension wave equation, that is, solutions that vary as

(21.42)

Try using this form to start your program and see if you can find standing waves. Use large values for _ω_.
  4. When conducting physics demonstrations, we set up standing-wave patterns by driving one end of the string periodically. Try doing the same with your program; that is, build into your code the condition that for all times

(21.43)

Try to vary _A_ and _ω_ until a normal mode (standing wave) is obtained.
  5. (For the exponential density case.) If you were able to find standing waves, then verify that this string acts like a high-frequency filter, that is, there is a frequency below which no waves occur.
  6. For the catenary problem, plot your results showing _both_ the disturbance _u_ ( _x, t_ ) about the catenary and the actual height _y_ ( _x, t_ ) above the horizontal for a plucked string initial condition.
  7. Try the first two normal modes for a uniform string as the initial conditions for the catenary. These should be close to, but not exactly, normal modes.
  8. We derived the normal modes for a uniform string after assuming that _k_ ( _x_ ) = _ω_ / _c_ ( _x_ ) is a constant. For a catenary without too much _x_ variation in the tension, we should be able to make the approximation

(21.44)

See if you get a better representation of the first two normal modes if you include some _x_ dependence in _k_.

## 21.5 Vibrating Membrane (2D Waves)

**Problem** An elastic membrane is stretched across the top of a square box of sides π and attached securely. The tension per unit length in the membrane is _T._ Initially, the membrane is placed in the asymmetrical shape

(21.45)

where _u_ is the vertical displacement from equilibrium. Your _problem_ is to describe the motion of the membrane when it is released from rest (Kreyszig, 1998).

The description of wave motion on a membrane is basically the same as that of 1D waves on a string discussed in Section 21.2, only now we have wave propagation in two directions. Consider Figure 21.6 showing a square section of the membrane under tension _T._ The membrane moves only vertically in the _z_ direction, yet because the restoring force arising from the tension in the membrane varies in both the _x_ and _y_ directions, there is wave motion along the surface of the membrane.

Although the tension is constant over the small area in Figure 21.6, there will be a net vertical force on the segment if the angle of incline of the membrane varies as we move through space. Accordingly, the net force on the membrane in the _z_ direction as a result of the change in _y_ is

(21.46)

where _θ_ is the angle of incline at _y_ \+ Δ _y_ and _φ_ is the angle at _y._ Yet if we assume that the displacements and the angles are small, then we can make the approximations:

(21.47)

(21.48)

**Figure 21.6** A small part of an oscillating membrane and the forces that act on it.

Similarly, the net force in the _z_ direction as a result of the variation in _y_ is

(21.49)

The membrane section has mass _ρ_ Δ _x_ Δ _y_ , where _ρ_ is the membrane's mass per unit area. We now apply Newton's second law to determine the acceleration of the membrane section in the _z_ direction as a result of the sum of the net forces arising from both the _x_ and _y_ directions:

(21.50)

(21.51)

This is the 2D version of the wave equation (21.4) that we studied previously in one dimension. Here c, the propagation velocity, is still the square root of tension over density; only now it is tension per unit length and mass per unit area.

## 21.6 Analytical Solution

The analytic or numerical solution of the partial differential equation (21.51) requires us to know both the boundary and the initial conditions. The boundary conditions hold for all times and were given when we were told that the membrane is attached securely to a square box of side π:

(21.52)

(21.53)

As required for a second-order equation, the initial conditions has two parts, the shape of the membrane at time _t_ = 0, and the velocity of each point of the membrane. The initial configuration is

(21.54)

Second, we are told that the membrane is released from rest, which means

(21.55)

where we write partial derivative because there are also spatial variations.

The analytic solution is based on the guess that because the wave equation (21.51) has separate derivatives with respect to each coordinate and time, the full solution _u_ ( _x, y, t_ ) is the product of separate functions of _x, y,_ and _t_ :

(21.56)

After substituting this into (21.51) and dividing by  , we obtain

(21.57)

The only way that the LHS of (21.57) can be true for all time while the RHS is also true for all coordinates, is if both sides are constant:

(21.58)

(21.59)

(21.60)

In (21.59) and (21.60), we have included further deduction that because each term on the RHS of (21.58) depends on either _x_ or _y_ , then the only way for their sum to be constant is if each term is a constant, in this case – _k_ 2. The solutions of these equations are standing waves in the _x_ and _y_ directions, which of course are all sinusoidal function,

(21.61)

(21.62)

(21.63)

We now apply the boundary conditions:

(21.64)

The fixed values for the eigenvalues _m_ and _n_ describing the modes for the _x_ and _y_ standing waves are equivalent to fixed values for the constants _q_ 2 and _k_ 2. Yet because  , we must also have a fixed value for _ξ_ 2:

(21.65)

The full space–time solution now takes the form

(21.66)

where _k_ and _q_ are integers. Because the wave equation is linear in _u,_ its most general solution is a linear combination of the eigenmodes (21.66):

(21.67)

While an infinite series is not a good algorithm, the initial and boundary conditions means that only the _k_ = 2, _q_ = 1 term contributes, and we have a closed form solution:

(21.68)

where _c_ is the wave velocity. You should verify that initial and boundary conditions are indeed satisfied.

**Listing 21.2** Waves2D.py solves the wave equation numerically for a vibrating membrane.

## 21.7 Numerical Solution for 2D Waves

The development of an algorithm for the solution of the 2D wave equation (21.51) follows that of the 1D equation in Section 21.2.2. We start by expressing the second derivatives in terms of central differences:

(21.69)

(21.70)

(21.71)

After discretizing the variables,  , we obtain our time-stepping algorithm by solving for the future solution in terms of the present and past ones:

(21.72)

where as before  . Whereas the present ( _k_ ) and past ( _k_ – 1) solutions are known after the first step, to initiate the algorithm we need to know the solution at _t_ = –Δ _t_ , that is, before the initial time. To find that, we use the fact that the membrane is released from rest:

(21.73)

**Figure 21.7** The standing wave pattern on a square box top at three different times.

After substituting (21.73) into (21.72) and solving for _u_ 1, we obtain the algorithm for the first step:

(21.74)

Because the displacement  . is known at time _t_ = 0 ( _k_ = 0), we compute the solution for the first time step with (21.74) and for subsequent steps with (21.72).

The program `**Wave2D.py**` in Listing 21.2 solves the 2D wave equation using the time-stepping (leapfrog) algorithm. The program `**Waves2Danal.py**` computes the analytic solution. The shape of the membrane at three different times are shown in Figure 21.7.

1) Some similar but independent studies can also be found in Rawitscher _et al._ (1996).

# 22
Wave Equations II: Quantum Packets and Electromagnetic

* * *

This chapter continues the discussion of the numerical solution of wave equations begun in Chapter 21, now to equations that require algorithms with a bit more sophistication. First, we explore quantum wave packets, which have their real and imaginary parts solved for at slightly differing (split) times. Then, we explore electromagnetic waves, which have the extra complication of being vector waves with interconnected _E_ and _H_ fields, which also get solved for at split times.

* * *

## 22.1 Quantum Wave Packets

**Problem** An experiment places an electron with a definite momentum and position in a 1D region of space the size of an atom. It is confined to that region by some kind of attractive potential. Your _problem_ is to determine the resultant electron behavior in time and space.

## 22.2 Time-Dependent Schrödinger Equation (Theory)

Because the region of confinement is the size of an atom, we must solve this problem quantum mechanically. Because the particle has both a definite momentum and position, it is best described as a wave packet, which implies that we must now solve the time-dependent Schrödinger equation for both the spatial and time dependence of the wave packet. Accordingly, this is a different problem from the bound state one of a particle confined to a box, considered in Chapters 7 and , where we solved the eigenvalue problem for stationary states of the time-independent Schrödinger equation.

We model an electron initially localized in space at _x_ = 5 with momentum _k_ 0 (  = 1 in our units) by a wave function that is a wave packet consisting of a Gaussian multiplying a plane wave:

**Figure 22.1** The position as a function of time of a localized electron confined to a square well (computed with the code `**SqWell.py**` available in the instructor's site). The electron is initially on the left with a Gaussian wave packet. In time, the wave packet spreads out and collides with the walls.

**Figure 22.2** The probability density as a function of time for an electron confined to a 1D harmonic oscillator potential well. (a) A conventional surface plot, (b) a color visualization.

(22.1)

To solve the _problem_ , we must determine the wave function for all later times. If (22.1) was an eigenstate of the Hamiltonian, its exp(−i _ωt_ ) time dependence can be factored out of the Schrödinger equation (as is usually carried out in textbooks). However,   for this _ψ_ , and so we must solve the full time-dependent Schrödinger equation. To show you where we are going, the resulting wave packet behavior is shown in Figures 22.1 and 22.2.

The time and space evolution of a quantum particle is described by the 1D time-dependent Schrödinger equation,

(22.2)

(22.3)

where we have set 2 _m_ = 1 to keep the equations simple. Because the initial wave function is complex (in order to have a definite momentum associated with it), the wave function will be complex for all times. Accordingly, we decompose the wave function into its real and imaginary parts:

(22.4)

(22.5)

(22.6)

(22.7)

where _V_ ( _x_ ) is the potential acting on the particle.

### 22.2.1 Finite-Difference Algorithm

The time-dependent Schrödinger equation can be solved with both implicit (large-matrix) and explicit (leapfrog) methods. The extra challenge with the Schrödinger equation is to establish that the integral of the probability density   remains constant (conserved) to a high level of precision for all time. For our project, we use an _explicit_ method that improves the numerical conservation of probability by solving for the real and imaginary parts of the wave function at slightly different or "staggered" times (Askar and Cakmak, 1977; Visscher, 1991; Maestri _et al.,_ 2000). Explicitly, the real part _R_ is determined at times 0, Δ _t_ ,..., and the imaginary part _I_ at 1/2Δ _t_ , 3/2Δ _t_ ,.... The algorithm is based on (what else?) the Taylor expansions of _R_ and _I_ :

(22.8)

where _α_ = Δ _t_ /2(Δ _x_ )2. In the discrete form with  , we have

(22.9)

(22.10)

where the superscript _n_ indicates the time and the subscript _i_ the position.

The probability density _ρ_ is defined in terms of the wave function evaluated at three different times:

(22.11)

Although probability is not conserved exactly with this algorithm, the error is two orders higher than that in the wave function, and this is usually quite satisfactory. If it is not, then we need to use smaller steps. While this definition of _ρ_ may seem strange, it reduces to the usual one for Δ _t_ → 0 and so can be viewed as part of the art of numerical analysis. We will ask you to investigate just how well probability is conserved. We refer the reader to Koonin (1986) and Visscher (1991) for details on the stability of the algorithm.

### 22.2.2 Wave Packet Implementation, Animation

In Listing 22.1, you will find the program `**HarmosAnimate.py**` that solves for the motion of the wave packet (22.1) inside a harmonic oscillator potential. The program `**Slit.py**` on the instructor's site solves for the motion of a Gaussian wave packet as it passes through a slit (Figure 22.3). You should solve for a wave packet confined to the square well:

  1. Define arrays `**psr[751,2]**` and `**psi[751,2]**` for the real and imaginary parts of _ψ_ , and `**Rho[751]**` for the probability. The first subscript refers to the _x_ position on the grid, and the second to the present and future times.
  2. Use the values _σ_ 0 = 0.5, Δ _x_ = 0.02, _k_ 0 = 17π, and Δ _t_ = 1/2Δ _x_ 2.

**Figure 22.3** The probability density as a function of position and time for an electron incident upon and passing through a slit. Significant reflection is seen to occur.

  3. Use (22.1) for the initial wave packet to define `**psr[j,1]**` for all _j_ at _t_ = 0 and to define `**psi[j,1]**` at _t_ = 1/2Δ _t_.
  4. Set `**Rho[1] = Rho[751] = 0.0**` because the wave function must vanish at the infinitely high well walls.
  5. Increment time by 1/2Δ _t_. Use (22.9) to compute `**psr[j,2]**` in terms of `**psr[j,1]**`, and (22.10) to compute `**psi[j,2]**` in terms of `**psi[j,1]**`.
  6. Repeat the steps through all of space, that is, for _i_ = 2−750.
  7. Throughout all of space, replace the present wave packet (second index equal to 1) by the future wave packet (second index 2).
  8. After you are sure that the program is running properly, repeat the time-stepping for ∼ 5000 steps.

**Listing 22.1** **HarmosAnimate.py** solves the time-dependent Schrödinger equation for a particle described by a Gaussian wave packet moving within a harmonic oscillator potential.

  1. _Animation:_ Output the probability density after every 200 steps for use in animation.
  2. Make a surface plot of probability vs. position vs. time. This should look like Figure 22.1 or 22.2.
  3. Make an animation showing the wave function as a function of time.
  4. Check how well the probability is conserved for early and late times by determining the integral of the probability over all of space,  , and seeing by how much it changes in time (its specific value doesn't matter because that is just normalization).
  5. What might be a good explanation of why collisions with the walls cause the wave packet to broaden and break up? ( _Hint_ : The collisions do not appear so disruptive when a Gaussian wave packet is confined within a harmonic oscillator potential well.)

**Figure 22.4** The probability density as a function of _x_ and _y_ at three different times for an electron confined to a 2D parabolic tube (infinite in the y direction). The electron is initially placed in a Gaussian wave packet in both the _x_ and _y_ directions, and it is to be noted how there is spreading of the wave packet in the _y_ direction, but not in the _x_ direction.

### 22.2.3 Wave Packets in Other Wells (Exploration)

**1D Well** Now confine the electron to a harmonic oscillator potential:

(22.12)

Take the momentum _k_ 0 = 3π, the space step Δ _x_ = 0.02, and the time step  . Note that the wave packet broadens yet returns to its initial shape!

**2D Well** Now confine the electron to a 2D parabolic tube (Figure 22.4):

(22.13)

The extra degree of freedom means that we must solve the 2D PDE:

(22.14)

Assume that the electron's initial localization is described by the 2D Gaussian wave packet:

(22.15)

Note that you can solve the 2D equation by extending the method we just used in 1D or you can look at the next section where we develop a special algorithm.

## 22.3 Algorithm for the 2D Schrödinger Equation

One way to develop an algorithm for solving the time-dependent Schrödinger equation in 2D is to extend the 1D algorithm to another dimension. Rather than that, we apply quantum theory directly to obtain a more powerful algorithm (Maestri _et al.,_ 2000). First, we note that Equation 22.14 can be integrated in a formal sense (Landau and Lifshitz, 1976) to obtain the operator solution:

(22.16)

(22.17)

where   is an operator that translates a wave function by an amount of time _t_ and   is the Hamiltonian operator. From this formal solution, we deduce that a wave packet can be translated ahead by time Δ _t_ via

(22.18)

where the superscripts denote time _t_ = _n_ Δ _t_ and the subscripts denote the two spatial variables _x_ = _i_ Δ _x_ and _y_ = _j_ Δ _y_. Likewise, the inverse of the time evolution operator moves the solution back one time step:

(22.19)

While it would be nice to have an algorithm based on a direct application of (22.19), the references show that the resulting algorithm would not be stable. That being so, we base our algorithm on an indirect application (Askar and Cakmak, 1977), namely, the relation between the difference in _ψ_ _n_ +1 and _ψ_ _n_ -1:

(22.20)

where the difference in sign of the exponents is to be noted. The algorithm derives from combining the _0_ (Δ _x_ 2) expression for the second derivative obtained from the Taylor expansion,

(22.21)

with the corresponding-order expansion of the evolution equation (22.20). Substituting the resulting expression for the second derivative into the 2D time-dependent Schrödinger equation results in1)

(22.22)

where _α_ = Δ _t_ /2(Δ _x_ )2. We convert this complex equations to coupled real equations by substituting them into the wave function _ψ_ = _R_ \+ _iI_ ,

(22.23)

(22.24)

This is the algorithm we use to integrate the 2D Schrödinger equation. To determine the probability, we use the same expression (22.11) used in 1D.

### 22.3.1 Exploration: Bound and Diffracted 2D Packet

  1. Determine the motion of a 2D Gaussian wave packet within a 2D harmonic oscillator potential:

(22.25)

  2. Center the initial wave packet at ( _x, y_ ) = (3.0, −3) and give it momentum ( _k_ 0 _x_ , _k_ 0 _y_ ) = (3.0, 1.5).
  3. Young's single-slit experiment has a wave passing through a small slit with the transmitted wave showing interference effects. In quantum mechanics, where we represent a particle by a wave packet, this means that an interference pattern should be formed when a particle passes through a small slit. Pass a Gaussian wave packet of width 3 through a slit of width 5 (Figure 22.3), and look for the resultant quantum interference.

## 22.4 Wave Packet–Wave Packet Scattering2)

We have just seen how to represent a quantum particle as a wave packet and how to compute the interaction of that wave packet/particle with an external potential. Although external potentials do exist in nature, realistic scattering often involves the interaction of one particle with another, which in turn would be represented as the interaction of a wave packet with a _different_ wave packet. We have already done the hard work needed to compute wave packet–wave packet scattering in our implementation of wave packet–potential scattering even in 2D. We now need only to generalize it a bit.

Two interacting particles are described by the time-dependent Schrödinger equation in the coordinates of the two particles

(22.26)

where, for simplicity, we assume a one-dimensional space and again set _ _= 1. Here _m i_ and _x i_ are the mass and position of particle _i_ = 1, 2. Knowledge of the two-particle wavefunction _ψ_ ( _x_ 1, _x_ 2, _t_ ) at time _t_ permits the calculation of the probability density of particle 1 being at _x_ 1 and particle 2 being at _x_ 2:

(22.27)

The fact that particles 1 and 2 must be located somewhere in space leads to the normalization constraint on the wavefunction

(22.28)

The description of a particle within a multiparticle system by a single-particle wavefunction is an approximation unless the system is uncorrelated, in which case the total wavefunction can be written in product form. However, it is possible to deduce meaningful one-particle probabilities (also denoted by _ρ_ ) from the two-particle density by integrating over the other particle:

(22.29)

Of course, the true solution of the Schrödinger equation is _ψ_ ( _x_ 1, _x_ 2, _t_ ), but we find it hard to unravel the physics in a three-variable complex function, and so will usually view _ρ_ ( _x_ 1, _t_ ) and _ρ_ ( _x_ 2, _t_ ) as two separate wave packets colliding.

If particles 1 and 2 are identical, then their total wavefunction must be symmetric (s) or antisymmetric (a) under interchange of the particles. We impose this condition on our numerical solution _ψ_ ( _x_ 1, _x_ 2), by forming the combinations

(22.30)

(22.31)

The cross term in (22.31) places an additional correlation into the wave packets.

### 22.4.1 Algorithm

The algorithm for the solution of the two-particle Schrödinger equation (22.26) in one dimension _x_ , is similar to the one we outlined for the solution for a single particle in the two dimensions _x_ and _y_ , only now with _x_ → _x_ 1 and _y_ → _x_ 2. As usual, we assume discrete space and time

(22.32)

We employ the improved algorithm for the time derivative introduced by Askar and Cakmak (1977), which uses a central difference algorithm applied to the formal solution (22.16)

(22.33)

(22.34)

where we have assumed Δ _x_ 1 = Δ _x_ 2 = Δ _x_ and formed the ratio _λ_ = Δ _t_ /Δ _x_ 2. Again we will take advantage of the extra degree of freedom provided by the complexity of the wavefunction

(22.35)

and staggered times to preserve probability better (Visscher, 1991). The algorithm (22.34) then separates into the pair of coupled equations

(22.36)

(22.37)

### 22.4.2 Implementation

We assume that the particle–particle potential is central and depends only on the relative distance between particles 1 and 2 (the method can handle any _x_ 1 and _x_ 2 functional dependences if needed). For example, we have used both a "soft" potential with a Gaussian dependence and a "hard" potential with a square-well dependence:

(22.38)

where _α_ is the range parameter and _V_ 0 is the depth parameter.

Because we are solving a PDE, we must specify both initial and boundary conditions. For scattering, we assume that particle 1 is initially at   with momentum _k_ 1, and that particle 2 is initially far away at   with momentum _k_ 2. Since the particles are initially too far apart to be interacting, we assume that the initial wave packet is a product of independent wave packets for particles 1 and 2

(22.39)

Because of the Gaussian factors here, _ψ_ is not an eigenstate of the momentum operator   for either particle 1 or 2, but instead contains a spread of momenta about the mean, initial momenta _k_ 1 and _k_ 2. If the wave packet is made very broad ( _σ_ → ∞), we would obtain momentum eigenstates, but then would have effectively eliminated the wave packets. Note that while the Schrödinger equation may separate into one equation in the relative coordinate _x_ = _x_ 1 − _x_ 2 and another in the center-of-mass coordinate _X_ = ( _m_ 1 _x_ 1 \+ _m_ 2 _x_ 2)/( _m_ 1 \+ _m_ 2), the initial condition (22.39), or more general ones, cannot be written as a product of separate functions of _x_ and _X_ , and accordingly, a solution of the partial differential equation in two variables is required (Landau, 1996).

We start the staggered-time algorithm with the real part the wavefunction (22.39) at _t_ = 0 and the imaginary part at _t_ = Δ _t_ /2. The initial imaginary part follows by assuming that Δ _t_ /2 is small enough and _σ_ is large enough for the initial time dependence of the wave packet to be that of the plane wave parts:

(22.40)

In an actual scattering experiment, a projectile starts at infinity and the scattered particles are observed also at infinity. We model that by solving our partial differential equation within a box of side _L_ that in ideal world would be much larger than both the range of the potential and the width of the initial wave packet. This leads to the boundary conditions

(22.41)

The largeness of the box minimizes the effects of the boundary conditions during the collision of the wave packets, although at large times there will be artificial collisions with the box that do not correspond to actual experimental conditions.

Some typical parameters we used are  ,  ,  , and  . The original C code is available on the Web (Maestri _et al.,_ 2000). Note that our space step size is 1/1400th of the size of the box _L_ , and l/70th of the size of the wave packet. Our time step is 1/20 000th of the total time _T_ , and l/2000th of a typical time for the wave packet. In all cases, the potential and wave packet parameters are chosen to be similar to those used in the one-particle studies by Goldberg _et al_. (1967). The time and space step sizes were determined by trial and error until values were found that provided stability and precision. Importantly, ripples during interactions found in earlier studies essentially disappear when (more accurate) small values of Δ _x_ are employed. In general, stability is obtained by making Δ _t_ small enough, with simultaneous changes in Δ _t_ and Δ _x_ made to keep _λ_ = Δ _t_ /Δ _x_ 2 constant. Total probability, as determined by a double Simpson's rule integration of (22.28), is typically conserved to 13 decimal places, impressively close to machine precision. In contrast, the mean energy, for which we do not use an optimized algorithm, is conserved only to 3 places.

### 22.4.3 Results and Visualization

We solve our problem in the center-of-momentum system by taking _k_ 2 = − _k_ 1 (particle 1 moving to larger _x_ values and particle 2 to smaller _x_ ). Because the results are time dependent, we make movies of them, and since the movies show the physics much better than the still images, we recommend that the reader look at them (Maestri _et al.,_ 2000). We first tested the procedure by emulating the one-particle collisions with barriers and wells studied by Goldberg _et al._ (1967) and presented by Schiff. We made particle 2 ten times heavier than particle 1, which means that particle 2's initial wave packet moves at 1/10th the speed of particle 1's, and so is similar to a barrier. On the left of Fig. 22.5, we see six frames from an animation of the two-particle density _ρ_ ( _x_ 1, _x_ 2, _t_ ) as a simultaneous function of the particle positions _x_ 1 and _x_ 2. On the right of Fig. 22.5 we show, for this same collision, the _single-particle_ densities _ρ_ 1( _x_ 1, _t_ ) and _ρ_ 2( _x_ 2, _t_ ) extracted from _ρ_ ( _x_ 1, _x_ 2, _t_ ) by integrating out the dependence on the other particle via (22.29). Because the mean kinetic energy equals twice the maximum height of the potential barrier, we expect complete penetration of the packets, and indeed, at time 18 we see on the right that the wave packets have large overlap, with the repulsive interaction "squeezing" particle 2 (it gets narrower and taller). During time 22–40, we see part of wave packet 1 reflecting off wave packet 2 and then moving back to smaller _x_ (the left). From times 26–55, we also see that a major part of wave packet 1 gets "trapped" inside of wave packet 2 and then leaks out rather slowly.

When looking at the two-particle density _ρ_ ( _x_ 1, _x_ 2, _t_ ) on the left of Fig. 22.5, we see that for times 1-26, the _x_ 2 position of the peak of changes very little with time, which is to be expected since particle 2 is heavy. In contrast, the _x_ 1 dependence in _ρ_ ( _x_ 1, _x_ 2, _t_ ) gets broader with time, develops into two peaks at time 26, separates into two distinct parts by time 36, and then, at time 86 after reflecting off the walls, returns to particle 2's position. We also notice in both these figures that at time 40 and thereafter, particle 2 (our "barrier") fissions into reflected and transmitted waves. As this comparison of the visualizations on the right and left of Figures 22.5 demonstrates, it seems easier to understand the physics by superimposing two single-particle densities (thereby discarding information on correlations) than by examining the two-particle density.

**Figure 22.5** (a) Six frames from an animation of the two-particle density _ρ_ ( _x_ 1, _x_ 2, _t_ )as a function of the position of particle 1 with mass _m_ and of the position of particle 2 with mass 10 _m_. (b) This same collision as seen with the single-particle densities _ρ_ ( _x_ 1, _t_ ) and _ρ_ ( _x_ 2, _t_ ). The numbers in the left-hand corners are the times in units of 100Δ _t_. Note that each plot ends at the walls of the containing box, and that particle 1 "bounces off" a wall between times 36 and 86.

**Figure 22.6** A time sequence of two single-particle wave packets scattering from each other. The particles have equal mass, a mean kinetic energy equal to a quarter of the well's depth, and the wavefunction has been symmetrized.

In Fig. 22.6, we see nine frames from the movie of an attractive _m–m_ collision in which the mean energy equals one-quarter of the well depth. The initial packets speed up as they approach each other, and at time 60, the centers have already passed through each other. After that, a transmitted and reflected wave for each packet is seen to develop (times 66–78). Although this may be just an artifact of having two particles of equal mass, from times 110–180, we see that each packet appears to capture or "pick up" a part of the other packet and move off with it.

Note in Fig. 22.6 that at time 180, the wave packets are seen to be interacting with the wall (the edges of the frames), as indicated by the interference ripples between incident and reflected waves. Also note that at time 46 and thereafter two additional small wave packets are seen to be traveling in opposite directions to the larger initial wave packets. These small packets are numerical artifacts and arise because outgoing waves satisfy the same differential equations as incoming waves.

## 22.5 E&M Waves via Finite-Difference Time Domain

* * *

Simulations of electromagnetic (EM) waves are of tremendous practical importance. Indeed, the fields of nanotechnology and spintronics rely heavily on such simulations. The basic techniques used to solve for EM waves are essentially the same as those we used for string and quantum waves: set up a grid in space and time and then step the initial solution forward in time one step at a time. When used for _E &M_ simulations, this technique is known as the _finite difference time domain_ (FDTD) method. What is new for E&M waves is that they are vector fields, with the variations of one vector field generating the other. Our treatment of FDTD does not do justice to the wealth of physics that can occur, and we recommend Sullivan (2000) for a more complete treatment and Ward _et al._ (2005) (and their Web site) for modern applications.

* * *

**Problem** You are given a region in space in which the _E_ and _H_ fields are known to have a sinusoidal spatial variation

(22.42)

(22.43)

with all other components vanishing. Determine the fields for all _z_ values at all subsequent times.

## 22.6 Maxwell's Equations

The description of EM waves via Maxwell's equations is given in many textbooks. For propagation in just one dimension ( _z_ ) and for free space with no sinks or sources, four coupled PDEs result:

(22.44)

(22.45)

(22.46)

(22.47)

**Figure 22.7** A single EM pulse traveling along the _z_ -axis. The coupled _E_ and _H_ pulses are indicated by solid and dashed curves, respectively, and the pulses at different _z_ values correspond to different times.

As indicated in Figure 22.7, we have chosen the electric field **E** ( _z_ , _t_ ) to oscillate (be polarized) in the _x_ direction and the magnetic field **H** ( _z_ , _t_ ) to be polarized in the _y_ direction. As indicated by the bold arrow in Figure 22.7, the direction of power flow for the assumed transverse electromagnetic (TEM) wave is given by the right-hand rule applied to **E × H**. Note that although we have set the initial conditions such that the EM wave is traveling in only one dimension ( _z_ ), its electric field oscillates in a perpendicular direction ( _x_ ), and its magnetic field oscillates in yet a third direction ( _y_ ); so while some may call this a 1D wave, the vector nature of the fields means that the wave occupies all three dimensions.

## 22.7 FDTD Algorithm

We need to solve the two coupled PDEs (22.46) and (22.47) appropriate for our problem. As is usual for PDEs, we approximate the derivatives via the central-difference approximation, here in both time and space. For example,

(22.48)

(22.49)

We next substitute the approximations into Maxwell's equations and rearrange the equations into the form of an algorithm that advances the solution through time. Because only first derivatives occur in Maxwell's equations, the equations are simple, although the electric and magnetic fields are intermixed.

As introduced by Yee (Yee, 1966), we set up a space–time lattice (Figure 22.8) in which there are half-integer time steps as well as half-integer space steps. The magnetic field will be determined at integer time sites and half-integer space sites (open circles), and the electric field will be determined at half-integer time sites and integer space sites (filled circles). While this is an extra level of complication, the transposed lattices do lead to an accurate and robust algorithm. Because the fields already have subscripts indicating their vector nature, we indicate the lattice position as superscripts, for example,

**Figure 22.8** The algorithm for using the known values of _E_ x and _H_ y at three earlier times and three different space positions to obtain the solution at the present time. Note that the values of _E_ x are determined on the lattice of filled circles, corresponding to integer space indices and half-integer time indices. In contrast, the values of _H_ y are determined on the lattice of open circles, corresponding to half-integer space indices and integer time indices.

(22.50)

Maxwell's equations (22.46) and (22.47) now become the discrete equations

To repeat, this formulation solves for the electric field at integer space steps ( _k_ ) but half-integer time steps ( _n_ ), while the magnetic field is solved for at half-integer space steps but integer time steps.

We convert these equations into two simultaneous algorithms by solving for _E x_ at time _n_ \+ 1/2, and _H y_ at time _n_ :

(22.51)

(22.52)

The algorithms must be applied simultaneously because the space variation of _H_ _y_ determines the time derivative of _E_ _x_ , while the space variation of _E_ _x_ determines the time derivative of _H_ _y_ (Figure 22.8). These algorithms are more involved than our usual time-stepping ones in that the electric fields (filled circles in Figure 22.8) at future times _t_ = _n_ \+ 1/2 are determined from the electric fields at one time step past _t_ = _n_ − 1/2, and the magnetic fields at half a time step past _t_ = _n_. Likewise, the magnetic fields (open circles in Figure 22.8) at future times _t_ = _n_ \+ 1 are determined from the magnetic fields at one time step past _t_ = _n_ , and the electric field at half a time step past _t_ = _n_ \+ 1/2. In other words, it is as if we have two interleaved lattices, with the electric fields determined for half-integer times on lattice 1 and the magnetic fields at integer times on lattice 2.

Although these half-integer times appear to be the norm for FDTD methods (Taflove and Hagness , 1989; Sullivan, 2000), it may be easier for some readers to understand the algorithm by doubling the index values and referring to even and odd times:

(22.53)

(22.54)

This makes it clear that _E_ is determined for even space indices and odd times, while _H_ is determined for odd space indices and even times.

We simplify the algorithm and make its stability analysis simpler by renormalizing the electric fields to have the same dimensions as the magnetic fields,

(22.55)

The algorithms (22.51) and (22.52) now become

(22.56)

(22.57)

(22.58)

Here, _c_ is the speed of light in vacuum and _β_ is the ratio of the speed of light to grid velocity Δ _z_ /Δ _t_.

The space step Δ _z_ and the time step Δ _t_ must be chosen so that the algorithms are stable. The scales of the space and time dimensions are set by the wavelength and frequency, respectively, of the propagating wave. As a minimum, we want at least 10 grid points to fall within a wavelength:

(22.59)

The time step is then determined by the Courant stability condition (Taflove and Hagness , 1989; Sullivan, 2000) to be

(22.60)

As we have seen before, (22.60) implies that making the time step smaller improves precision and maintains stability, but making the space step smaller must be accompanied by a simultaneous decrease in the time step in order to maintain stability (you should check this).

**Listing 22.2** **FDTD.py** solves Maxwell's equations via FDTD time stepping (finite-difference time domain) for linearly polarized wave propagation in the z direction in free space.

**Figure 22.9** The _E_ field (light) and the _H_ field (dark) at the initial time (a) and at a later time (b). Periodic boundary conditions are used at the ends of the spatial region, which means that the large _z_ wave continues into the _z_ = 0 wave.

### 22.7.1 Implementation

In Listing 22.2, we provide a simple implementation of the FDTD algorithm for a _z_ lattice of 200 sites and in Figure 22.9 we show some results. The initial conditions correspond to a sinusoidal variation of the _E_ and _H_ fields for all _z_ values in for 0 ≤ _z_ ≤ 200:

(22.61)

The algorithm then steps out in time for as long as the user desires. The discrete form of Maxwell equations used are:

where 1 ≤ _k_ ≤ 200, and beta is a constant. The second index takes the values 0 and 1, with 0 being the old time and 1 the new. At the end of each iteration, the new field throughout all of space becomes the old one, and a new one is computed. With this algorithm, the spatial endpoints `****k=0****` and `****k=xmax-1****` remain undefined. We define them by assuming periodic boundary conditions:

### 22.7.2 Assessment

  1. Impose boundary conditions such that all fields vanish on the boundaries. Compare the solutions so obtained to those without explicit conditions for times less than and greater than those at which the pulses hit the walls.
  2. Examine the stability of the solution for different values of Δ _z_ and Δ _t_ and thereby test the Courant condition (22.60).
  3. Extend the algorithm to include the effect of entering, propagating through, and exiting a dielectric material placed within the _z_ integration region.
  4. Ensure that you see both transmission and reflection at the boundaries.
  5. Investigate the effect of varying the dielectric's index of refraction.
  6. The direction of propagation of the pulse is in the direction of **E × H** , which depends on the relative phase between the **E** and **H** fields. (With no initial **H** field, we obtain pulses both to the right and the left.)
  7. Modify the program so that there is an initial **H** pulse as well as an initial **E** pulse, both with a Gaussian times a sinusoidal shape.
  8. Verify that the direction of propagation changes if the **E** and **H** fields have relative phases of 0 or π.
  9. Investigate the resonator modes of a wave guide by picking the initial conditions corresponding to plane waves with nodes at the boundaries.
  10. Investigate standing waves with wavelengths longer than the size of the integration region.
  11. Simulate unbounded propagation by building in periodic boundary conditions into the algorithm.
  12. Place a medium with periodic permittivity in the integration volume. This should act as a frequency-dependent filter, which does not propagate certain frequencies at all.

### 22.7.3 Extension: Circularly Polarized Waves

We now extend our treatment to EM waves in which the **E** and _H_ fields, while still transverse and propagating in the _z_ direction, are not restricted to linear polarizations along just one axis. Accordingly, we add to (22.46) and (22.47):

(22.62)

(22.63)

When discretized in the same way as (22.51) and (22.52), we obtain

(22.64)

(22.65)

To produce a circularly polarized traveling wave, we set the initial conditions

(22.66)

(22.67)

We take the phases to be   so that their difference  which leads to circular polarization. We include the initial conditions in the same manner as we did the Gaussian pulse, only now with these cosine functions.

Listing 22.3 gives our implementation `****EMcirc.py****` for waves with transverse two-component **E** and **H** fields. Some results of the simulation are shown in Figure 22.10, where you will note the difference in phase between **E** and **H**.

**Figure 22.10** _E_ and _H_ fields at _t_ = 100 for a circularly polarized wave in free space.

**Listing 22.3** **CircPolartzn.py** solves Maxwell's equations via FDTD time-stepping for circularly polarized wave propagation in the z direction in free space.

## 22.8 Application:Wave Plates

**Problem** Develop a numerical model for a wave plate that convert a linearly polarized EM wave into a circularly polarized one.

As can be seen in Figure 22.11a wave plate is an optical device that alters the polarization of light traveling through it by shifting the relative phase of the components of the polarization vector. A quarter-wave plate introduces a relative phase of _λ_ /4, where _λ_ is the wavelength of the light. Physically, a wave plate is often a birefringent crystal in which the different propagation velocities of waves in two orthogonal directions leads to the phase change. The amount of phase change is adjusted by varying the thickness of the plate.

**Figure 22.11** One frame from the program `****quarterwave.py****` (on Instructor's site) showing a linearly polarized EM wave entering a quarter-wave plate from the left and leaving as a circularly polarized wave on the right (the arrow on the left oscillates back and forth at 45° while the one on the right rotates).

To attack this problem, we apply our FDTD method of solving Maxwell equations. We start with a linear polarized wave with both _E_ _x_ and _E_ _y_ components propagating along the _z_ direction. The wave enters the plate and emerges from it still traveling in the _z_ direction, but now with the relative phase of these fields shifted. Of course, because this is an EM wave, there will also be coupled magnetic field components present, in this case _H_ _x_ and _H_ _y_ , and they too will need be computed.

**Theory** Maxwell equations for a wave propagating along the z-axis are:

(22.68)

(22.69)

We take as initial conditions a wave incident from the left along the _z_ -axis, linearly polarized (electric field direction of 45°), with corresponding, and perpendicular, H components:

(22.70)

(22.71)

Because only the relative phases matter, we simplify the calculation by assuming that the _E y_ and _H_ _x_ components do not have their phases changed, but that the _E_ _x_ and _H_ _y_ components do (in this case by _λ_ /4 when they leave the plate). Of course, after leaving the plate and traveling in free space, there are no further changes in the relative phase.

## 22.9 Algorithm

As in Section 22.7 and Figure 22.8, we follow the FDTD approach of using known values of _E_ _x_ and _H_ _y_ at three earlier times and three different space positions to obtain the solution at the present time. With the renormalized electric fields as in (22.55), this leads to the beautifully symmetric equations:

(22.72)

(22.73)

(22.74)

(22.75)

## 22.10 FDTD Exercise and Assessment

  1. Modify the FDTD program of Listing 22.2 so that it solves the algorithm (22.72)–(22.75). Use _β_ = 0.01.
  2. After each time step, impose a gradual increment of the phase so that the total phase change will be one-quarter of a wavelength. Our program for this, `****quarterplat.py****`, is on the instructor's page.
  3. Verify that the plate converts an initially linearly polarized wave into a circularly polarized one.
  4. Verify that the plate converts an initially circularly polarized wave into a linearly polarized one.
  5. What happens if you put two plates together? Three? Four? (Verify!)

1) For reference sake, note that the constants in the equation change as the dimension of the equation changes; that is, there will be different constants for the 3D equation, and therefore our constants are different from the references! 2) This section is based on the Master of Science thesis of Jon Maestri. It is included in his memory.

# 23
Electrostatics via Finite Elements

* * *

In Chapter 19, we discussed the simple, but powerful, _finite-differences_ method for solving Poisson's and Laplace's equations on a lattice in space. In this chapter, we provide a basic outline of the _finite-element_ method (FEM) for solving PDEs. Our usual approach to solving PDEs uses the _finite-difference_ method to approximate various derivatives in terms of the finite differences of a function evaluated upon a fixed grid. The FEM, in contrast, breaks space up into multiple geometric objects (elements), determines an approximate form for the solution appropriate to each element, and then matches the solutions up at the elements′edges. The FEM is ultimately more efficient and powerful than the finite-differences method; however, much more work is required to derive the algorithm. In practice, it is rare to solve a PDE from scratch by deriving the FEM for a particular problem. Rather, and for good reasons, many FEM applications use highly developed FEM packages that get customized for an individual problem. (Python's finite element library is _FiPy_.) Our aim is to give the reader some basic understanding of the FEM, not to develop a practitioner. Accordingly, we examine a 1D problem in some detail, and then outline the similar steps followed for the same equation extended to 2D.

* * *

## 23.1 Finite-Element Method

The theory and practice of FEM as a numerical method for solving partial differential equations have been developed over the last 30 years and is still an active field of research. One of the theoretical strengths of FEM is that its mathematical foundations allow for elegant proofs of the convergence of its solutions. One of the practical strengths of FEM is that it offers great flexibility for problems on irregular domains, or for problems with highly varying conditions or even singularities. Although finite-difference methods are simpler to implement than FEM, they are less robust mathematically and for big problems less efficient in terms of computer time. Finite elements, in turn, are more complicated to implement, but more appropriate and precise for complicated equations and complicated geometries. In addition, the same basic finite-element technique can be applied to many problems with only minor modifications, and yields solutions that may be evaluated throughout all space, not just on a grid. In fact, the FEM with various preprogrammed multigrid packages has very much become the standard for large-scale engineering applications. Our discussion is based upon Shaw (1992); Li (2014); Otto (2011).

## 23.2 Electric Field from Charge Density (Problem)

As shown in Figure 23.1, you are given two conducting plates a distance _b_ − _a_ apart, with the lower one kept at potential _U a_, the upper plate at potential _U b_, and a uniform charge density _ρ_ ( _x_ ) placed between them. Your _problem_ is to compute the electric potential between the plates.

## 23.3 Analytic Solution

The relation between charge density _ρ_ ( _x_ ) and potential _U_ ( _x_ ) is given by Poisson's equation (19.6). For our problem, the potential _U_ changes only in the _x_ direction, and so the PDE becomes the ODE:

(23.1)

where we have set _ρ_ ( _x_ ) = 1/4π to simplify the programming. The solution we want is subject to the Dirichlet boundary conditions:

(23.2)

(23.3)

Although, we know the analytic solution (23.3), we shall develop the FEM for solving the ODE as if it was a PDE (it would be in 2D), and as if we did not know the solution. Although we will not demonstrate it, this method works equally well for any charge density _ρ_ ( _x_ ).

**Figure 23.1** A finite element solution to Laplace's equation for two metal plates with a charge density between them. The dots are the nodes _x i_, and the lines connecting the nodes are the finite elements.

## 23.4 Finite-Element (Not Difference) Methods, 1D

In an FEM, the domain in which the PDE is solved is split into finite subdomains, called _elements_ , and a _trial solution_ to the PDE in each subdomain is hypothesized. Then the parameters of the trial solution are adjusted to obtain a _best fit_ (in the sense of Chapter 7) to the exact solution. Essentially, this approach converts a given PDE into an integral equation known as the _weak_ or _variational_ form ("weak" because there is no longer the requirement that the second derivative of the solution exists). A trial solution on each element is then postulated, and this leads to the numerically intensive work of finding the best values for the parameters in the trial solution, and in matching up the various trial solutions from different subdomains.

In general, an FEM solution follows six steps (Li, 2014):

  1. Derivation of a _weak form_ of the PDE. This is equivalent to a least-squares minimization of the integral of the difference between the approximate and exact solutions.
  2. Discretization of the computational domains.
  3. Generation of interpolating or trial functions.
  4. Conversion of the "weak form" integral equation into a set of linear equations.
  5. Implementation of the boundary conditions.
  6. Solution of the resulting linear system of equations.

### 23.4.1 Weak Form of PDE

Finite-difference methods yield an approximate solution of an approximate PDE. Finite-element methods yield the best possible global agreement between an approximate solution and the exact solution. We start the FEM with the differential equation we want to solve,

(23.4)

We form an integral of the product of the exact solution _U_ ( _x_ ) and the approximate solution or _trial solution_ _Ф_ ( _x_ ) over the solution domain. This will be used as a measure of overall agreement between the two solutions. We assume that the trial vanishes at the endpoints, _Ф_ ( _a_ ) = _Ф_ ( _b_ ) = 0 (we satisfy general boundary conditions later). We next multiply both sides of the differential equation (23.1) by _Ф_ and integrate by parts from _a_ to _b_ :

(23.5)

(23.6)

(23.7)

(23.8)

Equation 23.8 is the _weak_ form of the PDE, "weak" in the sense that it does not require the existence of the second derivative of _U_ , or the continuity of _ρ_. Because the approximate and exact solutions are related by the integral of their difference over the entire domain, the algorithm provides the global best agreement between the two.

### 23.4.2 Galerkin Spectral Decomposition

The approximate solution to the weak PDE proceeds via three steps. First, we split the full domain of the PDE into subdomains called _elements_ , then we find approximate solutions within each element, and finally we match the elemental solutions onto each other. For our 1D problem, the subdomain elements are straight lines of equal length, while for the 2D problem to be considered soon, the elements are triangles (Figure 23.4).

The critical step in the FEM is the expansion of the solution _U_ in terms of a set of basis functions _φ_ _i_ :

(23.9)

Even when the basis functions are not sines or cosines, this expansion is still called a _spectral_ decomposition. We will choose _φ i_'s that are convenient for computation, and so the solution reduces to determining the unknown expansion coefficients _α j_. Later, in order to satisfy the boundary conditions, we will add an additional term to this expansion.

Considerable study has gone into determining the effectiveness of different basis functions _φ i_ that are used to represent the solution on the finite elements. If the sizes of the finite elements are made sufficiently small, then good accuracy is obtained with simple piecewise-continuous basis functions _φ i_. For our 1D problem, we use finite _elements_ that are line segments between _x i_ and _x_ _i_ +1, and we use _basis functions_ , representing the solution on each line segment, that have the form of triangle or "hats" between _x_ _i_ −1 and _x_ _i_ +1 (Figure 23.2). We also require that each basis function equals 1 at the _x_ _i_ vertex, _φ i_( _x i_) = 1:

(23.10)

**Figure 23.2** Basis functions used in finite-elements solution of the 1D Laplace equation. (a) A set of overlapping basis functions _φ i_. Each function is a triangle from _x_ _i_ −1 to _x_ _i_ +1. (b) A Piecewise-linear function. (c) A piecewise-quadratic function.

Because this choice means that each basis function equals 0 or 1 at the nodes,

(23.11)

the values of the expansion coefficients _α i_ must equal the values of the (still unknown) solution at the nodes:

(23.12)

(23.13)

Equation 23.13 makes it clear that the expansion in terms of basis functions is essentially an interpolation between the actual solution at the nodes.

### 23.4.2.1 Solution via Linear Equations

Because the basis functions _φ i_ in (23.9) are known, solving for _U_ ( _x_ ) involves determining the coefficients _α j_, which, as we just said, are the unknown values of the true solution _U_ ( _x_ ) on the nodes. We determine those values by substituting the expansions for _U_ ( _x_ ) and _Ф_ ( _x_ ) into the weak form of the PDE (23.8). This converts the integral equation into a set of simultaneous linear equations. As discussed in Chapter 6, there is a standard matrix form for a set of linear equations,

(23.14)

Our equations fit that, with _y_ a vector of unknowns, and where we still need to specify the known _stiffness matrix_ **A** and the known load matrix **_b_**. To that end, we substitute the expansion   into the weak form (23.8) to obtain:

By successively selecting   we obtain _N_ simultaneous linear equations for the unknown _α j_'s:

(23.15)

We factor out the unknown _α j_'s and write the equations out explicitly:

Because we have chosen the _φ i_'s to be the simple hat functions, the derivatives are easy to evaluate analytically (for other bases they can be carried out numerically):

(23.16)

The integrations are now fairly simple:

(23.17)

(23.18)

(23.19)

We rewrite these equations in the standard matrix form (23.14) with _y_ constructed from the unknown _α j_'s, and the tridiagonal stiffness matrix **A** constructed from the integrals over the derivatives:

(23.20)

(23.21)

The elements in **A** are just combinations of inverse step sizes and so do not change for different charge densities _ρ_ ( _x_ ). This is part of what makes FEM so efficient once set up. The elements in _**b**_ do change for different _ρ_ 's, but the required integrals can be performed analytically or with Gaussian quadrature (Chapter 5). Once **A** and _**b**_ are computed, highly efficient methods from a linear algebra package are used to solve the matrix equations for _y_ , and thus the expansion coefficients _α j_.

### 23.4.2.2 Dirichlet Boundary Conditions

Because the basis functions vanish at the endpoints, a solution expanded in them must also vanishes there. This will not do in general, and so we must add to our general solution _U_ ( _x_ ), a particular solution _U aφ0_( _x_ ) that satisfies the boundary conditions (Li, 2014):

(23.22)

where _U a_ = _U_ ( _x a_). We substitute _U_ ( _x_ ) − _U aφ0_( _x_ ) into the weak form of the PDE to obtain ( _N_ \+ 1) simultaneous equations, still of the form **_Ay_** = **_b_** , but now with

(23.23)

This is equivalent to adding a unit element to **A** and adding a new load vector element:

(23.24)

To impose the boundary condition at _x_ = _b_ , we again add a particular solution _U_ _b_ _φ_ _N_ −1( _x_ ) and substitute it into the weak form to obtain

(23.25)

So now we need to solve the linear equations **_Ay_** = **_b'_**. For 1D problems, 100–1000 equations are common, while for 3D problems there may be millions. Because the number of calculations varies approximately as _N_ 2, it is important to utilize an efficient and accurate algorithm, because otherwise the round-off error can easily dominate the solution.

## 23.5 1D FEM Implementation and Exercises

In Listing 23.1, we give our program `LaplaceFEM_1D.py` that determines the 1D FEM solution, and in Figure 23.3 we show that solution. We see on the left of the figure that three elements do not provide even visual agreement with the analytic result, whereas _N_ = 11 elements do.

  1. Examine the FEM solution for the choice of parameters

(23.26)

  2. Generate your own triangulation by assigning explicit _x_ values at the nodes over the interval [0, 1].
  3. Start with _N_ = 3 and solve the equations for _N_ values up to 1000.
  4. Examine the stiffness matrix **A** and ensure that it is triangular.
  5. Verify that the integrations used to compute the load vector **_b_** are accurate.
  6. Verify that the solution of the linear equation **_Ay_** = **_b_** is correct.
  7. Plot the numerical solution for _U_ ( _x_ ) for _N_ = 10, 100, and 1000, and compare with the analytic solution.

**Figure 23.3** Exact (line) vs. FEM solution (points) for the two-plate problem for _N_ = 3 and _N_ = 11 finite elements ( _N_ = 3 displaced upwards for clarity). On this scale, the _N_ = 11 solution IS identical to the exact one.

  8. The log of the relative global error (number of significant figures) is

(23.27)

Plot the global error vs. _x_ for _N_ = 10, 100, and 1000.

**Listing 23.1** **LaplaceFEM_1D.py** provides an FEM solution of the 1D Laplace equation via a Galerkin spectral decomposition. The resulting matrix equations are solved with Matplotlib.

### 23.5.1 1D Exploration

  1. Modify your program to use piecewise-quadratic functions for interpolation, and compare the results obtained to those obtained with the linear functions.
  2. Explore the resulting electric potential and check that the charge distribution between the plates has the explicit _x_ dependence

(23.28)

## 23.6 Extension to 2D Finite Elements

The steps followed to derive the 2D finite elements method are similar to those for the 1D method, with the big difference being that the finite elements are now 2D triangles as opposed to 1D lines. Figure 23.4 shows how an arbitrarily shaped domain might be decomposed into triangles. Although life is simpler if all the finite elements are of the same size and shape, this is not necessary, and, indeed, as we have shown in the figure, higher precision and faster run times maybe obtained by picking smaller domains in regions where the solution is known to vary rapidly, and picking larger domains in regions of slow variation. As you can imagine, 2D and 3D FEMs can get rather complicated, but not to worry, we will just outline the 2D method and refer the interested reader to Polycarpou (2006) and Reddy (1993) for fuller discussions.

**Figure 23.4** (a) Decomposition of a domain into triangular elements. Smaller triangles are used in regions of rapid variation and larger triangles are used in regions of slow variation. Discretization errors occur at boundaries. (b) A decomposition of a rectangular domain into 32 right triangles on a mesh with 25 nodes.

We extend our previous 1D method to solve the 2D version of the Laplace equation (19.4),

(23.29)

There are now 2D Dirichlet boundary conditions:

(23.30)

Here, _h_ is the height and _w_ is the width of the rectangular domain in which we desire a solution. Because our problem domain is now rectangular, it is easy to divide it into right triangles, as we have shown in Figure 23.4b.

### 23.6.1 Weak Form of PDE

For the 2D problem, the weak form of the PDE again follows from multiplying both sides of the PDE by the trial solution _Φ_ , and then integrating (Polycarpou, 2006):

(23.31)

Here, _Ω_ is a surface boundary of the domain in which we seek a solution, _Γ_ is a perimeter around the surface, _U_ is the solution of the PDE, and _n x_ and _n y_ are outward-facing unit normal to _Γ_. For Dirichlet boundary conditions the contribution of the line integral on the RHS vanishes.

### 23.6.2 Galerkin's Spectral Decomposition

As in the 1D method, the approximate solution _U_ ( _x_ , _y_ ) is expanded in a set _φ i_( _x_ , _y_ ) of basis functions, in this case 2D functions:

(23.32)

After setting _U_ = _φ j_ for _j_ = 1, 2,... , _N_ − 1, the weak form of the PDE becomes a set of linear equations:

(23.33)

We rewrite these equations in the standard matrix form (23.14) for linear equations:

(23.34)

Here the _U i_'s are the vectors of unknowns, the _stiffness matrix_ elements are

(23.35)

and the load vector **_b_** is given by (23.20).

### 23.6.3 Triangular Elements

Triangular elements are often used in 2D FEM because they can be fit into many arbitrary geometries with little overlap and with little discretization error at the boundary edges (see Figure 23.4). As we see in Figure 23.5a, we take these elements to be triangles of arbitrary shape, with the CCW arrow indicating the direction in which the nodes are numbered. While it is easier to fit arbitrary shaped triangles into a general region, it is easier to do mathematics with right triangles, such as the master triangle shown in Figure 23.5b. The latter have their orthogonal sides lying along the   and _η_ axes, with the ( _x_ , _y_ ) and ( , _η_ ) coordinates related by a linear coordinate transformation.

These master triangles are the linear interpolation functions in the   and _η_ variables that are used in 2D FEM. For example, the linear function for node 1 has the form

(23.36)

The constants are determined by evaluating the functions at each node, for example,

(23.37)

**Figure 23.5** (a) Linear triangular elements in the _x_ – _y_ plane. (b) Linear triangular element (master element) in the  _η_ plane.

(23.38)

(23.39)

Similar evaluations at the other nodes yield (Polycarpou, 2006):

(23.40)

With these interpolation functions in hand, it is possible to express the _x_ and _y_ coordinates of any point inside an element in terms of the master coordinates:

(23.41)

(23.42)

(23.43)

Next, we take these discrete forms for the interpolation functions, return to the Galerkin spectral decomposition, and use (23.35) to evaluate the **A** matrix. The required derivatives are evaluated using the chain rule:

(23.44)

(23.45)

We write these equations in the matrix form as

(23.46)

where the 2 × 2 matrix that defines the coordinate transformation between the ( _x_ , _y_ ) and the ( , _η_ ) derivatives is called the _Jacobian_ matrix _J_. After substitution of the explicit forms for the _φ_ 's, the Jacobian takes the simple form:

(23.47)

Likewise, the derivatives in the **A** matrix can be expressed in terms of the _x_ and _y_ derivatives by using the inverse of the Jacobain matrix:

(23.48)

(23.49)

Continued evaluation of the Galerkin matrix elements yields

(23.50)

(23.51)

After similar evaluations for _φ_ 2 and _φ_ 3, we obtain the six needed derivatives:

(23.52)

(23.53)

(23.54)

### 23.6.4 Solution as Linear Equations

The final evaluations of the stiffness matrix elements are made using the   and _η_ coordinates, for example,

(23.55)

These elements are found to form a symmetric matrix with values:

(23.56)

(23.57)

(23.58)

Next we evaluates the coordinate transformations:

(23.59)

(23.60)

After substituting for   and _η_ , we are left with the desired interpolation functions expressed in terms of just _x_ and _y_.

### 23.6.5 Imposing Boundary Conditions

The procedure to impose Dirichlet's boundary conditions for the 2D case is essentially the same as that for the 1D case (Section 23.4.2.2), with it now applied to all nodes that lie on the boundary _Γ_.

**Listing 23.2** The code **LaplaceFEM_2D.py** solves the 2D Laplace equation using a finite elements method.

### 23.6.6 FEM 2D Implementation and Exercise

As shown in Figure 23.4b, our application of 2D FEM has the solution domain covered by a collection of triangular elements. Each triangle in the mesh is numbered, in this case from 1 to 32. In addition, the three vertices of each triangle are numbered in a counter-clockwise direction from 1 to 3. Next, each node in the mesh (the dark circles in Figure 23.4 where lines intersect) is numbered, in this case from 1 to 25. Accordingly, the stiffness matrix **A** in (23.34) has dimension 25 × 25, while the load vector **_b_** has dimension 25 × 1.

Listing 23.2 presents our implementation of the 2D FEM solution to the 2D Laplace equation, based on the Matlab code of Polycarpou (2006). It utilizes 800 elements and 441 nodes. The output of this code is essentially the same as our solution to the same problem using the finite-differences method.

### 23.6.7 FEM 2D Exercises

  1. Examine the effect of varying the domain height and width, as well as the number of elements.
  2. Compare this numerical solution to the analytic one (the Fourier series in Section 19.3) and determine how the precision changes as the number of elements is varied.
  3. Modify the program so that it solves the parallel plate capacitor problem and compare to the finite-difference solution.

# 24
Shocks Waves and Solitons

* * *

In the first part of this chapter, we extend the discussion of waves in Chapters 21 and  by progressively including nonlinearities, dispersion, and hydrodynamic effects. We end up with the Korteweg–de Vries equation and shallow-water solitons. In the second part of this chapter, we explore the inclusion of related nonlinear physics for the pendulum chain, and end up with the Sine–Gordon equation and solitons in solids.

* * *

## 24.1 Shocks and Solitons in Shallow Water

In 1834, J. Scott Russell (Russell, 1944) observed on the Edinburgh–Glasgow canal (repeated recently in Figure 24.1a):

> _I was observing the motion of a boat which was rapidly drawn along a narrow channel by a pair of horses, when the boat suddenly stopped – not so the mass of water in the channel which it had put in motion; it accumulated round the prow of the vessel in a state of violent agitation, then suddenly leaving it behind, rolled forward with great velocity, assuming the form of a large solitary elevation, a rounded, smooth and well-defined heap of water, which continued its course along the channel apparently without change of form or diminution of speed. I followed it on horseback, and overtook it still rolling on at a rate of some eight or nine miles an hour, preserving its original figure some thirty feet long and a foot to a foot and a half in height. Its height gradually diminished, and after a chase of one or two miles I lost it in the windings of the channel. Such, in the month of August 1834, was my first chance interview with that singular and beautiful phenomenon..._

Russell also noticed that an initial, arbitrary waveform set in motion in the channel evolves into two or more waves that move at different velocities and progressively move apart until they form individual solitary waves. In Figure 24.2b, we see a single step-like wave breaking up into approximately eight of these solitary waves (now called _solitons_ ). These eight solitons occur so frequently that some consider them the equivalent of normal modes for nonlinear systems. Russell went on to produce these solitary waves in a laboratory and to empirically deduced that their speed _c_ is related to the depth _h_ of the water in the canal and to the amplitude _A_ of the wave by

**Figure 24.1** (a) A re-creation on the Union Canal near Edinburgh of Russel's soliton (www.ma.hw.ac.uk/solitons/press.html, Nature v. 376, 3 August 1995, p. 373). (b) Two shallow-water solitary waves crossing each other computed with the code `**Soliton.py**`. The taller soliton on the left catches up with and overtakes the shorter one at _t_   5. The waves resume their original shapes after the collision.

(24.1)

where _g_ is the acceleration as a result of gravity. Equation 24.1 implies an effect not found in linear systems, namely, that waves with greater amplitudes _A_ travel faster than those with smaller amplitudes. Observe that this is similar to the formation of shock waves, but different from _dispersion_ in which waves of different wavelengths have different velocities. The dependence of _c_ on the amplitude _A_ is illustrated in Figure 24.2, where we see a taller soliton catching up with and passing through a shorter one.

**Problem** Explain Russell's observations and see if they relate to the formation of _tsunamis_. The latter are ocean waves that form from sudden changes in the level of the ocean floor, and then travel over long distances without dispersion or attenuation, possibly reeking havoc on distant shores.

## 24.2 Theory: Continuity and Advection Equations

The motion of a fluid is described by the continuity equation and the Navier–Stokes equation (Landau and Lifshitz, 1976). We will discuss the former here and the latter in Section 25.2. The continuity equation describes the conservation of mass:

(24.2)

**Figure 24.2** The formation of a tsunami. A single two-level waveform at time zero progressively breaks up into eight solitons (labeled) as time increases. The tallest soliton (1) is narrower and faster in its motion to the right. You can generate an animation of this with the program **SolitonAnimate.py**.

Here, _ρ_ ( _x, t_ ) is the mass density, **_v_** ( _ **x** , t_) is the velocity of the fluid, and the product _**j** = ρ **v**_ is the mass current. As its name implies, the divergence **∇ ⋅ _j_** describes the spreading of the current in a region of space, as might occur if there were a current source there. Physically, the continuity equation (24.2) states that changes in the density of the fluid within some region of space arise from the flow of current in and out of that region.

For 1D flow in the _x_ direction, and for a fluid that is moving with a constant velocity _v = c_ , the continuity equation (24.2) takes the simple form

(24.3)

This equation is known as the _advection equation,_ where the term "advection" is used to describe the horizontal transport of a quantity from one region of space to another as a result of a flow velocity field. For instance, advection describes dissolved salt transported in water.

The advection equation looks like a first-derivative form of the wave equation, and indeed, the two are related. A simple substitution proves that any function with the form of a traveling wave,

(24.4)

will be a solution of the advection equation. If we consider a surfer riding along the crest of a traveling wave, that is, remaining at the same position relative to the wave's shape as time changes, then the surfer does not see the shape of the wave change in time, which implies that

(24.5)

The speed of the surfer is, therefore d _x_ / d _t_ = _c_ , which is a constant. Any function _f_ ( _x – ct_ ) is clearly a traveling wave solution in which an arbitrary pulse is carried along by the fluid at velocity _c_ without changing shape.

### 24.2.1 Advection Implementation

Although the advection equation is simple, trying to solve it by a simple differencing scheme (the leapfrog method) may lead to unstable numerical solutions. As we shall see when we look at the nonlinear version of this equation, there are better ways to solve it. Listing 24.1 presents our code `**AdvecLax.py**` for solving the advection equation using the Lax–Wendroff method (a better method).

**Listing 24.1** **AdvecLax.py** solves the advection equation via the Lax–Wendroff scheme.

## 24.3 Theory: Shock Waves via Burgers' Equation

In a later section, we will examine the Korteweg–de Vries equation's description of solitary waves. In order to understand the physics contained in that equation, we study, one at a time, some of the terms in it. To start, consider Burgers' equation (Burgers, 1974):

(24.6)

(24.7)

where the second equation is the _conservative form_. This equation can be viewed as a variation on the advection equation (24.3) in which the wave speed _c =  u_ is proportional to the amplitude of the wave, as Russell found for his waves. The second, nonlinear, term in Burgers' equation leads to some unusual behaviors. Indeed, von Neumann studied this equation as a simple model for turbulence (Falkovich and Sreenivasan, 2006).

In the advection equation (24.3), all points on the wave move at the same speed _c_ , and so the shape of the wave remains unchanged in time. In Burgers' equation (24.6), the points on the wave move ("advect") themselves such that the local speed depends on the local wave's amplitude, with the high parts of the wave moving progressively faster than the low parts. This changes the shape of the wave in time; if we start with a wave packet that has a smooth variation in height, the high parts will speed up and push their way to the front of the packet, thereby forming a sharp leading edge known as a _shock wave_ (Tabor, 1989). A shock wave solution to Burgers' equation with ε = 1 is shown in Figure 24.3.

**Figure 24.3** A visualization showing the wave height vs. position for increasing times showing the formation of a shock wave (sharp edge) from an initial sine wave.

### 24.3.1 Lax–Wendroff Algorithm for Burgers' Equation

We first solve Burgers' equation (24.3) via the usual approach in which we express the derivatives as central differences. This leads to a leapfrog scheme for the future solution in terms of present and past ones:

(24.8)

Here, _u_ 2 is the square of _u_ and is not its second derivative, and _β_ is a ratio of constants known as the _Courant–Friedrichs–Lewy_ (CFL) _number_. As you should prove for yourself, _β_ < 1 is required for stability.

While we have used a leapfrog method successfully in the past, its low-order approximation for the derivative becomes inaccurate when the gradients can get large, as happens with shock waves, and so the leapfrog algorithm may become unstable (Press _et al.,_ 1994). The _Lax–Wendroff method_ attains better stability and accuracy by retaining second-order differences for the time derivative:

(24.9)

To covert (24.9) to an algorithm, we use Burgers' equation _∂u/∂t_ = − _ ∂_ ( _u_ 2/2)/ _∂x_ for the first-order time derivative. Likewise, we use Burger's equation to express the second-order time derivative in terms of space derivatives:

(24.10)

(24.11)

We next substitute these derivatives into the Taylor expansion (24.9) to obtain

(24.12)

We now replace the outer _x_ derivatives by central differences of spacing Δ _x_ /2:

(24.13)

Next we approximate _u_ ( _x_ ± Δ _x_ /2, _t_ ) by the average of adjacent grid points,

(24.14)

and apply a central-difference approximation to the second derivatives:

(24.15)

Finally, putting all these derivatives together yields the discrete form

(24.16)

where we have substituted the CFL number _β_. This Lax–Wendroff scheme is explicit, centered upon the grid points, and stable for _β_ < 1 (small nonlinearities).

### 24.3.2 Implementation and Assessment of Burgers' Shock Equation

  1. Write a program to solve Burgers' equation via the leapfrog method.
  2. Define arrays `**u0[100]**` and `**u[100]**` for the initial data and the solution.
  3. Take the initial wave to be sinusoidal, `**u0[i]**`= 3 sin(3.2 _x_ ), with speed _c_ = 1.
  4. Incorporate the boundary conditions `**u[0]=0**` and `**u[100]=0**`.
  5. Keep the CFL number _β_ < 1 for stability.
  6. Now modify your program to solve Burgers' shock equation (24.7) using the Lax–Wendroff method (24.16).
  7. Save the initial data and the solutions for a number of times in separate files for plotting.
  8. Plot the initial wave and the solution for several time values on the same graph in order to see the formation of a shock wave (see Figure 24.3).
  9. Run the code for several increasingly large CFL numbers. Is the stability condition _β_ < 1 correct for this nonlinear problem?
  10. Compare the leapfrog and Lax–Wendroff methods. With the leapfrog method you should see shock waves forming but breaking up into ripples as the square edge develops. The ripples are numerical artifacts. The Lax–Wendroff method should give a better shock wave (square edge), although some ripples may still occur.

Listing 24.1 presents our implementation of the Lax–Wendroff method.

## 24.4 Including Dispersion

We have just seen that Burgers' equation can turn an initially smooth wave into a square-edged shock wave. An inverse wave phenomenon is _dispersion,_ in which a waveform disperses or broadens as it travels through a medium. Dispersion does not cause waves to lose energy and attenuate, but rather to lose information with time. Physically, dispersion may arise when the propagating medium has structures with a spatial regularity equal to some fraction of a wavelength. Mathematically, dispersion may arise from terms in the wave equation that contain higher order space derivatives. For example, consider the waveform

(24.17)

corresponding to a plane wave traveling to the right ("traveling" because the phase _kx_ – _ωt_ remains unchanged if you increase _x_ with time). When this _u_ ( _x, t_ ) is substituted into the advection equation (24.3), we obtain

(24.18)

This equation is an example of a _dispersion relation,_ that is, a relation between frequency _ω_ and wave vector _k_. Because the _group velocity_ of a wave

(24.19)

the linear dispersion relation (24.18) leads to all frequencies having the same group velocity _c_ and thus _dispersionless_ propagation.

Let us now imagine that a wave is propagating with a small amount of _dispersion,_ that is, with a frequency that has somewhat less than a linear increase with the wave number _k_ :

(24.20)

Note that we skip the even powers in (24.20), so that the group velocity,

(24.21)

is the same for waves traveling to the left the or the right. Now we work backward. If plane-wave solutions like (24.17) were to arise from a wave equation, then (as verified by substitution) the _ω_ term of the dispersion relation (24.20) would arise from a first-order time derivative, the _ck_ term from a first-order space derivative, and the _k_ 3 term from a third-order space derivative:

(24.22)

We leave it as an exercise to show that solutions to this equation do indeed have waveforms that disperse in time.

## 24.5 Shallow-Water Solitons: The KdeV Equation

* * *

In this section, we put all of the pieces together that are needed to generate shallow-water solitary waves. This is a subject for which the computer has been absolutely essential for discovery and understanding. In addition, we recommend that you look at some of the soliton animations we provide online.

* * *

We want to understand the unusual water waves that occur in shallow, narrow channels such as canals (Abarbanel _et al.,_ 1993; Tabor, 1989). The analytic description of this "heap of water" was given by Korteweg and deVries (1895) (KdeV) with the partial differential equation

(24.23)

As we discussed in Section 24.1 in our study of Burgers' equation, the nonlinear term _ u∂u/∂t_ leads to a sharpening of the wave and ultimately a _shock_ wave. In contrast, as we discussed in our study of dispersion, the _∂_ 3 _u/∂x_ 3 term produces broadening, while the _∂u/∂t_ term produces traveling waves. For the proper parameters and initial conditions, the dispersive broadening exactly balances the nonlinear narrowing, and a stable traveling wave is formed.

Korteweg and de Vries solved (24.23) analytically and proved that the speed (24.1) given by Russell is in fact correct. Seventy years after its discovery, the KdeV equation was rediscovered by Zabusky and Kruskal (1965), who solved it numerically and found that a cos( _x/L_ ) initial condition broke up into eight solitary waves (Figure 24.2). They also found that the parts of the wave with larger amplitudes moved faster than those with smaller amplitudes, which is why the higher peaks tend to be on the right in Figure 24.2. As if wonders never cease, Zabusky and Kruskal, who coined the name _soliton_ for these solitary waves, also observed that a faster peak passed through a slower one unscathed (Figure 24.1).

### 24.5.1 Analytic Soliton Solution

The trick in analytic approaches to these types of nonlinear equations is to substitute a guessed solution that has the form of a traveling wave

(24.24)

This form means that if we move with a constant speed _c_ , we will see a constant wave form (but now the speed will depend on the magnitude of _u_ ). There is no guarantee that this form of a solution exists, but it is a lucky guess because substituting it into the KdeV equation produces a solvable ODE:

(24.25)

(24.26)

where _ξ_ 0 is the initial phase. We see in (24.26) an amplitude that is proportional to the wave speed _c_ , and a sech2 function that gives a single lump-like wave. This is an analytic form for a soliton.

### 24.5.2 Algorithm for KdeV Solitons

The KdeV equation is solved numerically using a finite-difference scheme with the time and space derivatives given by central-difference approximations:

(24.27)

To approximate _∂_ 3 _u_ ( _x, t_ )/ _∂x_ 3, we expand _u_ ( _x, t_ ) to   about the four points _u_ ( _x_ ± 2Δ _x, t_ ) and _u_ ( _x_ ± Δ _x, t_ ),

(24.28)

which we solve for  . Finally, the factor _u_ ( _x, t_ ) in the second term of (24.23) is taken as the average of three _x_ values all with the same _t_ :

(24.29)

We substitute these approximations to obtain the algorithm for the KdeV equation:

(24.30)

To apply this algorithm to predict future times, we need to know _u_ ( _x, t_ ) at present and past times. The initial time solution _u_ _i_ ,1 is known for all positions _i_ via the initial condition. To find _u_ _i_ ,2, we use a forward-difference scheme in which we expand _u_ ( _x, t_ ), keeping only two terms for the time derivative:

(24.31)

The keen observer will note that there are still some undefined columns of points, namely,  , and  , where _N_ max is the total number of grid points. A simple technique for determining their values is to assume that _u_ 1,2 = 1 and  . To obtain _u_ 2,2 and  , assume that _u_ i+2,2 = _u_ _i_ +1,2 and _u_ _i_ −2,2 = _u_ _i_ −1,2 (avoid _u_ _i_ +2,2 for _i_ = _N_ max − l, and _u_ _i_ −2,2 for _i_ = 2). To carry out these steps, approximate (24.31) so that

(24.32)

The truncation error and stability condition for our algorithm are related:

(24.33)

(24.34)

The first equation shows that smaller time and space steps lead to a smaller approximation error, yet because the round-off error increases with the number of steps, the total error does not necessarily decrease (Chapter 3). Yet, we are also limited in how small the steps can be made by the stability condition (24.34), which indicates that making Δ _x_ too small always leads to instability. Care and experimentation are required.

### 24.5.3 Implementation: KdeV Solitons

Modify or run the program Soliton.py in Listing 24.2 that solves the KdeV equation (24.23) for the initial condition

(24.35)

with parameters _ _= 0.2 and _μ_ = 0.1. Start with Δ _x_ = 0.4 and Δ _t_ = 0.1. These constants are chosen to satisfy (24.33) with | _u_ | = 1.

  1. Define a 2D array `**u[131,3]**` with the first index corresponding to the position _x_ and the second to the time _t_. With our choice of parameters, the maximum value for _x_ is 130 × 0.4 = 52.
  2. Initialize the time to _t_ = 0 and assign values to `**u[i,1]**`.
  3. Assign values to `**u[i,2],**` _i_ = 3, 4,..., 129, corresponding to the next time interval. Use (24.31) to advance the time but note that you cannot start at _i_ = 1 or end at _i_ = 131 because (24.31) would include `**u[132,2]**` and `**u[–1,1],**` which are beyond the limits of the array.
  4. Increment the time and assume that `**u[1,2]**` = 1 and `**u[131,2]**` = 0. To obtain `**u[2,2]**` and `**u[130,2],**` assume that `**u[i+2,2]**` = `**u[i+1,2]**` and `**u[i–2,2]**` = `**u[i-1,2]**`. Avoid `**u[i+2,2]**` for `**i = 130,**` and `**u[i-2,2]**` for `**i = 2**`. To do this, approximate (24.31) so that (24.33) is satisfied.
  5. Increment time and compute `**u[i,j]**` for `**j = 3**` and for `**i= 3, 4,..., 129,**` using (24.30). Again follow the same procedures to obtain the missing array elements `**u[2, j]**` and `**u[130,j]**` (set `**u[1,j = 1.**` and `**u[131, j] = 0**`). As you print out the numbers during the iterations, you will be convinced that it was a good choice.
  6. Set `**u[i,1] = u[i,2]**` and `**u[i,2] = u[i,3]**` for all `**i.**` In this way, you are ready to find the next `**u[i,j]**` in terms of the previous two rows.
  7. Repeat the previous two steps about 2000 times. Write your solution to a file after approximately every 250 iterations.
  8. Use your favorite graphics tool to plot your results as a 3D graph of disturbance _u_ vs. position _and vs_. time.
  9. Observe the wave profile as a function of time and try to confirm Russell's observation that a taller soliton travels faster than a smaller one.

**Listing 24.2** **Soliton.py** solves the KdeV equation for 1D solitons corresponding to a "bore" initial conditions.

The code **SolitonAnimate.py** produces an animation.

### 24.5.4 Exploration: Solitons in Phase Space, Crossing

  1. Explore what happens when a tall soliton collides with a short one.
    1. a) Start by placing a tall soliton of height 0.8 at _x_ = 12 and a smaller soliton in front of it at _x_ = 26:

(24.36)

    2. b) Do they reflect from each other? Do they go through each other? Do they interfere? Does the tall soliton still move faster than the short one after the collision (Figure 24.1)?
  2. Construct phase-space plots   of the KdeV equation for various parameter values. Note that only very specific sets of parameters produce solitons. In particular, by correlating the behavior of the solutions with your phase-space plots, show that the soliton solutions correspond to the _separatrix_ solutions to the KdeV equation. In other words, the stability in time for solitons is analogous to the infinite period for a pendulum balanced straight upward.

## 24.6 Solitons on Pendulum Chain

In 1955, Fermi, Ulam, and Pastu were investigating how a 1D chain of coupled oscillators disperses waves. Because waves of differing frequencies traveled through the chain with differing speeds, a pulse, which inherently includes a range of frequencies, broadens as time progresses. Surprisingly, when the oscillators were made more realistic by introducing a nonlinear term into Hooke's law

(24.37)

Fermi, Ulam, and Pastu found that even in the presence of dispersion, a sharp pulse in the chain would survive indefinitely. Your _problem_ is to explain how this combination of dispersion and nonlinearity can combine to produce a stable pulse.

In Chapter 14, we studied nonlinear effects in a single pendulum arising from large oscillations. Now we go further and couple a number of those pendulums together. As shown in Figure 24.4, we take as our model a 1D chain of identical, equally spaced pendulums connected by a torsion bar that twists as the pendulums swing. The angle _θ_ _i_ measures the displacement of pendulum _i_ from its equilibrium position, and _a_ the distance between pivot points. If all the pendulums are set off swinging together, _θ_ _i_ ≡ _θ_ _j_ , the coupling torques would vanish and we would have our old friend, the equation for a realistic pendulum. We assume that three torques act on each pendulum, a gravitational torque trying to return the pendulum to its equilibrium position, and the two torques from the twisting of the bar to the right and to the left of the pendulum. The equation of motion for pendulum _j_ follows from Newton's law for rotational motion:

**Figure 24.4** A 1D chain of pendulums coupled with a torsion bar on top. The pendulums swing in planes perpendicular to the length of the bar.

(24.38)

(24.39)

(24.40)

where _I_ is the moment of inertia of each pendulum, _L_ is the length of the pendulum, and _k_ is the torque constant of the bar. The nonlinearity in (24.40) arises from the sin _θ_   _θ_ − _θ_ 3/6 +... dependence of the gravitational torque. As it stands, (24.40) is a set of coupled nonlinear equations, with the number of equations equal to the number of oscillators, which would be large for a realistic solid.

### 24.6.1 Including Dispersion

Consider a surfer remaining on the crest of a wave. Since he/she does not see the wave form change with time, his/her position is given by a function of the form __f( _kx_ − _ωt_ ). Consequently, to his/her the wave has a constant phase

(24.41)

The surfer's (phase) velocity is the rate of change of _x_ with respect to time,

(24.42)

which is constant. In general, the frequency _ω_ may be a nonlinear function of _k_ , in which case the phase velocity varies with frequency and we have _dispersion_. If the wave contained just one frequency, then you would not observe any dispersion, but if the wave was a pulse composed of a range of Fourier components, then it would broaden and change shape in time as each frequency moved with a differing phase velocity. So although dispersion does not lead to an energy loss, it may well lead to a loss of information content as pulses broaden and overlap.

The functional relation between frequency _ω_ and the wave vector _k_ is called a _dispersion relation_. If the Fourier components in a wave packet are centered around a mean frequency _ω_ 0, then the pulse's information travels, not with the phase velocity, but with the _group velocity_

(24.43)

A comparison of (24.42) and (24.43) makes it clear that when there is dispersion the group and phase velocities may well differ.

To isolate the dispersive aspect of (24.40), we examine its linear version

(24.44)

where   is the natural frequency for any one pendulum. Because we want to determine if a wave with a single frequency propagates on this chain, we start with a traveling wave with frequency _ω_ and wavelength _λ_ ,

(24.45)

and then test if it is a solution. Substitution of (24.45) into the wave equation (24.44) produces the _dispersion relation_ (Figure 24.5):

(24.46)

To have dispersionless propagation (all frequencies propagate with the same velocity), we need a linear relation between _ω_ and _k_ :

(24.47)

This will be true for the chain only if _ka_ is small, because then cos _ka_   1 and _ω_   _ω_ 0.

Not only does the dispersion relation (24.46) change the speed of waves, but it also limits which frequencies can propagate on the chain. In order to have real _k_ solutions, _ω_ must lie in the range

(24.48)

**Figure 24.5** The dispersion relation for a linearized chain of pendulums.

The minimum frequency _ω_ 0 and the maximum frequency _ω_ * are related through the limits of cos _ka_ in (24.46),

(24.49)

Waves with _ω_ < _ω_ 0 do not propagate, while those with _ω_ > _ω_ * are nonphysical because they correspond to wavelengths _λ_ < 2 _a_ , that is, oscillations where there are no particles. These high and low _ω_ cutoffs change the shape of a propagating pulse, that is, cause dispersion.

### 24.6.2 Continuum Limit, the Sine-Gordon Equation

If the wavelengths in a pulse are much longer than the pendulum–pendulum repeat distance _a_ , that is, if _ka_  : 1, the chain can be approximated as a continuous medium. In this limit, _a_ becomes the continuous variable _x_ , and the system of coupled ordinary differential equations becomes a single, partial differential equation:

(24.50)

(24.51)

(24.52)

If we measure time in units of   and distances in units of  , we obtain the standard form of the sine-Gordon equation (SGE)1):

(24.53)

where the sin _θ_ on the RHS introduces the nonlinear effects.

### 24.6.3 Analytic SGE Solution

The nonlinearity of the SGE (24.53) makes it hard to solve analytically. There is, however, a trick. Guess a functional form of a traveling wave, substitute it into (24.53) and thereby convert the PDE into a solvable ODE:

(24.54)

You should recognize (24.54), the equation of motion for the realistic pendulum with no driving force and no friction, as our old friend. The constant _v_ is a velocity in natural units, and separates different regimes of the motion:

(24.55)

Although the equation is familiar, it does not mean that an analytic solution exists. However, for an energy _E_ = ±1, we have motion along the separatrix, and have a solution with the characteristic _soliton_ form,

(24.56)

This soliton corresponds to a solitary _kink_ traveling with velocity _v_ = −1 that flips the pendulums around by 2 _π_ as it moves down the chain. There is also an _antikink_ in which the initial _θ_ = _π_ values are flipped to final _θ_ = − _π_.

### 24.6.4 Numeric Solution: 2D SGE Solitons

We have already solved for 1D solitons arising from the KdeV equation. The elastic-wave solitons that arise from the SGE can be easily generalized to two dimensions, as we do here with the 2D generalization of the SGE equation (24.53):

(24.57)

Whereas the 1D SGE describes wave propagation along a chain of connected pendulums, the 2D form might describe wave propagation in nonlinear elastic media. Interestingly enough, the same 2D SGE also occurs in quantum field theory, where the soliton solutions have been suggested as models for elementary particles (Christiansen and Lomdahl, 1981; Christiansen and Olsen, 1978; Argyris, 1991). The idea is that, like elementary particles, the solutions are confined to a region of space for a long period of time by nonlinear forces, and do not radiate away their energy. We solve (24.57) in a finite region of 2D space and for positive times:

(24.58)

We take _x_ 0 = _y_ 0 = 7 and impose the _boundary conditions_ that the derivative of the displacement vanishes at the ends of the region:

(24.59)

We also impose the _initial condition_ that at time _t_ = 0 the waveform is that of a pulse (Figure 24.6) with its surface at rest:

(24.60)

We discretize the equation first by looking for solutions on a space–time lattice:

(24.61)

(24.62)

Next we replace the derivatives in (24.57) by their finite-difference approximations:

(24.63)

To make the algorithm simpler and establish stability, if we make the time and space steps proportional,  . This leads to all of the   terms dropping out:

(24.64)

Likewise, the discrete form of vanishing initial velocity (24.60) becomes

(24.65)

This will be useful in setting the initial conditions. The lattice points on the edges and corners cannot be obtained from these relations, but must be obtained by applying the boundary conditions (24.59):

**Figure 24.6** A circular ring soliton at times 8, 20, 40, 60, 80, and 120. This has been proposed as a model for an elementary particle.

(24.66)

(24.67)

Similarly, the other derivatives in (24.59) give

(24.68)

where _N_ max is the number of grid points used for one space dimension.

### 24.6.5 2D Soliton Implementation

  1. Define an array _u_ [ _N_ max, _N_ max, 3] with _N_ max = 201 for the space slots and the 3 for the time slots.
  2. The solution (24.60) for the initial time _t_ = 0 is placed in __u_ [ _m_ , _l_ , 1]_.
  3. The solution for the second time Δ _t_ is placed in _u_ ( _m_ , _l_ , 2), and the solution for the next time, 2Δ _t_ , is placed in _u_ [ _m_ , _l_ , 3].
  4. Assign the constants,  .
  5. Start off at _t_ = 0 with the initial conditions and impose the boundary conditions to this initial solution. This is the solution for the first time step, defined over the entire 201 × 201 grid.
  6. For the second time step, increase time by Δ _t_ and use (24.64) for all points in the plane. Do not include the edge points.
  7. At the edges, for _i_ = 1, 2,..., 200, set

  8. To find values for the four points in the corners for the second time step, again use initial condition (24.64):

  9. For the third time step (the future), use (24.64).
  10. Continue the propagation forward in time, reassigning the future to the present, and so forth. In this way, we need to store the solutions for only three time steps.

### 24.6.6 SGE Soliton Visualization

We see in Figure 24.6 the time evolution of a circular ring soliton that results for the stated initial conditions. We note that the ring at first shrinks in size, then expands, and then shrinks back into another (but not identical) ring soliton. A small amount of the particle does radiate away, and in the last frame we can notice some interference between the radiation and the boundary conditions. An animation of this sequence can be found online.

1) The name "sine-Gordon" is either a reminder that the SGE is like the Klein–Gordon equation of relativistic quantum mechanics with a sin _u_ added to the RHS, or a reminder of how clever one can be in thinking up names.

# 25
Fluid Dynamics

* * *

We have already introduced some fluid dynamics in the previous chapter's discussion of shallow-water solitons. This chapter confronts the more general equations of computational fluid dynamics (CFD) and their solutions.1) The mathematical description of the motion of fluids, although not a new subject, remains a challenging one. Equations are complicated and nonlinear, there are many degrees of freedom, the nonlinearities may lead to instabilities, analytic solutions are rare, and the boundary conditions for realistic geometries (like airplanes) are not intuitive. These difficulties may explain why fluid dynamics is often absent from undergraduate and even graduate physics curricula. Nonetheless, as an essential element of the physical world that also has tremendous practical importance, we encourage its study. We recommend Fetter and Walecka (1980); Landau and Lifshitz (1987) for those interested in the derivations, and Shaw (1992) for more details about the computations.

## 25.1 River Hydrodynamics

**Problem** In order to give migrating salmon a place to rest during their arduous upstream journey, the Oregon Department of Environment is placing objects in several deep, wide, fast-flowing streams. One such object is a long beam of rectangular cross section (Figure 25.1a), and another is a set of plates (Figure 19.4b). The objects are to be placed far enough below the water's surface so as not to disturb the surface flow, and far enough from the bottom of the stream so as not to disturb the flow there either. Your _problem_ is to determine the spatial dependence of the stream's velocity when the objects are in place, and, specifically, whether the wake of the object will be large enough to provide a resting place for a meter-long salmon.

**Figure 25.1** Side view of the flow of a stream around a submerged beam (a) and around two parallel plates (b). Both beam and plates have length _L_ along the direction of flow. The flow is seen to be symmetric about the center-line and to be unaffected at the bottom and at the surface by the submerged object.

## 25.2 Navier–Stokes Equation (Theory)

As with our study of shallow-water waves, we assume that water is _incompressible_ and thus that its density _ρ_ is constant. We also simplify the theory by looking only at steady-state situations, that is, ones in which the velocity is not a function of time. However, to understand how water flows around objects, like our beam, it is essential to include the complication of frictional forces ( _viscosity_ ).

For the sake of completeness, we repeat here the first equation of hydrodynamics, the continuity equation (24.2):

(25.1)

Before proceeding to the second equation, we introduce a special time derivative, the _hydrodynamic_ derivative _D **v**_ / _Dt_ , which is appropriate for a quantity contained in a moving fluid (Fetter and Walecka, 1980):

(25.2)

This derivative gives the rate of change, as viewed from a stationary frame, of the velocity of material in _an element of flowing fluid_ and so incorporates changes as a result of the motion of the fluid (first term) as well as any explicit time dependence of the velocity. Of particular interest is that _D **v**_ / _Dt_ is of second order in the velocity, and so its occurrence introduces nonlinearities into the theory. You may think of these nonlinearities as related to the fictitious (inertial) forces that occur when we describe the motion in the fluid's rest frame (an accelerating frame to us).

The material derivative is the leading term in the _Navier–Stokes equation_ ,

(25.3)

(25.4)

Here, _ν_ is the kinematic viscosity, _P_ is the pressure, and (25.4) gives the derivatives in (25.3) in Cartesian coordinates. This equation describes transfer of the momentum of the fluid within some region of space as a result of forces and flow (think d _**p**_ /d _t_ = _**F**_ ). There is a simultaneous equation for each of the three velocity components. The _**v**_. ∇ _ **v**_ term in _D **v**_ / _Dt_ describes the transport of momentum in some region of space resulting from the fluid's flow and is often called the _convection_ or _advection_ term.2) The **∇** _P_ term describes the velocity change as a result of pressure changes, and the _v_ ∇2 ** _v_** term describes the velocity change resulting from viscous forces (which tend to dampen the flow).

The explicit functional dependence of the pressure on the fluid's density and temperature _P_ ( _ρ_ , _T_ , _x_ ) is known as the _equation of state of the fluid_ , and would have to be known before trying to solve the Navier–Stokes equation. To keep our problem simple, we assume that the pressure is independent of density and temperature. This leaves the four simultaneous partial differential equations to solve, (25.1) and (25.3). Because we are interested in _steady-state_ flow around an object, we assume that all time derivatives of the velocity vanish. Because we assume that the fluid is incompressible, the time derivative of the density also vanishes, and (25.1) and (25.3) become

(25.5)

(25.6)

The first equation expresses the equality of inflow and outflow and is known as the _condition of incompressibility_. In as much as the stream in our problem is much wider than the width ( _z_ direction) of the beam, and because we are staying away  from the banks, we will ignore the _z_ dependence of the velocity. The explicit PDEs we need to solve then reduce to

(25.7)

(25.8)

(25.9)

### 25.2.1 Boundary Conditions for Parallel Plates

The plate problem is relatively easy to solve analytically, and so we will do it! This will give us some experience with the equations as well as a check for our numerical solution. To find a unique solution to the PDEs (25.7)–(25.9), we need to specify boundary conditions. As far as we can tell, picking boundary conditions is somewhat of an acquired skill.

We assume that the submerged parallel plates are placed in a stream that is flowing with a constant velocity _V_ 0 in the horizontal direction (Figure 25.1b). If the velocity _V_ 0 is not too high or the kinematic viscosity _ν_ is sufficiently large, then the flow should be smooth and without turbulence. We call such flow _laminar_. Typically, a fluid undergoing laminar flow moves in smooth paths that do not close on themselves, like the smooth flow of water from a faucet. If we imagine attaching a vector to each element of the fluid, then the path swept out by that vector is called a _streamline_ or _line of motion_ of the fluid. These streamlines can be visualized experimentally by adding colored dye to the stream. We assume that the plates are so thin that the flow remains laminar as it passes around and through them.

If the plates are thin, then the flow upstream of them is not affected, and we can limit our solution space to the rectangular region in Figure 25.2. We assume that the length _L_ and separation _H_ of the plates are small compared to the size of the stream, so the flow returns to uniform as we get far downstream from the plates. As shown in Figure 25.2, there are boundary conditions at the _inlet_ where the fluid enters the solution space, at the _outlet_ where it leaves, and at the stationary plates. In addition, because the plates are far from the stream's bottom and surface, we assume that the dotted-dashed centerline is a plane of symmetry, with identical flow above and below the plane. We thus have four different types of boundary conditions to impose on our solution:

  1. **Solid plates** Because there is friction (viscosity) between the fluid and the plate surface, the only way to have laminar flow is to have the fluid's velocity equal to the plate's velocity, which means both are zero:

(25.10)

  2. Such being the case, we have smooth flow in which the negligibly thin plates lie along streamlines of the fluid (like a "streamlined" vehicle).
  3. **Inlet** The fluid enters the integration domain at the inlet with a horizontal velocity _V_ 0. Because the inlet is far upstream from the plates, we assume that the fluid velocity at the inlet is unchanged by the presence of the plates:

(25.11)

  4. **Outlet** Fluid leaves the integration domain at the outlet. While it is totally reasonable to assume that the fluid returns to its unperturbed state there, we are not told what that is. So, instead, we assume that there is a physical outlet at the end with the water just shooting out of it. Consequently, we assume that the water pressure equals zero at the outlet (as at the end of a garden hose) and that the velocity does not change in a direction normal to the outlet:

(25.12)

  5. **Symmetry plane** If the flow is symmetric about the _y_ = 0 plane, then there cannot be flow through the plane, and the spatial derivatives of the velocity components normal to the plane must vanish:

(25.13)

This condition follows from the assumption that the plates are along streamlines and that they are negligibly thin. It means that all the streamlines are parallel to the plates as well as to the water surface, and so it must be that _v y_ = 0 everywhere. The fluid enters in the horizontal direction, the plates do not change the vertical _y_ component of the velocity, and the flow remains symmetric about the centerline. There is a retardation of the flow around the plates because of the viscous nature of the flow and that of the _**v**_ = 0 boundary layers formed on the plates, but there are no actual _v y_ components.

**Figure 25.2** The boundary conditions for two thin submerged plates. The surrounding box is the integration volume within which we solve the PDEs and upon whose surface we impose the boundary conditions. In practice, the box is much larger than _L_ and _H_

### 25.2.2 Finite-Difference Algorithm and Overrelaxation

Now we develop an algorithm for solution of the Navier–Stokes and continuity PDEs using successive over-relaxation. This is a variation of the method used in Chapter 19, to solve Poisson's equation. We divide space into a rectangular grid with the spacing _h_ in both the _x_ and _y_ directions:

We next express the derivatives in (25.7)–(25.9) as finite differences of the values of the velocities at the grid points using central-difference approximations. For _v_ = 1 m2/s and _ρ_ = 1 kg/m3, this yields

(25.14)

(25.15)

(25.16)

Because _v_ _y_ ≡ 0 for this problem, we rearrange terms to obtain for _v_ _x_ :

(25.17)

We recognize in (25.17) an algorithm similar to the one we used in solving Laplace's equation by relaxation. Indeed, as we did there, we can accelerate the convergence by writing the algorithm with the new value of _v x_ given as the old value plus a correction (residual):

(25.18)

(25.19)

As performed with the Poisson equation algorithm, successive iterations sweep the interior of the grid, continuously adding in the residual (25.18) until the change becomes smaller than some set level of tolerance, | _r_ _i_ , _j_ | <  .

A variation of this method, _successive over-relaxation_ , increases the speed at which the residuals approach zero via an amplifying factor _ω_ :

(25.20)

The standard relaxation algorithm (25.18) is obtained with _ω_ = 1, an accelerated convergence ( _over-relaxation_ ) is obtained with _ω_ ≥ l, and _underrelaxation_ occurs for _ω_ < 1. Values _ω_ > 2 are found to lead to numerical instabilities. Although a detailed analysis of the algorithm is necessary to predict the optimal value for _ω_ , we suggest that you test different values for _ω_ to see which one provides the fastest convergence for your problem.

### 25.2.3 Successive Overrelaxation Implementation

  1. Modify the program `**Beam.py**`, or write your own, to solve the Navier–Stokes equation for the velocity of a fluid in a 2D flow. Represent the _x_ and _y_ components of the velocity by the arrays `**vx[Nx,Ny]**` and `**vy[Nx,Ny]**`.
  2. Specialize your solution to the rectangular domain and boundary conditions indicated in Figure 25.2.
  3. Use of the following parameter values,

leads to the analytic solution

(25.21)

  4. For the relaxation method, output the iteration number and the computed _v x_ and then compare the analytic and numeric results.
  5. Repeat the calculation and see if SOR speeds up the convergence.

## 25.3 2D Flow over a Beam

Now that the comparison with an analytic solution has shown that our CFD simulation works, we return to determining if the beam in Figure 25.1 might produce a good resting place for salmon. While we have no analytic solution with which to compare, our canoeing and fishing adventures have taught us that _standing waves_ with fish in them are often formed behind rocks in streams, and so we expect there will be a standing wave formed behind the beam.

## 25.4 Theory: Vorticity Form of Navier–Stokes Equation

We have seen how to solve numerically the hydrodynamics equations

(25.22)

(25.23)

These equations determine the components of a fluid's velocity, pressure, and density as functions of position. In analogy to electrostatics, where one usually solves for the simpler scalar potential and then takes its gradient to determine the more complicated vector field, we now recast the hydrodynamic equations into forms that permit us to solve two simpler equations for simpler functions, from which the velocity is obtained via a gradient operation.3)

We introduce the _stream function_ _**u**_ ( _ **x**_ ) from which the velocity is determined by the curl operator:

(25.24)

Note the absence of the _z_ component of velocity **_v_** for our problem. Because **∇**. **∇** × ** _u_** ≡ 0, we see that any **_v_** that can be written as the curl of **_u_** automatically satisfies the continuity equation **∇· _v_** = 0. Further, because _v_ for our problem has only _x_ and _y_ components, _u_ ( _x_ ) needs have only a _z_ component:

(25.25)

It is worth noting that in 2D flows, the contour lines _u_ = constant are the _streamlines_.

The second simplifying function is the _vorticity_ field **_w_** ( _x_ ), which is related physically and alphabetically to the angular velocity **_ω_** of the fluid. Vorticity is defined as the curl of the velocity (sometimes with a – sign):

(25.26)

Because the velocity in our problem does not change in the _z_ direction, we have

(25.27)

Physically, we see that the vorticity is a measure of how much the fluid's velocity curls or rotates, with the direction of the vorticity determined by the right-hand  rule for rotations. In fact, if we could pluck a small element of the fluid into space (so it would not feel the internal strain of the fluid), we would find that it is rotating like a solid with angular velocity **_ω_ ∝ _w_** (Lamb, 1993). That being the case, it is useful to think of the vorticity as giving the local value of the fluid's angular velocity vector. If **_w_** = 0, we have _irrotational_ flow.

The field lines of _w_ are continuous and move as if they are attached to the particles of the fluid. A uniformly flowing fluid has vanishing curl, while a nonzero vorticity indicates that the current curls back on itself or rotates. From the definition of the stream function (25.24), we see that the vorticity **_w_** is related to it by

(25.28)

where we have used a vector identity for **∇** × ( **∇** × **_u_** ). Yet the divergence **∇** • **_u_** = 0 because **_u_** has a component only in the _z_ -direction and that component is independent of _z_ (or because there is no source for **_u_** ). We have now obtained the basic relation between the stream function **_u_** and the vorticity **_w_** :

(25.29)

Equation 25.29 is analogous to Poisson's equation of electrostatics, ∇2 _φ_ = − 4π _ρ_ , only now each component of vorticity _w_ is a source for the corresponding component of the stream function **_u_**. If the flow is irrotational, that is, if **_w_** = 0, then we need only to solve Laplace's equation for each component of _u_. Rotational flow, with its coupled nonlinearities equations, leads to more interesting behavior.

As is to be expected from the definition of **_w_** , the vorticity form of the Navier–Stokes equation is obtained by taking the curl of the velocity form, that is, by operating on both sides with **∇** ×. After significant manipulations one obtains

(25.30)

This and (25.29) are the two simultaneous PDEs that we need to solve. In 2D, with **_u_** and **_w_** having only _z_ components, they are

(25.31)

(25.32)

So after all that work, we end up with two simultaneous, nonlinear, elliptic PDEs that look like a mixture of Poisson's equation with the wave equation. The equation for _u_ is Poisson's equation with source _w_ , and must be solved simultaneously with the second. It is this second equation that contains mixed products of the derivatives of _u_ and _w_ and thus introduces the nonlinearity.

### 25.4.1 Finite Differences and the SOR Algorithm

We solve (25.31) and (25.32) on an _N x_ × _N y_ grid of uniform spacing _h_ with

(25.33)

Because the beam is symmetric about its centerline (Figure 25.1a), we need the solution only in the upper half-plane. We apply the now familiar central-difference approximation to the Laplacians of _u_ and _w_ to obtain the difference equation

(25.34)

Likewise, for the product of the first derivatives,

(25.35)

The difference form of the vorticity of the Navier–Stokes equation (25.31) becomes

(25.36)

(25.37)

(25.38)

Note that we have placed _u_ _i_ , _j_ and _w_ _i_ , _j_ , on the LHS of the equations in order to obtain an algorithm appropriate to a solution by relaxation.

The parameter _R_ in (25.38) is related to the _Reynolds number_. When we solve the problem in natural units, we measure distances in units of grid spacing _h_ , velocities in units of initial velocity _V_ 0, stream functions in units of _V_ 0 _h_ , and vorticity in units of _V_ 0/ _h_. The second form is in regular units and is dimensionless. This _R_ is known as the _grid Reynolds number_ and differs from the physical _R_ , which has a pipe diameter in place of the grid spacing _h_.

The grid Reynolds number is a measure of the strength of the coupling of the nonlinear terms in the equation. When the physical _R_ is small, the viscosity acts as a frictional force that damps out fluctuations and keeps the flow smooth. When _R_ is large ( _R_   2000), physical fluids undergo phase transitions from laminar to turbulent flow in which turbulence occurs at a cascading set of smaller and smaller space scales (Reynolds, 1883). However, simulations that produce the onset of turbulence have been a research problem because Reynolds first experiments in 1883 (Falkovich and Sreenivasan, 2006), possibly because the laminar flow is stable against small perturbations and some large-scale "kick" may be needed to change laminar to turbulent flow. Recent research along these lines have been able to find unstable, traveling-wave solutions to the Navier–Stokes equations, and the hope is that these may lead to a turbulent transition (Fitzgerald, 2004).

As discussed in Section 25.2.2, the finite-difference algorithm can have its convergence accelerated by the use of successive over-relaxation (25.36):

(25.39)

Here, _ω_ is the over-relaxation parameter and should lie in the range 0 < _ω_ < 2 for stability. The residuals are just the changes in a single step, _r_ (1) = _u_ new − _u_ old and _r_ (2) = _w_ new − _w_ old:

(25.40)

### 25.4.2 Boundary Conditions for a Beam

A well-defined solution of these elliptic PDEs requires a combination of (less than obvious) boundary conditions on _u_ and _w_. Consider Figure 25.3, based on the analysis of Koonin (1986). We assume that the inlet, outlet, and surface are far from the beam, which may not be evident from the not-to-scale figure.

  1. **Freeflow** If there were no beam in the stream, then we would have free flow with the entire fluid possessing the inlet velocity:

(25.41)

**Figure 25.3** Boundary conditions for flow around the beam in Figure 25.1. The flow is symmetric about the centerline, and the beam has length _L_ in the _x_ direction (along flow).

    1. (Recall that we can think of _w_ = 0 as indicating no fluid rotation.) The center-line divides the system along a symmetry plane with identical flow above and below it. If the velocity is symmetric about the centerline, then its _y_ component must vanish there:

(25.42)

  2. **Centerline** The centerline is a streamline with _u_ = constant because there is no velocity component perpendicular to it. We set _u_ = 0 according to (25.41). Because there cannot be any fluid flowing into or out of the beam, the normal component of velocity must vanish along the beam surfaces. Consequently, the streamline _u_ = 0 is the entire lower part of Figure 25.3, that is, the center-line and the beam surfaces. Likewise, the symmetry of the problem permits us to set the vorticity _w_ = 0 along the centerline.
  3. **Inlet** At the inlet, the fluid flow is horizontal with uniform _x_ component _V_ 0 at all heights and with no rotation:

(25.43)

  4. **Surface** We are told that the beam is sufficiently submerged so as not to disturb the flow on the surface of the stream. Accordingly, we have free-flow conditions on the surface:

(25.44)

  5. **Outlet** Unless something truly drastic happens, the conditions on the far downstream outlet have little effect on the far upstream flow. A convenient choice is to require the stream function and vorticity to be constant:

(25.45)

  6. **Beamsides** We have already noted that the normal component of velocity _v x_ and stream function _u_ vanish along the beam surfaces. In addition, because the flow is viscous, it is also true that the fluid "sticks" to the beam somewhat and so the tangential velocity also vanishes along the beam's surfaces. While these may all be true conclusions regarding the flow, specifying them as boundary conditions would over-restrict the solution (see Table 19.2 for elliptic equations) to the point where no solution may exist. Accordingly, we simply impose the _no-slip_ boundary condition on the vorticity _w_. Consider a grid point ( _x_ , _y_ ) on the upper surface of the beam. The stream function _u_ at a point ( _x_ , _y_ \+ _h_ ) above it can be related via a Taylor series in _y_ :

(25.46)

  7. Because **_w_** has only a _z_ component, it has a simple relation to **∇** × **_ν_** :

(25.47)

  8. Because of the fluid's viscosity, the velocity is stationary along the beam top:

(25.48)

  9. Because the current flows smoothly along the top of the beam, _v y_ must also vanish. In addition, because there is no _x_ variation, we have

(25.49)

  10. After substituting these relations into the Taylor series (25.46), we can solve for _w_ and obtain the finite-difference version of the top boundary condition:

(25.50)

  11. Similar treatments applied to other surfaces yield the boundary conditions.

(25.51)

### 25.4.3 SOR on a Grid

`**Beam.py**` in Listing 25.1 is our program for the solution of the vorticity form of the Navier–Stokes equation. You will notice that while the relaxation algorithm is rather simple, some care is needed in implementing many boundary conditions. Relaxation of the stream function and of the vorticity are performed by separate functions.

**Listing 25.1** **Beam.py** solves the Navier–Stokes equation for the flow over a plate.

### 25.4.4 Flow Assessment

  1. Use `**Beam.py**` as a basis for your solution for the stream function _u_ and the vorticity _w_ using the finite-differences algorithm (25.36).
  2. A good place to start your simulation is with a beam of size _L_ = 8 _h_ , _H_ = _h_ , Reynolds number _R_ = 0.1, and intake velocity _V_ 0 = 1. Keep your grid small during debugging, say, _N x_ = 24 and _N y_ = 70.
  3. Explore the _convergence_ of the algorithm.
    1. a) Print out the iteration number and _u_ values upstream from, above, and downstream from the beam.
    2. b) Determine the number of iterations necessary to obtain three-place convergence for successive relaxation ( _ω_ = 0).
    3. c) Determine the number of iterations necessary to obtain three-place convergence for successive over-relaxation ( _ω_   0.3). Use this number for future calculations.
  4. Change the beam's horizontal placement so that you can see the undisturbed current entering on the left and then developing into a standing wave. Note that you may need to increase the size of your simulation volume to see the effect of all the boundary conditions.
  5. Make surface plots including contours of the stream function _u_ and the vorticity _w_. Explain the behavior seen.
  6. Is there a region where a big fish can rest behind the beam?
  7. The results of the simulation (Figure 25.4) are for the one-component stream function _u_. Make several visualizations showing the fluid velocity throughout the simulation region. Note that the velocity is a vector with two components (or a magnitude and direction), and both degrees of freedom are interesting to visualize. A plot of vectors would work well here.

**Figure 25.4** Two visualizations of the stream function _u_ for Reynold's number _R_ = 5. The one on the left uses contours, while the one on the right uses color.

**Figure 25.5** (a) The velocity field around the beam as represented by vectors. (b) The vorticity as a function of _x_ and _y_ for the same velocity field. Rotation is seen to be largest behind the beam.

  8. Explore how increasing the Reynolds number _R_ changes the flow pattern. Start at _R_ = 0 and gradually increase _R_ while watching for numeric instabilities. To overcome the instabilities, reduce the size of the relaxation parameter _ω_ and continue to larger _R_ values.
  9. Verify that the flow around the beam is smooth for small _R_ values, but that it separates from the back edge for large _R_ , at which point a small vortex develops, as can be seen in Figure 25.5.

### 25.4.5 Exploration

  1. Determine the flow behind a circular rock in the stream.
  2. The boundary condition at an outlet far downstream should not have much effect on the simulation. Explore the use of other boundary conditions there.
  3. Determine the pressure variation around the beam.

1) We acknowledge some helpful reading of by Satoru S. Kano. 2) We discuss pure advection in Section 24.1. In oceanology or meteorology, convection implies the transfer of mass in the vertical direction where it overcomes gravity, whereas advection refers to transfer in the horizontal direction. 3) If we had to solve only the simpler problem of _irrotational flow_ (no turbulence), then we would be able to use a scalar velocity potential, in close analogy to electrostatics (Lamb, 1993). For the more general _rotational flow_ , two vector potentials are required.

# 26
Integral Equations of Quantum Mechanics

* * *

> There are people – amongst whom I would include myself – who detest happy endings.
>
> _Vlasimir Nabokov, Pnin_

We have put this chapter off till last because it is the one in which we have the closest personal connections. The power and accessibility of high-speed computers have changed the view as to what kind of equations are soluble. We have seen how even nonlinear differential equations can be solved easily and can give new insight into the physical world. In this chapter, we examine how the integral equations of quantum mechanics can be solved for both bound and scattering states. We start by extending our treatment of the eigenvalue problem, earlier solved as a coordinate-space differential equation, to the equivalent integral-equation problem in momentum space. Then, we treat the singular integral equations for scattering, a problem whose multiple challenges have been met well by computational physics.

* * *

## 26.1 Bound States of Nonlocal Potentials

**Problem** A particle undergoes a many-body interaction with a medium (Figure 26.1). Although, we can write down the many-body Schrödinger equation that describes this interaction as the sum of many potentials, the number of coordinates involved is too large for a practical solution. Instead theorists have derived an effective single-particle potential that accounts for the many particles present by having the potential that is _nonlocal,_ that is, the effective potential at **_r_** that depends on the wave function at the **_r′_** values of the other particles (Landau, 1996):

(26.1)

**Figure 26.1** A dark particle moving in a dense medium in which it interacts with all particles present. The nonlocality of the potential felt by the dark particle at _**r**_ arises from the interactions at all **_r′_**.

This type of interaction leads to a Schrödinger equation which is a combined integral and differential ("integrodifferential") equation:

(26.2)

Your _problem_ is to figure out how to find the bound-state energies _E n_ and wave functions _ψ n_ for the equation in (26.2).1)

## 26.2 Momentum–Space Schrödinger Equation (Theory)

One way of dealing with (26.2) is by going to momentum space where it becomes the integral equation (Landau, 1996)

(26.3)

We restrict our solution to angular momentum _l_ = 0 partial waves, but for simplicity of notation we do not include an index to indicate that. In (26.3), _V_ ( _k, p_ ) is the momentum–space representation (double Fourier transform) of the coordinate-space potential,

(26.4)

_ψ n_( _k_ ) is the momentum–space wave function (the probability amplitude for finding the particle with momentum _k_ ), and is the Fourier transform of _ψ n_( _r_ ):

(26.5)

Equation 26.3 is an integral equation for _ψ n_( _k_ ). It differs from an integral representation of _ψ n_( _k_ ), because the integral in it cannot be evaluated until _ψ n_( _p_ ) is known. Although this may seem like an insurmountable barrier, we will transform this equation into a matrix equation that can be solved with the matrix techniques discussed in Chapter 6.

### 26.2.1 Integral to Matrix Equations

We approximate the integral over the potential as a weighted sum over _N_ integration points (usually Gauss quadrature points2)) for _p_ = _k j_, _j_ = 1, _N_ :

(26.6)

This converts the integral equation (26.3) to the algebraic equation

(26.7)

Equation 26.7 contains the _N_ unknown function values _ψ n_( _k j_), the unknown _E n_, as well as the unknown functional dependence of _ψ n_( _k_ ). We eliminate the functional dependence of _ψ n_( _k_ ) by restricting the solution to the same values of _k = k i_ as used in the approximation of the integral, which means that we are solving for wave function for _k_ values on the grid of Figures 26.2. This leads to a set of _N_ coupled linear equations in ( _N_ \+ 1) unknowns:

(26.8)

As a concrete example, for _N_ = 2 we have two simultaneous linear equations

**Figure 26.2** The grid of momentum values on which the integral equation is solved.

Of course, realistic examples require more than two integration points for precision.

We write our coupled equations (26.8) in the matrix form as

(26.9)

or as explicit matrices

(26.10)

Equation 26.9 is the matrix representation of the Schrödinger equation (26.3). The wave function _ψ n_( _k_ ) on the grid is the _N_ × 1 vector

(26.11)

The astute reader may be questioning the possibility of solving _N_ equations for ( _N_ \+ 1) unknowns, _ψ n_( _k i_), and _E n_. Only sometimes, and only for certain values of _E n_ (eigenvalues), will the computer be able to find solutions. To see how this arises, we try to apply the matrix inversion technique (which we will use successfully for scattering in Section 26.3). We rewrite (26.9) as

(26.12)

and multiply both sides by the inverse of [ **H** – _E nI_] to obtain the formal solution

(26.13)

This equation tells us that (1) if the inverse exists, then we have the _trivial_ solution _ψ n_ ≡ 0, which is not revealing, and (2) for a nontrivial solution to exist, our assumption that the inverse exists must be incorrect. Yet we know from the theory of linear equations that the inverse fails to exist when the determinant vanishes:

(26.14)

Equation 26.14 is the _N_ \+ 1th equation needed to find unique solutions to the eigenvalue problem. Although, there is no guarantee that solutions of (26.14) can always be found, if they are found they are the desired _eigenvalues_ of (26.9).

### 26.2.2 Delta-Shell Potential (Model)

To keep things simple and to have an analytic answer to compare with, we consider the local delta-shell potential:

(26.15)

This might be a good model for an interaction that occurs when two particles are predominantly a fixed distance _b_ apart. We use (26.4) to determine its momentum–space representation:

(26.16)

_Beware:_ We have chosen this potential because it is easy to evaluate the momentum–space matrix element of the potential. However, its singular nature in _r_ space leads to (26.16) having a very slow falloff in _k_ space, and this causes the integrals to converge so slowly that numerics are not as precise as we would like.

If the energy is parameterized in terms of a wave vector _κ_ by _E n_ = _−κ_ 2/2 _μ_ , then for this potential there is, at most, one bound state and it satisfies the transcendental equation (Gottfried, 1966)

(26.17)

Note that bound states occur only for attractive potentials. For the present case, this requires _λ_ < 0.

**Exercise** Pick some values of _b_ and _λ_ and verify that (26.17) can be solved for _κ_.

### 26.2.3 Binding Energies Solution

An actual computation may follow two paths. The first calls subroutines to evaluate the determinant of the [ **H** − _E nI_] matrix in (26.14), and then to _search_ for those values of energy for which the computed determinant vanishes. This provides _E n_, but not wave functions. The other approach calls an eigenproblem solver that may give some or all eigenvalues and eigenfunctions. In both the cases, the solution is obtained iteratively, and you may be required to guess starting values for both the eigenvalues and eigenvectors. In Listing 26.1, we present our solution of the integral equation for bound states of the delta-shell potential using the NumPy matrix library and the `**gauss**` method for Gaussian quadrature points and weights.

**Listing 26.1** **Bound.py** solves the Lippmann–Schwinger integral equation for bound states within a delta-shell potential. The integral equations are converted to matrix equations using Gaussian grid points, and they are solved with linalg.

  1. Write a program, or modify ours, to solve the integral equation (26.9) for the delta-shell potential (26.16). Either find the _E n'_s for which the determinant vanishes _or,_ find the eigenvalues and eigenvectors for this **H**.
  2. Set the scale by setting 2 _μ_ = 1 and _b_ = 10.
  3. Set up the potential and Hamiltonian matrices **V** ( _i, j_ ) and **H** ( _i, j_ ) for Gaussian quadrature integration with at least _N_ = 16 grid points.
  4. Adjust the value and sign of _λ_ for bound states. A good approach is to start with a large negative value for _λ_ and then make it less negative. You should find that the eigenvalue moves up in energy.
  5. _Note:_ Your eigenenergy solver may return several eigenenergies. The true bound state will be at negative energy and change little as the number of grid points changes. The others are numerical artifacts.
  6. Try increasing the number of grid points in steps of 8, for example, 16, 24, 32, 64,..., and see how the energy changes.
  7. Extract the best value for the bound-state energy and estimate its precision by seeing how it changes with the number of grid points.
  8. If you are solving the eigenvalue problem, check your solution by comparing the RHS and LHS in the matrix multiplication [ **H** ][ _ψ n_] = _E n_[ _ψ n_].
  9. Verify that, regardless of the potential's strength, there is only a single bound state and that it gets deeper as the magnitude of _λ_ increases. Compare with (26.17).

### 26.2.4 Wave Function (Exploration)

  1. Determine the momentum–space wave function _ψ n_( _k_ ) using an eigen problem solver. Does _ψ n_( _k_ ) fall off at _k_ → ∞? Does it oscillate? Is it well behaved at the origin?
  2. Using the same points and weights as used to evaluate the integral in the integral equation, determine the coordinate-space wave function via the Bessel transform

(26.18)

Does _ψ n_( _r_ ) fall off as you would expect for a bound state? Does it oscillate? Is it well behaved at the origin?
  3. Compare the _r_ dependence of this _ψ n_( _r_ ) to the analytic wave function:

(26.19)

## 26.3 Scattering States of Nonlocal Potentials

**Problem** Again we have a particle interacting with the nonlocal potential discussed for bound states (Figure 26.3a), only now the particle has sufficiently high energy that it scatters from rather than binds with the medium. Your _problem_ is to determine the scattering cross section for scattering from a nonlocal potential.

**Figure 26.3** (a) A projectile (dark particle at _r_ ) scattering from a dense medium. (b) The same process viewed in the CM system where the projectile and target always have equal and opposite momenta.

## 26.4 Lippmann–Schwinger Equation (Theory)

Because experiments measure scattering amplitudes and not wave functions, it is more direct to have a theory dealing with amplitudes rather than wave functions.3) An integral form of the Schrödinger equation dealing with the scattering amplitude _R_ is the _Lippmann–Schwinger equation_ :

(26.20)

where the symbol   in (26.20) indicates the Cauchy principal-value prescription for avoiding the singularity arising from the zero of the denominator (we discuss how to do that next). As for the bound-state problem, this equation is for partial wave _l_ = 0 and _ _= 1. In(26.20), the momentum _k_ 0 is related to the energy _E_ and the reduced mass _μ_ by

(26.21)

The initial and final COM momenta _k_ and _k′_ are the momentum–space variables. The experimental observable that results from a solution of (26.20) is the diagonal matrix element _R_ ( _k_ 0, _k_ 0), which is related to the scattering phase shift _δ_ 0 and thus the cross section:

(26.22)

Note that (26.20) is not just the evaluation of an integral, it is an integral equation in which _R_ ( _p, k_ ) is integrated over all _p_. Yet because _R_ ( _p, k_ ) is unknown, the integral cannot be evaluated until after the equation is solved!

### 26.4.1 Singular Integrals (Math)

A _singular_ integral

(26.23)

is one in which the integrand _g_ ( _k_ ) is singular at a point _k_ 0 within the integration interval [ _a, b_ ], yet with the integral   remaining finite. (If the integral itself were infinite, we could not compute it.) Unfortunately, computers are notoriously bad at dealing with infinite numbers, and if an integration point gets too near the singularity, overwhelming subtractive cancellation or overflow may occur. Consequently, we apply some results from complex analysis before evaluating singular integrals numerically.4)

In Figure 26.4, we show three ways to avoid the singularity of an integrand. The paths in Figure 26.4a and b move the singularity slightly off the real _k_ -axis by giving the singularity a small imaginary part ±i _є_. The Cauchy principal-value prescription   in Figure 26.4c is seen to follow a path that "pinches" both sides of the singularity at _k_ 0, but does not to pass through it:

(26.24)

The preceding three prescriptions are related by the identity

(26.25)

which follows from Cauchy's residue theorem.

**Figure 26.4** Three different paths in the complex _k′_ plane used to evaluate line integrals when there are singularities. Here the singularities are at _k_ and _−k,_ and the integration variable is _k′._

### 26.4.2 Numerical Principal Values

A direct numerical evaluation of the principal value limit (26.24) is troublesome because of the large cancellations that occur near the singularity. A better algorithm follows from the mathematical theorem

(26.26)

This equation says that the curve of 1/( _k_ − _k_ 0) as a function of _k_ has equal and opposite areas on both sides of the singular point _k_ 0. If we break the integral up into one over positive _k_ and one over negative _k,_ a change of variable _k → −k_ permits us to rewrite (26.26) as

(26.27)

We observe that the principal-value exclusion of the singular point's contribution to the integral is equivalent to a simple subtraction of the zero integral (26.27):

(26.28)

Note that there is no   on the RHS of (26.28) because the integrand is no longer singular at _k = k_ 0 (it is proportional to the d _f_ / d _k_ ) and can therefore be evaluated numerically using the usual rules. The integral (26.28) is called the _Hilbert transform_ of _f_ and also arises in subjects like inverse problems.

### 26.4.3 Reducing Integral Equations to Matrix Equations (Method)

Now that we can handle singular integrals, we go back to reducing the integral equation (26.20) to a set of linear equations that can be solved with matrix methods. We start by rewriting the principal-value prescription as a definite integral (Haftel and Tabakin, 1970):

(26.29)

We convert this integral equation to linear equations by approximating the integral as a sum over _N_ integration points (usually Gaussian) _k j_ with weights _w j_:

(26.30)

We note that the last term in (26.30) implements the principal-value prescription and cancels the singular behavior of the previous term. Equation 26.30 contains the ( _N_ \+ 1) unknowns _R_ ( _k j_, _k_ 0) for _j_ = 0, _N_. We turn it into ( _N_ \+ 1) simultaneous equations by evaluating it for ( _N_ \+ 1) _k_ values on a grid (Figure 26.2) consisting of the observable momentum _k_ 0 and the integration points:

(26.31)

There are now ( _N_ \+ 1) linear equations for ( _N_ \+ 1) unknowns _R i_ ≡ _R_ ( _k i_, _k_ 0):

(26.32)

We express these equations in the matrix form by combining the denominators and weights into a single denominator vector _D_ :

(26.33)

The linear equations (26.32) now assume the matrix form

(26.34)

where _R_ and _V_ are _column vectors_ of length _N_ \+ 1:

(26.35)

We call the matrix [1 − _DV_ ] in (26.34) the wave matrix _F_ and write the integral equation as the matrix equation

(26.36)

With _R_ the unknown vector, (26.36) is in the standard form _AX_ = _B_ , which can be solved by the mathematical subroutine libraries discussed in Chapter 6.

### 26.4.4 Solution via Inversion, Elimination

An elegant (but alas not most efficient) solution to (26.36) is by matrix inversion:

(26.37)

Because the inversion of even complex matrices is a standard routine in mathematical libraries, (26.37) is a _direct solution_ for the _R_ amplitude. Unless you need the inverse for other purposes (like calculating wave functions), a more efficient approach is to use Gaussian _elimination_ to find an [ _R_ ] that solves [ _F_ ][ _R_ ] = [ _V_ ] without computing the inverse.

**Listing 26.2** **Scatt.py** solves the Lippmann–Schwinger integral equation for scattering froma delta-shell potential. The singular integral equations are regularized by a subtraction, converted to matrix equations using Gaussian grid points, and then solved with matrix library routines.

### 26.4.5 Scattering Implementation

For the scattering problem, we use the same delta-shell potential (26.16) discussed in Section 26.2.2 for bound states:

(26.38)

This is one of the few potentials for which the Lippmann–Schwinger equation (26.20) has an analytic solution (Gottfried, 1966) with which to check:

(26.39)

Our results were obtained with 2 _μ_ = 1, _λb_ = 15, and _b_ = 10, the same as in (Gottfried, 1966). In Figure 26.5, we give a plot of sin2 _δ_ 0 vs. _kb,_ which is proportional to the scattering cross section arising from the angular momentum _l_ = 0 phase shift. It is seen to reach its maximum values at energies corresponding to resonances. In Listing 26.2, we present our program for solving the scattering integral equation using the NumPy Linear Algebra matrix library and the `**gauss**` method for quadrature points. For your implementation:

  1. Set up the matrices `**V[], D[], and F[,]**`. Use at least _N_ = 16 Gaussian quadrature points for your grid.
  2. Calculate the matrix _F_ −1 using a library subroutine.
  3. Calculate the vector **_R_** by matrix multiplication **_R_** = _F_ −1 _V_.
  4. Deduce the phase shift _δ_ from the _i_ = 0 element of _R:_

(26.40)

  5. Estimate the precision of your solution by increasing the number of grid point in steps of two (we found the best answer for _N_ = 26). If your phase shift changes in the second or third decimal place, you probably have that much precision.
  6. Plot sin2 _δ_ vs. energy   starting at zero energy and ending at energies where the phase shift is again small. Your results should be similar to those in Figure 26.5. Note that a _resonance_ occurs when _δ l_ increases rapidly through π/2, that is, when sin2 _δ_ 0 = 1.
  7. Check your answer against the analytic results (26.39).

**Figure 26.5** The energy dependence of the cross section for _l_ = 0 scattering from an attractive delta-shell potential with _λb_ = 15. The dashed curve is the analytic solution (26.39), and the solid curve results from numerically solving the integral Schrödinger equation, either via direct matrix inversion or via LU decomposition.

### 26.4.6 Scattering Wave Function (Exploration)

The _F_ −1 matrix that occurred in our solution to the integral equation

(26.41)

is actually quite useful. In scattering theory, it is known as the _wave matrix_ because it is used in expansion of the wave function:

(26.42)

Here _N_ 0 is a normalization constant and the _R_ amplitude is appropriate for standing-wave boundary conditions. So once we know _F_ 1, we also know the wave function.

  1. Plot _u_ ( _r_ ) and compare it to a free wave.

1) We use natural units in which _ħ_ ≡ 1. 2) See Chapter 5, for a discussion of numerical integration. 3) To make the presentation simpler, but still perfectly valid, we solve for the _reaction matrix R,_ but call it the scattering amplitude. 4) Singh and Thompson (1993) describe a different approach using Maple and Mathematica.

# A
Codes, Applets, and Animations

**Table A.1** Python codes, Chapters 1–.

Name | Listing No. | Description | Name | Listing No. | Description
---|---|---|---|---|---
EasyVisual | 1.1 | Visual easy plot | 3GraphVisual | 1.2 | Visual multiplots
3Dshapes | 1.3 | Visual 3D shapes | EasyMatPlot | 1.4 | Matplot 2D
GradesMatPlot | 1.5 | Matplot multiplots | MatPlot2figs | 1.6 | Matplot multiplots
PondMatPlot | 1.7 | Matplot scatter plot | Simple3Dplot | 1.8 | Matplot surface
Scatter3dPlot | 1.9 | Matplot 3D scatter | EqHeatAnimate | 1.10 | Matplot animation
MayaLines | 1.12 | Mayavi flow lines | MayaYlm | 1.10 | Mayavi surface
Area | 2.1 | Simple screen I/O | AreaFormatted | 2.2 | Formatted I/O
Directives | 2.3 | I/O directives, escape | Limits | 2.4 | Machine precision
Bessel | 3.1 | Downward recursion | Walk | 4.1 | Random walk
Walk3D | 4.1 | 3D random walk | DecaySound | 4.2 | Spontaneous decay
TrapMethods | 5.1 | Trapezoid rule | IntegGauss | 5.2 | Gaussian quadrature
vonNeuman | 5.3 | von Neumann rejection | Eigen | 6.34 | Matrix eigenvalues
Matrix | 6.5 | Matrix array mult | NewtonNDanimate | 6.1 | N-D Newton–Raphson
Bisection | 7.1 | Bisection algorithm | NewtonCD | 7.2 | Newton–Raphson search
Lagrange | 7.5 | Lagrange interpolation | Spline | 7.3 | Spline fitting
SplineInteract | 7.3 | Interactive splines | Fit | 7.4 | Least-squares fitting
rk4 | 8.1 | rk4 ODE solver | rk45 | 8.2 | Adaptive step rk4
ABM | 8.3 | ABM ODE solver | QuantumNumerov | 9.1 | Schrödinger equation
QuantumEigen | 9.2 | Quantum eigen rk4 | ProjectileAir | 9.3 | Projectile with drag
TuneNumPy | 11.5 | NumPy vectors | Tune.f90 | 11.6 | Fortran tuning
Tune | 11.7 | Python tuning | Tune4 | 11.8 | Python loop unroll
tune4.f95 | 11.9 | Fortran loop unrolling | SumArraysCuda | 11.16 | CUDA GPU program
SumArraysCuda2 | 11.17 | GPU blocks | DFTcomplex | 12.1 | Complex DFT
DFTreal | 12.2 | Real DFT | FourierMatplot | 12.2 | Interactive DFT
NoiseSincFilter | 12.4 | Fourier filtering | FFT.py | 12.3 | Fast Fourier transform
FFTappl | 12.3 | FFT + graphs | CWT | 13.1 | Continuous wavelets
DWT | 13.2 | Discrete wavelets | Bugs | 14.1 | Logistic bifurcations
LyapLog | 14.2 | Lyapunov coefficient | Entropy | 14.3 | Shannon entropy
PredatorPrey | 14.4 | Population dynamics | Fern3D | 16.1 | 3D fern
Column | 16.2 | Column growth | Gameoflife | 16.3 | Game of life
Islands.pov | 16.4 | Ray tracing | Film | online | Film deposition
Coastline | online | Box counting | DLA | 16.7 | Aggregation
Fern | 16.1 | 1D fern | Sierpin | 16.2 | Sierpinsky gasket
IsingViz | 17.1 | Ising model | WangLandau | 17.2 | Wang–Landau MC
QMC | 17.3 | Quantum MC | QMCbouncer | 17.4 | QMC bouncer
MD | 18.1 | 1D MD | MD2D | 18.1 | 2D MD
LaplaceLine | 19.1 | Laplace equation | EqHeat | 20.1 | Heat equation solution
EqHeat | 20.1 | Heat equation | HeatCNTridiag | 20.2 | Better heat
EqStringAnimate | 21.1 | Wave equation | EqStringMatPlot.py | online | Waves with Matplot
Waves2D | 21.2 | 2D wave equation | Waves2Danal | online | Analytic membrane
HarmosAnimate | 22.1 | Quantum packet | FDTD | 22.2 | Finite difference time domain
CircPolarztn | 22.3 | FDTD circular |  |  |
LaplaceFEM_1D | 23.1 | Finite element 1D | LaplaceFEM_2D | 23.2 | Finite element 2D
AdvecLax | 24.1 | Advection equation | Soliton | 24.2 | KdeV solitons
SolitonAnimate | online | Soliton movie | Beam | 25.1 | Navier–Stokes equation
BeamContour | 25.4 | Flow contours | Bound | 26.1 | Integral equation eigen
Scatt | 26.2 | Integral equation scatter |  |  |

**Table A.2** Animations (A player such as VLC or QuickTime required formpeg and avi, and a Web browser for gifs).

**Directory** | **Chapter** | **Directory** | **Chapter**
---|---|---|---
DoublePendulum (see also applets) | 15 | Fractals (see also applets) | 16
MapleWaveMovie (requires Maple) | 21 | Laplace (DX movie) | 19
MD | 18 | TwoSlits | 22
2D solitons | 21,25 | Utilities (scripts, colormaps) |
Waves (animated gifs need browser) | 21 |  |

# Bibliography

  1. Abarbanel, H.D.I., Rabinovich, M.I., and Sushchik, M.M. (1993) _Introduction to Nonlinear Dynamics for Physicists_ , World Scientific, Singapore.
  2. Abramowitz, M. and Stegun, I.A. (1972) _Handbook of Mathematical Functions_ , 10th edn, US Govt. Printing Office, Washington.
  3. Addison, P.S. (2002) _The Illustrated Wavelet Transform Handbook_ , Institute of Physics Publishing, Bristol and Philadelphia.
  4. Allan, M.P. and Tildesley, J.P. (1987) _Computer Simulations of Liquids_ , Oxford Science Publications, Oxford.
  5. Amdahl, G. (1967) _Validity of the Single-Processor Approach to Achieving Large-Scale Computing Capabilities_ , Proc. AFIPS, p. 483.
  6. Ancona, M.G. (2002) _Computational Methods for Applied Science and Engineering_ , Rinton Press, Princeton.
  7. Anderson, E., Bai, Z., Bischof, C., Demmel, J., Dongarra, J., Du Croz, J., Greenbaum, A., Hammarling, S., McKenney, A., Ostrouchov, S., and Sorensen, D. (2013) _LAPACK Users' Guide_ , 3rd edn, SIAM, Philadelphia, www.netlib.org (accessed 22 March 2015).
  8. Anderson, J.A., Lorenz, C.D., and Travesset, A. (2008) HOOMD-blue, general purpose molecular dynamics simulations. _J. Comput. Phys_., 227 (10), 5342, codeblue.umich.edu/hoomd-blue (accessed 22 March 2015).
  9. Arfken, G.B. and Weber, H.J. (2001) _Mathematical Methods for Physicists_ , Harcourt/Academic Press, San Diego.
  10. Argyris, J., Haase, M., and Heinrich, J.C. (1991) _Comput. Methods Appl. Mech. Eng_ , **86** , 1.
  11. Armin, B. and Shlomo, H. (eds) (1991) _Fractals and Disordered Systems_ , Springer, Berlin.
  12. Askar, A. and Cakmak, A.S. (1977) _J. Chem. Phys_., **68** , 2794.
  13. Banacloche, J.G. (1999) A quantum bouncing ball. _Am. J. Phys_., **67** , 776.
  14. Barnsley, M.F. and Hurd, L.P. (1992) _Fractal Image Compression_ , A.K. Peters, Wellesley.
  15. Beazley, D.M. (2009) _Python Essential Reference_ , 4th edn, Addison-Wesley, Reading, MA, USA.
  16. Becker, R.A. (1954) _Introduction to Theoretical Mechanics_ , McGraw-Hill, New York.
  17. Bevington, P.R. and Robinson, D.K. (2002) _Data Reduction and Error Analysis for the Physical Sciences_ , 3rd edn, McGraw-Hill, New York.
  18. Bleher, S., Grebogi, C., and Ott, E. (1990) Bifurcations in chaotic scattering. _Physica D_ , **46** , 87.
  19. Briggs, W.L. and Henson, V.E. (1995) _The DFT, An Owner's Manual_ , SIAM, Philadelphia.
  20. Bunde, A. and Havlin, S. (eds) (1991) _Fractals and Disordered Systems_ , Springer, Berlin.
  21. Burgers, J.M. (1974) _The Non-Linear Diffusion Equation; Asymptotic Solutions and Stattistical Problems_ , Reidel, Boston.
  22. Car, R. and Parrinello, M. (1985) _Phys. Rev. Lett_., **55** , 2471.
  23. Cencini, M., Ceconni, F. and Vulpiani, A. (2010) _Chaos From Simple Models To Complex Systems_ , World Scientific, Singapore.
  24. Christiansen, P.L. and Lomdahl, P.S. (1981) _Physica D_ , **2** , 482.
  25. Christiansen, P.L. and Olsen, O.H. (1978) _Phys. Lett. A_ , **68** , 185; Christiansen, P.L. and Olsen, O.H. (1979) _Phys. Scr_., **20** , 531.
  26. Clark University (2011) _Statistical and Thermal Physics Curriculum Development Project_ , stp.clarku.edu/ (accessed 22 March 2015); _Density of States of the 2D Ising Model_.
  27. CPUG, Computational Physics degree program for Undergraduates (2009), physics.oregonstate.edu/CPUG (accessed 22 March 2015).
  28. Crank, J. and Nicolson, P. (1946) _Proc. Cambridge Philos. Soc_., **43** , 50.
  29. Cooley, J.W. and Tukey, J.W. (1965) _Math. Comput_., **19** , 297.
  30. Courant, R., Friedrichs, K., and Lewy, H. (1928) _Math. Ann_., **100** , 32.
  31. Critchley, S. (2014) _The Dangers of Certainty: A Lesson from Auschwitz_ , New York Times, New York.
  32. Danielson, G.C. and Lanczos, C. (1942) _J.Franklin Inst_., **233** , 365.
  33. Daubechies, I. (1995) _Wavelets and other phase domain localization methods_ , Proc. Int. Congr. Math., **1, 2,** Basel, 56, Birkhäuser, Basel.
  34. DeJong, M.L. (1992) Chaos and the simple pendulum. _Phys. Teach_., **30** , 115.
  35. Dongarra, J. (2011) _On the Future of High Performance Computing: How to Think for Peta and Exascale Computing_ , Conference on Computational Physics 2011, Gatlinburg; _Emerging Technologies for High Performance Computing_ , GPU Club presentation, University of Manchester, www.netlib.org/utk/people/JackDongarra/SLIDES/gpu-0711.pdf (accessed 22 March 2015).
  36. Dongarra, J., Sterling, T., Simon, H., and Strohmaier, E. (2005) High-performance computing. _Comput. Sci. Eng., **7**_ , 51.
  37. Dongarra, J., Hittinger, J., Bell, J., Chacson, L., Falgout, R., Heroux, M., Hovland, P., Ng, E., Webster, C., and Wild, S. (2014) _Applied Mathematics Research for Exascale Computing_ , US Department of Energy Report, <http://www.osti.gov/bridge> (accessed 22 March 2015).
  38. Donnelly, D. and Rust, B. (2005) The fast Fourier transform for experimentalists. _Comput. Sci. Eng., 7_ , 71.
  39. Eclipse an open development platform (2014) www.eclipse.org (accessed 22 March 2015).
  40. Ercolessi, F. (1997) A molecular dynamics primer, www.ud.infn.it/~ercolessi/md/ (accessed 22 March 2015).
  41. Faber, R. (2010) CUDA, Supercomputing for the Masses: Part 15, www.drdobbs.com/architecture-and-design/cuda-supercomputing-for-the-masses-part/222600097 (accessed 22 March 2015).
  42. Falkovich, G. and Sreenivasan, K.R. (2006) Lesson from hydrodynamic turbulence. _Phys. Today_ , **59** , 43.
  43. Family, F. and Vicsek, T. (1985) _J. Phys. A_ , **18** , L75.
  44. Feigenbaum, M.J. (1979) _J. Stat. Phys_., **21** , 669.
  45. Fetter, A.L. and Walecka, J.D. (1980) _Theoretical Mechanics of Particles and Continua_ , McGraw-Hill, New York.
  46. Feynman, R.P. and Hibbs, A.R. (1965) _Quantum Mechanics and Path Integrals_ , McGraw-Hill, New York.
  47. Fitzgerald, R. (2004) New experiments set the scale for the onset of turbulence in pipe flow. _Phys. Today_ , **57** , 21.
  48. Fosdick L.D., Jessup, E.R. Schauble, C.J.C., and Domik, G. (1996) _An Introduction to High Performance Scientific Computing_ , MIT Press, Cambridge.
  49. Fox, G. (1994) _Parallel Computing Works_!. Morgan Kaufmann, San Diego.
  50. Gara, A., Blumrich, M.A., Chen, D., Chiu, G.L.-T., Coteus, P., Giampapa, M.E., Haring, R.A., Heidelberger, P., Hoenicke, D., Kopcsay, G.V., Liebsch, T.A., Ohmacht, M., Steinmacher-Burow, B.D., Takken, T., and Vranas, P. (2005) Overview of the Blue Gene/L system architecure. _IBM J. Res Dev_., **49** , 195; Feldman, M., IBM Specs Out Blue Gene/Q Chip, (2011) _HPC Wire_ , August 22 2011.
  51. Garcia, A.L. (2000) _Numerical Methods for Physics_ , 2nd edn, Prentice-Hall, Upper Saddle River, NJ, USA.
  52. Gibbs, R.L. (1975) The quantum bouncer. _Am. J. Phys_., 43, 25.
  53. Gnuplot (2014) gnuplot homepage www.gnuplot.info (accessed 22 March 2015).
  54. Goldberg, A., Schey, H.M., and Schwartz, J.L. (1967) Computer-generated motion pictures of one-dimensional quantum-mechanical transmission and reflection phenomena. _Am. J. Phys_., **35** , 177–186.
  55. Goodings, D.A. and Szeredi, T. (1992) The quantum bouncer by the path integral method. _Am.j.Phys_., **59** , 924.
  56. Goswani, J.C. and Chan, A.K. (1999) _Fundamentals of Wavelets_ , John Wiley & Sons, New York.
  57. Gottfried, K. (1966) _Quantum Mechanics_ , Benjamin, New York.
  58. Gould, H., Tobochnik, J., and Christian, W. (2006) _An Introduction to Computer Simulations Methods_ , 3rd edn, Addison-Wesley, Reading, USA.
  59. Graps, A. (1995) An introduction to wavelets. _Comput. Sci. Eng_., **2** , 50.
  60. Gurney, W.S.C. and Nisbet, R.M. (1998) _Ecological Dynamics_ , Oxford University Press, Oxford.
  61. Haftel, M.I. and Tabakin, F. (1970) _Nucl. Phys_., **158** , 1.
  62. Hardwich, J. (1996) Rules for Optimization, www.cs.cmu.edu/~jch/java (accessed 22 March 2015).
  63. Hartmann, W.M. (1998) _Signals, Sound, and Sensation_ , AIP Press, Springer, New York.
  64. Higgins, R.J. (1976) Fast Fourier transform: An introduction with some minicomputer experiments. _Am. J. Phys_., **44** , 766.
  65. Hildebrand, F.B. (1956) _Introduction to Numerical Analysis_ , McGraw-Hill, New York.
  66. Hinsen, K. (2013) Software development for reproducible research. _Comput. Sci. Eng_ , **4** (15), 60–63, www.computer.org/portal/web/cise/home (accessed 22 March 2015).
  67. History of Python (2009) The History of Python python-history.blogspot.com/2009/01/brief-timeline-of-python.html (accessed 22 March 2015).
  68. Hockney, R.W. and J.W Eastwood (1988) _Computer Simulation Using Particles_ , Adam Hilger, Bristol.
  69. Hubble, E. (1929) A relation between distance and radial velocity among extra-galactic nebulae. _Proc. Natl. Acad. Sci. USA_ , 15 (3), 168.
  70. Hunag, K. (1987) _Statistical Mechanics_ , John Wiley & Sons, New York.
  71. Jackson, J.D. (1988) _Classical Electrodynamics_ , 3rd edn, John Wiley & Sons, New York.
  72. Jackson, J.E. (1988) _A User's Guide to Principal Components_ , John Wiley & Sons, New York.
  73. Jolliffe, IY. (2001) _Principal Component Analysis_ , 2nd edn, Springer, New York.
  74. José, J.V. and Salatan, E.J. (1988) _Classical Dynamics_ , Cambridge University Press, Cambridge.
  75. Kennedy, R. (2006) _The case of Pollock's Fractals Focuses on Physics_ , New York Times, 2, 5 December 2006.
  76. Kirk, D. and Wen-Mei, WH. (2013) _Programming Massively Parallel Processors_ , 2nd edn, Morgan Kauffman, Waltham.
  77. Kittel, C. (2005) _Introduction to Solid State Physics_ , 8th edn, John Wiley & Sons, Inc., Hoboken.
  78. Klöckner, A. (2014) PyCUDA, mathema.tician.de/software/pycuda (accessed 22 March 2015).
  79. Koonin, S.E. (1986) _Computational Physics_ , Benjamin, Menlo Park, CA.
  80. Korteweg, D.J. and deVries, G. (1895) _Philos. Mag_ , **39** , 4.
  81. Kreyszig, E. (1998) _Advanced Engineering Mathematics_ , 8th edn, John Wiley Sons, New York.
  82. Lamb, H. (1993) _Hydrodynamics_ , 6th edn, Cambridge University Press, Cambridge.
  83. Landau, D.P. and Wang, F. (2001) Determining the density of states for classical statistical models: A random walk algorithm to produce a flat histogram. _Phys. Rev. E_ , **64** , 056101; Landau, D.P., Tsai, S.-H., and Exler, M. (2004) A new approach to Monte Carlo simulations in statistical physics: Wang–Landau sampling. _Am. J. Phys_., **72** , 1294.
  84. Landau, L.D. and Lifshitz, E.M. (1987) _Fluid Mechanics_ , 2nd edn, Butterworth-Heinemann, Oxford.
  85. Landau, L.D. and Lifshitz, E.M. (1976) _Quantum Mechanics_ , Pergamon, Oxford.
  86. Landau, L.D. and Lifshitz, E.M. (1976) _Mechanics_ , 3rd edn, Butterworth-Heinemann, Oxford.
  87. Landau, R.H. (2008) Resource letter CP-2: Computational physics. _Am. J. Phys_., **76** , 296.
  88. Landau, R.H. (2005) _A First Course in Scientific Computing_ , Princeton University Press, Princeton.
  89. Landau, R.H. (1996) _Quantum Mechanics II, A Second Course in Quantum Theory_ , 2nd edn, John Wiley & Sons, New York.
  90. Lang, W.C. and Forinash, K. (1998) Time-frequency analysis with the continuous wavelet transform. _Am. J. Phys_., **66** , 794.
  91. Langtangen, H.P. (2008) _Python Scripting for Computational Science_ , Springer, Heidelberg.
  92. Langtangen, H.P. (2009) _A Primer on Scientific Programming with Python_ , Springer, Heidelberg.
  93. Li, Z. (2014) Numerical Methods for Partial Differential Equations – Finite Element Method, www4.ncsu.edu/~zhilin/ (accessed 22 March 2015).
  94. Lorenz, E.N. (1963) Deterministic non-periodic flow./. _Atmos. Sci_., **20** , 130.
  95. Lotka, A.J. (1925) _Elements of Physical Biology_ , Williams and Wilkins, Baltimore.
  96. MacKeown, P.K. (1985) _Am. J. Phys_., **53** , 880.
  97. MacKeown, P.K. and Newman, D.J. (1987) _Computational Techniques in Physics_ , Adam Hilger, Bristol.
  98. Maestri, J.J.V., Landau, R.H., and Páez, M.J. (2000) Two-particle Schrödinger equation animations of wave packet-wave packet scattering. _Am. J. Phys_., **68** , 1113; <http://physics.oregonstate.edu/~rubin/nacphy/ComPhys/PACKETS/>.
  99. Mallat, P.G. (1982) A theory for multireso-lution signal decomposition: The wavelet representation. _IEEE Trans. Pattern Anal. Mach. Intell_., **11** (7), 674.
  100. Mandelbrot, B. (1967) How long is the coast of Britain? _Science_ , **156** , 638.
  101. Mandelbrot, B. (1982) _The Fractal Geometry of Nature_ , Freeman, San Francisco.
  102. Manneville, P. (1990) _Dissipative Structures and Weak Turbulence_ , Academic Press, San Diego.
  103. Mannheim, P.D. (1983) The physics behind path integrals in quantum mechanics. _Am. J. Phys_., 51, 328.
  104. Marion, J.B. and Thornton, S.T. (2003) _Classical Dynamics of Particles and Systems_ , 5th edn, Harcourt Brace Jovanovich, Orlando.
  105. Mathews, J. (2002) _Numerical Methods for Mathematics, Science and Engineering_ , Prentice-Hall, Upper Saddle River.
  106. Metropolis, M., Rosenbluth, A.W., Rosen-bluth, M.N., Teller, A.H., and Teller, E. (1953) _J. Chem. Phys_., **21** , 1087.
  107. Moon, F.C. and Li, G.-X. (1985) _Phys. Rev. Lett_., **55** , 1439.
  108. Morse, P.M. and Feshbach, H. (1953) _Methods of Theoretical Physics_ , McGraw-Hill, New York.
  109. Motter, A. and Campbell, D. (2013) Chaos at fifty. _Phys. Today_ , **66** (5), 27.
  110. Nelson, M., Humphrey, W., Gursoy, A., Dalke, A., Kalé, L., Skeel, R.D., and Schulten, K. (1996) NAMD – Scalable Molecular Dynamics. _J. Supercomput. Apps. High Perform. Comput_., **10** , 251–268, www.ks.uiuc.edu/Research/namd (accessed 22 March 2015).
  111. Nesvizhevsky, V.V., Borner, H.G., Petukhov, A.K., Abele, H., Baessler, S., Ruess, F.J., Stoferle, T., Westphal, A., Gagarski, A.M., Petrov, G.A., and Strelkov, A.V. (2002) Quantum states of neutrons in the Earth's gravitational field. _Nature_ , **415** , 297.
  112. NIST Digital Library of Mathematical Functions (2014) dlmf.nist.gov/ (accessed 22 March 2015).
  113. Numerical Python (2013) NumPy numpy.scipy.org (accessed 22 March 2015).
  114. NumPy Tutorial, Tentative (2015) Tentative NumPy Tutorial wiki.scipy.org/Tentative_NumPy_Tutorial (accessed 22 March 2015).
  115. Oliphant, T.E. (2006) _Guide to NumPy_ , csc.ucdavis.edu/~chaos/courses/nlp/Software/NumPyBook.pdf (accessed 22 March 2015).
  116. Ott, E. (2002) _Chaos in Dynamical Systems_ , Cambridge University Press, Cambridge.
  117. Otto A. (2011) Numerical Simulations of Fluids and Plasmas, how.gi.alaska.edu/ao/sim (accessed 22 March 2015).
  118. Pancake, C.M. (1996) Is parallelism for you?, _Comput. Sci. Eng_., **3** , 18.
  119. Peitgen, H.-O., Jürgens, H., and Saupe, D. (1992) _Chaos and Fractals_ , Springer, New York.
  120. Penna, T.J.P. (1994) _Comput. Phys_., **9** , 341.
  121. Perez, F., Granger, B.E. and Hunter, J.D. (2010) Python: An Ecosystem for Scientifc Computing. _Comput. Sci. Eng_., **13** (2), www.computer.org/web/computingnow/cise (accessed 22 March 2015).
  122. Perlin, K. (1985) An Image Synthesizer, _Computer Graphics_ (Proceedings of ACM SIG-GRAPH 85) **24** , 3.
  123. Phatak, S.C. and Rao, S.S. (1995) Logistic map: A possible random-number generator. _Phys. Rev. E_ , **51** , 3670.
  124. Plischke, M. and Bergersen, B. (1994) _Equilibrium Statistical Physics_ , 2nd edn, World Scientific, Singapore.
  125. Polikar, R. (2001) The Wavelet Tutorial, users.rowan.edu/~polikar/WAVELETS/WTtutorial.html (accessed 22 March 2015).
  126. Polycarpou, A.C. (2006) _Introduction to the Finite Element Method in Electromagnetics_ , Morgan and Claypool, San Rafael.
  127. Potvin, J. (1993) _Comput. Phys., **7**_ , 149. (2013) Pov-Ray, Persistence of Vision Ray-tracer, www.povray.org (accessed 22 March 2015).
  128. Press, W.H., Flannery, B.P., Teukolsky, S.A., and Vetterling, W.T. (1994) _Numerical Recipes_ , Cambridge University Press, Cambridge.
  129. Python (2014) Python for Programmers, <https://wiki.python.org/moin/BeginnersGuide/Programmers> (accessed 22 March 2015).
  130. LearnPython.org (2014) Interactive Python Tutorial, <http://www.learnpython.org/> (accessed 22 March 2015).
  131. (2014)The Python Tutorial, docs.python.org/2/tutorial/ (accessed 22 March 2015).
  132. (2014) Python Index of Packages, pypi.python.org/pypi (accessed 22 March 2015).
  133. (2014) Python Documentation, www.python.org/doc (accessed 22 March 2015).
  134. Quinn, M.J. (2004) _Parallel Programming in C with MPI and OpenMP_ , McGraw-Hill, New York.
  135. Ramasubramanian, K. and Sriram, M.S. (2000) A comparative study of computation of Lyapunov spectra with different algorithms. _Physica D_ , **139** , 72.
  136. Rapaport, D.C. (1995) _The Art of Molecular Dynamics Simulation_ , Cambridge University Press, Cambridge.
  137. Rasband, S.N. (1990) _Chaotic Dynamics of Nonlinear Systems_ , John Wiley & Sons, New York.
  138. Rawitscher, G., Koltracht, I., Dai, H., and Ribetti, C. (1996) _Comput. Phys_., **10** , 335.
  139. Reddy, J.N. (1993) _An Introduction to the Finite Element Method_ , 2nd edn, McGraw-Hill, New York.
  140. Refson, K. (2000) Moldy, A General-Purpose Molecular Dynamics Simulation Program, cc-ipcp.icp.ac.ru/Moldy_2_16.html (accessed 22 March 2015).
  141. Reynolds, O. (1883) _Proc. R. Soc. Lond_., **35** , 84.
  142. Richardson. L.F. (1961) Problem of contiguity: an appendix of statistics of deadly quarrels. _General Syst. Yearbook_ , **6** , 139.
  143. Rowe, A.C.H. and Abbott, P.C. (1995) Daubechies wavelets and mathematica. _Comput. Phys_., **9** , 635.
  144. Russell, J.S. (1844) _Report of the 14th Meeting of the British Association for the Advancement of Science_ , John Murray, London.
  145. Sander, E., Sander, L.M., and Ziff, R.M. (1994) _Comput. Phys_., **8** , 420.
  146. Sanders, J. and Kandrot, E. (2011) _Cuda by Example_ , Addison Wesley, Upper Saddle River.
  147. Satoh, A. (2011) _Introduction to Practice of Molecular Simulation_ , Elsevier, Amsterdam.
  148. Scheck, F. (1994) _Mechanics, from Newton's Laws to Deterministic Chaos_ , 2nd edn, Springer, New York.
  149. Shannon, C.E. (1948) A mathematical theory of communication. _Bell Syst. Tech.J_., **27** , 379.
  150. (2014) SciPy, a Python-based ecosystem, www.scipy.org (accessed 22 March 2015).
  151. Shaw C.T. (1992) _Using Computational Fluid Dynamics_ , Prentice-Hall, Englewood Cliffs, NJ.
  152. Singh, P.P. and Thompson, W.J. (1993) _Comput. Phys., **7**_ , 388.
  153. Sipper, M. (1997) _Evolution of Parallel Cellular Machines_ , Springer, Heidelberg, cell-auto.com (accessed 22 March 2015).
  154. Smith, D.N. (1991) _Concepts of Object-Oriented Programming_ , McGraw-Hill, New York.
  155. Smith, L.I. (2002) A Tutorial on Principal Components Analysis, www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf (accessed 22 March 2015).
  156. Smith, S.W. (1999) _The Scientist and Engineer's Guide to Digital Signal Processing_ , California Technical Publishing, San Diego.
  157. Stetz, A., Carroll, J., Chirapatpimol, N., Dixit, M., Igo, G., Nasser, M., Ortendahl, D., and Perez-Mendez, V. (1973) _Determination of the Axial Vector Form Factor in the Radiative Decay of the Pion_ , LBL 1707.
  158. Sullivan, D. (2000) _Electromagnetic Simulations Using the FDTD Methods_ , IEEE Press, New York.
  159. Tabor, M. (1989) _Chaos and Integrability in Nonlinear Dynamics_ , John Wiley & Sons, New York.
  160. Taflove, A. and Hagness, S. (2000) _Computational Electrodynamics: The Finite Difference Time Domain Method_ , 2nd edn, Artech House, Boston.
  161. Tait, R.N., Smy, T., and Brett, M.J. (1990) _Thin Solid Films_ , **187** , 375.
  162. Thijssen J.M. (1999) _Computational Physics_ , Cambridge University Press, Cambridge.
  163. Thompson, W.J. (1992) _Computing for Scientists and Engineers_ , John Wiley & Sons, New York.
  164. Tickner, J. (2004) Simulating nuclear particle transport in stochastic media using Perlin noise functions. _Nucl. Instrum. Methods B_ , **203** , 124.
  165. Vallée, O. (2000) Comment on a quantum bouncing ball. _Am. J. Phys_., **68** , 672.
  166. van de Velde, E.F. (1994) _Concurrent Scientific Computing_ , Springer, New York.
  167. van den Berg, J.C. (ed.) (1999) _Wavelets in Physics_ , Cambridge University Press, Cambridge.
  168. Vano, J.A., Wildenberg, J.C., Anderson, M.B., Noel, J.K., and Sprott, J.C. (2006) Chaos in low-dimensional Lotka–Volterra models of competition. _Nonlinearity_ , **19** , 2391–2404.
  169. Visscher, P.B. (1991) _Comput. Phys_., 5, 596.
  170. Vold, M.J. (1959)J. _Colloid Sci_., **14** , 168.
  171. Volterra, V. (1926) Variazioni e fluttuazioni del numero d'individui in specie animali conviventi. _Mem. R. Accad. Naz. Lincei. Ser. VI_ , 2.
  172. Warburton, R.D.H. and Wang, J. (2004) Analysis of asymptotic projectile motion with air resistance using the Lambert W function. _Am. J.Phys_., **72** , 1404.
  173. Ward, D.W. and Nelson, K.A. (2005) Finite difference time domain, FDTD, simulations of electromagnetic wave propagation using a spreadsheet. _Comput. Appl. Eng. Educat_ , **13** (3), 213–221.
  174. Whineray, J. (1992) An energy representation approach to the quantum bouncer. _Am. J. Phys_., **60** , 948.
  175. (2014) _Principal component analysis_ , en.wikipedia.org/wiki/Principal_component_analysis (accessed 22 March 2015).
  176. Williams, G.P. (1997) _Chaos Theory Tamed_ , Joseph Henry Press, Washington.
  177. Witten, T.A. and Sander, L.M. (1981) _Phys. Rev. Lett_., **47** , 1400; Witten, T.A. and Sander, L.M. (1983) _Phys. Rev. B_ , 27, 5686.
  178. Wolf, A., Swift, J.B., Swinney, H.L., and Vastano, J.A. (1985) Determining Lyapunov exponents from a time series. _Physica D_ , **16** , 285.
  179. Wolfram S. (1983) Statistical mechanics of cellular automata. _Rev. Mod. Phys_., 55, 601.
  180. Yang, C.N. (1952) The spontaneous magnetization of a two-dimensional Ising model. _Phys. Rev_., **85** , 809.
  181. Yee, K. (1966) _IEEE Trans. Ant. Propagat_., **AP-14** , 302.
  182. Yue, K., Fiebig, K.M., Thomas, P.D., Chan, H.S., Shakhnovich, E.I., and Dill, A. (1995) _Proc. Natl. Acad. Sci. USA_ , **92** , 325.
  183. Zabusky, N.J. and Kruskal, M.D. (1965) _Phys. Rev. Lett_., **15** , 240.
  184. Zeller, C. (2008) High Performance Computing with CUDA, www.nvidia.com/object/sc10_cuda_tutorial.htmlP (accessed 22 March 2015).

# Index

## _a_

  * Accuracy
  * Adams–Bashforth–Moulton
  * Advection
  * Airy function
  * Algorithm
  * Alias
  * Amdahl's law
  * Analog filter
  * Animation
  * Antiferromagnet
  * Applet
  * Architecture _see also_ Memory
  * Arithmetic unit
  * Asymptote
  * Attractor
    * – predictable
    * – strange
  * Autocorrelation function

## _b_

  * Backtracking
  * Ballistic deposition
    * – correlated
  * Bandwidth
  * Base
  * Basic machine language
  * Beating
  * Beowulf
  * Bessel function
  * Bias
  * Bifurcation
    * – diagram
    * – dimension of
  * Binary number
  * Binary point
  * Binning
  * Bisection algorithm
  * Bit
    * – reversal
  * Blue Gene _see also_ IBM Blue Gene
  * Boltzmann distribution
  * Boolean
  * Bound state
  * Boundary condition
  * Box counting
  * Box-Muller method
  * Break command
  * Broadcasting
  * Buffer
  * Burgers' equation
  * Bus
  * Butterfly operation
  * Byte
    * – code

## _c_

  * Clanguage
  * Cache
    * – data
    * – misses
    * – programming
  * Canonical ensemble
  * Capacitor
  * Catenary
  * Cauchy principal value
  * Cellular automata
  * Central
    * – difference
    * – processing unit _see also_ CPU
    * – storage
  * Chaos
    * – Fourier analysis of
    * – of pendulum
    * – phase space
  * Chi-squared measure
  * CISC
  * Column-major order
  * Command-line interpreter
  * Communication
    * – time
  * Compiler
    * – just-in-time
  * Complex number
  * Compression
    * – lossless
    * – PCA
    * – wavelets
  * Computational
    * – physics
    * – science
    * – thinking
  * Computer language
  * Control structure
  * Convolution
  * Conway's Game of Life
  * Correlation
    * – auto
    * – coefficient
    * – growth
    * – PCA
  * Courant stability condition
  * Course grain parallel
  * Covariance
  * CPU
    * – design
    * – RISC
    * – time
  * Crank–Nicolson method
  * Cubic spline _see also_ Spline
  * Cumulative distribution
  * Curie temperature
  * Curve fitting _see_ Data fitting; Data fitting
  * Cycle time

## _d_

  * Data
    * – cache _see_ Cache
    * – compression
    * – dependency
    * – fitting
    * – parallel
    * – shared
    * – stream
    * – type
  * Data parallelism
  * Deadlock
  * Decay
    * – exponential
    * – simulation
    * – spontaneous
  * Density of states
  * Dependency
  * Deposition
    * – ballistic
    * – correlated ballistic
  * Derivative
    * – central difference
    * – forward difference
    * – second
  * DFT
  * Differential equation
    * – algorithm
    * – boundary condition
    * – dynamical form
    * – Euler's rule
    * – initial condition
    * – order
    * – partial _see also_ PDE
    * – Runge–Kutta algorithm
    * – type
  * Differential equations
    * – type
  * Differentiation
  * Diffusion-limited aggregation
  * Digital
  * Dimension
    * – array
    * – fractional
    * – Hausdorf–Besicovitch
    * – physical
    * – scheme
  * Discrete Fourier transform
  * Dispersion
    * – relation
  * Distributed memory
  * Domain decomposition
  * Double
    * – pendulum
    * – precision
  * Drag _see also_ friction
  * DRAM
  * Driving force
  * Duffing oscillator

## _e_

  * Eigenvalue
  * Electrostatic potential
  * Elliptic integral
  *   * Entropy
  * Equation
    * – Burgers'
    * – differential
    * – discrete
    * – integral _see also_ Integral
    * – Korteweg–de Vries
    * – motion
    * – Schrödinger
    * – Van der Pool
  * Ergodic
  * Error
    * – algorithmic
    * – approximation _see also_ algorithmic
    * – empirical
    * – integration
    * – minimum
    * – multiplicative
    * – N-D integration
    * – random
    * – roundoff
    * – total
    * – type
  * Euler's rule
  * Exchange energy
  * Executive
    * – system
    * – unit
  * Exponential decay
  * Extinction
  * Extrapolated difference

## _f_

  * Fast Fourier transform _see_ FFT
  * Feigenbaum constants
  * Ferromagnet
  * Fetch
  * Feynman
    * – path integrals
    * – postulates
    * – propagator
  * FFT
  * Filters
    * – analog
    * – digital
    * – sinc
    * – windowed
  * Fine grain parallel
  * Finite
    * – 2D element
    * – difference
    * – difference equation
    * – difference time domain
    * – element
  * Fitting
    * – best
    * – global
    * – goodness
    * – least squares
    * – linear
    * – linear least square
    * – local
    * – Newton–Raphson
    * – nonlinear
  * Fixed points in maps
  * Fixed-point number
  * Fixed-point numbers
  * Float _see_ Floating-point; Floating
  * Floating-point number
  * Floating-point numbers
  * FLOPS
  * Fluid Dynamics
  * Fortran
    * – _vs_ Python
  * Fourier
    * – analysis
    * – autocorrelation relation
    * – chaos
    * – decompositon
    * – discrete transform _see also_ Discrete
    * – fast transform _see_ FFT
    * – integral
    * – PDE solution
    * – sawtooth
    * – series
    * – short-time transform
    * – theorem
    * – transform
  * Fractal
    * – coastline
    * – dimension _see also_ Dimension
    * – plant
    * – Pollock painting
    * – tree
  * Friction
    * – in oscillation
    * – in oscillations
    * – in waves
    * – in pendulum
    * – in projectile motion
  * Functional integration

## _g_

  * Galerkin decomposition
  * Game of Life
  *   * Garbage
  * Gaussian
    * – distribution
    * – elimination
    * – quadrature
    * – – derivation
  * Gibbs overshoot
  * Global array language
  * Global optimization
  * GPU _see_ Graphical processing unit
  * Granularity
  * Graphical processing unit
    * – programming
  * Green's function
  * Grid point
  * Growth model
  * Guest

## _h_

  * Half-wave function
  * Hamilton's principle
  * Hardware
  * Harmonics
  * Heat bath
  * Heat equation
  * Hénon-Heiles potential
  * High performance computing
  * Hilbert transform
  * HPC _see_ High performance computing
  * Huygens's principle
  * Hyperbolic point

## _i_

  * IBM Blue Gene
  * IEEE floating-point
  * Importance sampling
  * Initial condition
  * Input/Output (I/O)
  * Instruction
    * – stack
    * – stream
  * Integral equation
  * Integration
    * – error
    * – from splines
    * – Gaussian quadrature
    * – mapping point
    * – mean value
    * – Monte Carlo
    * – multi-dimensional
    * – rejection techniques for
    * – scaling
    * – Simpson's rule
    * – spline
    * – trapezoid rule
    * – variance reduction
    * – von Neumann rejection
  * Integro-differential equation
  * Intermittency
  * Interpolation
    * – Lagrange
    * – spline
  * Interpreter
  * Inverse matrix
  * Ising model
    * – 2D

## _j_

  * Jacobi method
  * Jacobian matrix
  * Just-in-time compiler

## _k_

  * Kernel
  * Korteweg–de Vries equation

## _l_

  * Lag time
  * Lagrange interpolation
  * Language
    * – compiled
    * – computer
    * – high-level
    * – interpreted
    * – Python
  * Languages
    * – BASIC
    * – compiled
  * Laplace's equation
  * Latency
  * Lattice computation
  * Lattice point _see_ Grid point
  * Lax–Wendroff algorithm
  * Leap frog _see_ Time stepping
  * Least-squares fitting
  * Length of coastline
  * Lifetime
  * Limit cycle
  * Linear
    * – algebra
    * – congruent method
    * – least square fitting
    * – regression
    * – superposition
  * Linux
  * Lippmann–Schwinger equation
  *   * Load
    * – balancing
    * – module
  * Logistic map
  * Loop unrolling
  * Lorenz attractor
  * Lotka–Volterra model
  * Lyapunov coefficients

## _m_

  * Machine
    * – number
    * – precision
  * Magnetic material
  * Mantissa
  * Master and slave
  * Matplotlib
  * Matrix
    * – column-major order
    * – computing
    * – diagonalization
    * – equation
    * – inversion
    * – storage
    * – subroutine library
    * – tri-diagonal
  * Maxwell's Equations
  * Mayavi
  * Mean value theorem
  * Memory
    * – architecture
    * – conflict
    * – distributed
    * – page
    * – virtual
  * Message passing
  * Metropolis algorithm
  * Microcanonical ensemble
  * Microcode
  * Miller's device
  * MIMD
  * Mode locking
  * Molecular dynamics
  * Momentum space
  * Monte Carlo
    * – error in
    * – integration
    * – simulation
    * – technique
  * Multiple-core processor
  * Multiresolution analysis
  * Multitasking

## _n_

  * NAN
  * Navier–Stokes equation
  * Newton–Cotes methods
  * Newton–Raphson
    * – algorithm
    * – with backtracking
  * Node
  * Noise
    * – Perlin
    * – reduction
  * Nonlinear
    * – dynamics
    * – limit cycle
    * – map
    * – ODE
    * – oscillation _see_ Oscillation
  * Nonlocal potentials
  * Nonstationary signal
  * Normal
    * – distribution
    * – mode expansion
    * – number
  * Number
    * – binary
    * – complex
    * – fixed-point
    * – floating-point
    * – hexadecimal
    * – IEEE
    * – machine
    * – normal
    * – octal
    * – range of
    * – representation of
    * – subnormal
    * – uniform
  * Numerov method
  * NumPy
    * – optimization
  * Nyquist criterion
  * Nyquist-Shannon interpolation

## _o_

  * Object
    * – code
  * Octal number
  * ODE
    * – second order
  * ODExc
    * – second order
  * One cycle population
  * Operand
  * Operating system
  * Optimization
  * Oscillation
    * – anharmonic
    * – damped
    * – double pendulum
    * – driven
    * – electromagnetic
    * – Fourier analysis of
    * – from error
    * – harmonic
    * – in phase space
    * – isochronous
    * – nonlinear
    * – of pendulum
    * – population
    * – quantum
  * Over relaxation _see_ Relaxation
  * Overdetermined
  * Overflow
  * Overhead

## _p_

  * Padding of signal
  * Page
    * – fault
  * Parallel computing
    * – granularity
    * – master, slave
    * – message passing
    * – perfect
    * – performance
    * – pipeline
    * – programming
    * – strategy
    * – subroutines
    * – synchronous
    * – type
  * Partial differential equation _see also_ PDE
  * Path integration
  * PDE
    * – elliptic
    * – explicit solution
    * – hyperbolic
    * – implicit solution
    * – nonlinear
    * – parabolic
    * – type
    * – weak form of
  * Pendulum
    * – analytic solution
    * – bifurcation diagram
    * – chaotic
    * – coupled
  * Performance _see_ Tuning
  * Period doubling _see also_ Bifurcation
  * Periodic boundary condition
  * Perlin noise
  * Phantom bit
  * Phase space
  * Phase transition
  * Phase-space
  * Pipelined CPU
  * Planetary motion
  * Plot
    * – surface
  * Poisson's equation
  * Population dynamics
  * Potential
    * – delta shell
    * – Lennard–Jones
    * – momentum space
  * Pov-Ray
  * Power
    * – PCA
    * – residue method
    * – spectrum
  * Precision
    * – empirical
    * – machine
    * – test
  * Predator–prey model
  * Predictor–corrector method
  * Principal
    * – components analysis
    * – value
    * – value integral
  * Problem
    * – by subject
    * – solving paradigm
  * Programming
    * – design
    * – for virtual memory
    * – parallel
    * – reproducible
    * – structured
  * Projectile motion
  * Propagator
  * Protein folding
  * Pseudocode
  * Pseudorandom _see_ Random numbers
  * Pulsons
  * Pyramid scheme
  * Python
    * – algebraic tools
    * – arrays
    * – Canopy
    * – distribution
    * – I/O
    * – language
    * – library
    * – linear algebra
    * – list
    * – package
    * – reference
    * – virtual machine
    * – Visual package
    * – _vs_ Fortran

## _q_

  * Quadrature
  * Quantum
    * – bouncer
    * – mechanics
    * – scattering

## _r_

  * Race condition
  * Radioactive decay
  * Radix
  * RAM
  * Random
    * – generator
    * – linear congruent
    * – nonuniform
    * – number
    * – pseudo
    * – self-avoiding walk
    * – sequence
    * – test
    * – walk
  * Ray tracing
  * Recursion
  * Register
    * – working
  * Rejection technique
  * Relaxation
  * Resonance
    * – nonlinear
  * Reynolds number
  * RISC
  * rkN
  * Romberg extrapolation
  * Root mean square
  * Roundoff errors
  * Row-major order
  * Runge–Kutta

## _s_

  * Sampling
    * – importance
  * Sawtooth function
  * Scalability
  * Scattering
  * Schrödinger equation
    * – time dependent
  * Searching _see also_ Trial and error
  * Section size
  * Secular equation
  * Seed
  * Seeds
  * Self
    * – affine connection
    * – affinity
    * – limiting
    * – similar
  * Separatrix
  * Serial computing
  * Series summation
  * Shannon Entropy
  * Shell
  * Shock wave
  * Sierpiński gasket
  * Sign bit
  * Signal processing
  * Significant figure (part)
  * SIMD
  * Simpson's rule
  * Simulation
  * Sinc filter
  * Sine-Gordon equation
  * Single precision
  * Singular integral
  * SISD
  * Slave
  * SMP
  * Soliton
    * – crossing
    * – KdeV
    * – ring
    * – sine-Gordon
    * – water wave
  * Spline
    * – cubic
    * – natural
  * Spontaneous decay
  * SRAM
  * Stable state
  *   * Statistical mechanics
  * Stochastic
  * Storage
  * Strange attractor
  * Stride
  * Subnormal number
  * Subroutine
    * – library/package
  * Subscript _see also_ Dimension
    * – scheme
  * Subtask
  * Subtractive cancelation
  * Successive over-relaxation _see_ Relaxation
  * Supercomputer
  * Swap space
  * Symmetric processor _see_ SMP

## _t_

  * Task
  * Texture
  * Thermodynamics
  * Three body problem
  * Time delay
  * Time stepping
  * Top-down programming
  * Transient
  * Trapezoid rule
  * Trial and error
  * Trivial solution
  * Tuning _see_ Optimization
  * Two cycle
  * Two's complement

## _u_

  * Uncertainty principle
  * Underflow
  * Uniform
    * – distribution
    * – sequence
    * – sequences
    * – test
    * – weight
  * Unix

## _v_

  * Van der Pool equation
  * Variance
    * – reduction
  * Vector
    * – field
  * Vector processor
  * Vectorization
  * Velocity-Verlet algorithm
  * Verlet algorithm
  * Virtual machine
  * Virtual memory
  * Viscosity
  * Visualization
    * – of vector
  * Volume rendering
  * von Neumann
    * – rejection
    * – stability assessment
  * Vorticity
  * VPython

## _w_

  * Wang–Landau Sampling (WLS)
  * Wave
    * – electromagnetic
    * – equation
    * – function
    * – on catenary
    * – on string
    * – packet
    * – shallow water
  * Wavelet
    * – basis
    * – continuous
    * – Daubechies
    * – discrete transform (DWT)
    * – multiresolution analysis
    * – pyramid scheme
    * – transform
  * Weak form of PDE
  * Windows
  * Word length
  * Working set size

# **WILEY END USER LICENSE AGREEMENT**

Go to www.wiley.com/go/eula to access Wiley's ebook EULA.

redpajama_set_name: RedPajamaBook