Answers.Body: <p>The main advantage of using spectral operations comes from the <a href="https://en.wikipedia.org/wiki/Convolution_theorem" rel="nofollow noreferrer">Covolution theorem</a>, which states that the <a href="https://en.wikipedia.org/wiki/Fourier_transform" rel="nofollow noreferrer">Fourier transform</a> of the convolution of two functions is equal to the pointwise multiplication of the individual Fourier transforms of those functions.<br />
  Mathematically:</p>
  <p><span class="math-container">$\mathscr{F}|\it{f} \; * \it{g}| = \mathscr{F}|\,\it{f}\,| \mathscr{F}|\,\it{g}\,|  $</span></p>
  <p>Of course multiplying is much more convenient than convolving, and the Fourier transform is also a well known operation, and fast to compute, so that does not represent a downside.</p>
  <p>In practice, when it comes to computer vision, the convolution theorem become fundamental for many kinds of analytic de noising and filtering methods. Instead of applying a kernel on an image on teh spatial domain using convolution, you can simply move to the frequency domain through the Fourier transform and multiplying both, image and kernel.</p>
  <p>Moreover, when moving to the frequency domain, it become easier to access specific characteristics of images. Sharp lines  and edges for example are characterized by high frequencies, so we can extract them with a simple masking operation, instead of having to came up with a tough kernel in the spatial space. This is the core idea of <a href="https://www.javatpoint.com/dip-high-pass-vs-low-pass-filters" rel="nofollow noreferrer">low and high pass filters</a>, below here an example, where you can see that by masking the center (low frequencies) area of the spectra of two images, we can easily mix low frequency properties (e.g. shadows) and high frequency properties (e.g. edges) of two images. The same operation in the spatial domain would be tough if not impossible.</p>
  <p><a href="https://i.sstatic.net/lIdmn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lIdmn.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-20T17:01:49.570Z
Answers.Text: It may be also worth mentioning graph neural networks. Some of them exploit the convolution theorem.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-20T17:54:59.413Z
Answers.Text: thanks for the hint, not an expert on graph networks but I'll check!
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-20T10:14:15.100Z
Answers.LastActivityDate: 2022-01-20T10:14:15.100Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>In the Fourier analysis, an image is represented by frequencies and phases. This allows to filter or transform certain frequencies or phases.</p>
  <p>For example, Alan Oppenheim and Jae Lim have shown that phases contain information about the object contours in the image. <a href="https://ieeexplore.ieee.org/document/1456290" rel="nofollow noreferrer">Their paper</a> was published in 1981, so it is poorly scanned, and the images there are not clear. Better examples are in the following books:</p>
  <ol>
  <li>Danilo P. Mandic, Vanessa Su Lee Goh. Complex Valued Nonlinear Adaptive Filters (2009)</li>
  <li>Igor Aizenberg. Complex-Valued Neural Networks with Multi-Valued Neurons (2011)</li>
  </ol>
  <p>Here is an example from the second book.</p>
  <p>Consider two images, &quot;Lenna&quot; and &quot;Airplane&quot;,</p>
  <p><a href="https://i.sstatic.net/P6ULV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/P6ULV.png" alt="enter image description here" /></a></p>
  <p>Take their Fourier transform, swap magnitudes and phases of their Fourier spectra, and then perform the inverse Fourier transform. The result is as follows.</p>
  <p>Magnitudes from the &quot;Airplane&quot;, and phases from the &quot;Lenna&quot;:</p>
  <p><a href="https://i.sstatic.net/hgtIP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hgtIP.png" alt="enter image description here" /></a></p>
  <p>Magnitudes from the &quot;Lenna&quot;, and phases from the &quot;Airplane&quot;:</p>
  <p><a href="https://i.sstatic.net/KI6lB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KI6lB.png" alt="enter image description here" /></a></p>
  <p>So, the information about objects is contained mainly in the phase part of the spectra.</p>
  <p>This is what happens when amplitudes are replaced by the constant 1:</p>
  <p><a href="https://i.sstatic.net/n31QE.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/n31QE.png" alt="enter image description here" /></a></p>
  <p>This can be used to design algorithms for image processing.</p>
  <p>More examples of applying Fourier transform to image processing is shown in the lectures of Guillermo Sapiro where he makes really impressive things with images of HIV: <a href="https://www.coursera.org/learn/image-processing/lecture/QAjqp/2-image-processing-and-hiv-part-i-duration-23-51-optional-breaks-at-12-37-and-18" rel="nofollow noreferrer">part 1</a>, <a href="https://www.coursera.org/learn/image-processing/lecture/7Uy43/2-image-processing-and-hiv-part-ii-duration-16-30" rel="nofollow noreferrer">part 2</a>. That Prof.'s algorithms were used in Mars rovers.</p>
  <p>A very good explanation of spectral techniques is given in the book, Wilhelm Burger, Mark J. Burge. Digital Image Processing. An Algorithmic Introduction Using Java (2016).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-25T13:07:51.867Z
Answers.LastActivityDate: 2022-01-25T13:07:51.867Z
Answers.UserDisplayName: Vladislav Gladkikh
Body: <p>While reading about various types of mathematical operations on tensors, I encountered spectral operations for the first time.</p>
  <p>The description is as follows (<a href="http://web.archive.org/web/20220113022756/https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf#page=83" rel="nofollow noreferrer">p. 53 of this book</a>)</p>
  <blockquote>
  <p><em>Spectral ops</em> - Functions for transforming in and operating in the frequency domain, like <code>stft</code> and <code>hamming_window</code></p>
  </blockquote>
  <p>While going through different codes in computer vision, I have experience of using all other mathematical operations: Pointwise ops, Reduction ops, Comparison ops, and some others. But, I personally never come across any code using spectral operations on tensors.</p>
  <p>So, I am interested in knowing whether one has an advantage in learning spectral operations in the domain of computer vision?</p>
  
Tags: <computer-vision><academia><spectral-analysis>
Title: Why should one focus on spectral operations as a computer vision researcher?
UserDisplayName: hanugm
ThreadText: # Why should one focus on spectral operations as a computer vision researcher?
  
  (Asked by: username_0 on 2022-01-20)
  
  While reading about various types of mathematical operations on tensors, I encountered spectral operations for the first time.
  
  The description is as follows ([p. 53 of this book](http://web.archive.org/web/20220113022756/https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf#page=83))
  
  > *Spectral ops* - Functions for transforming in and operating in the frequency domain, like `stft` and `hamming_window`
  
  While going through different codes in computer vision, I have experience of using all other mathematical operations: Pointwise ops, Reduction ops, Comparison ops, and some others. But, I personally never come across any code using spectral operations on tensors.
  
  So, I am interested in knowing whether one has an advantage in learning spectral operations in the domain of computer vision?
  
  **Tags:** computer-vision, academia, spectral-analysis
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-01-20. Score: 1
  
  The main advantage of using spectral operations comes from the [Covolution theorem](https://en.wikipedia.org/wiki/Convolution_theorem), which states that the [Fourier transform](https://en.wikipedia.org/wiki/Fourier_transform) of the convolution of two functions is equal to the pointwise multiplication of the individual Fourier transforms of those functions.  
  Mathematically:
  
  $\mathscr{F}|\it{f} \; \* \it{g}| = \mathscr{F}|\,\it{f}\,| \mathscr{F}|\,\it{g}\,| $
  
  Of course multiplying is much more convenient than convolving, and the Fourier transform is also a well known operation, and fast to compute, so that does not represent a downside.
  
  In practice, when it comes to computer vision, the convolution theorem become fundamental for many kinds of analytic de noising and filtering methods. Instead of applying a kernel on an image on teh spatial domain using convolution, you can simply move to the frequency domain through the Fourier transform and multiplying both, image and kernel.
  
  Moreover, when moving to the frequency domain, it become easier to access specific characteristics of images. Sharp lines and edges for example are characterized by high frequencies, so we can extract them with a simple masking operation, instead of having to came up with a tough kernel in the spatial space. This is the core idea of [low and high pass filters](https://www.javatpoint.com/dip-high-pass-vs-low-pass-filters), below here an example, where you can see that by masking the center (low frequencies) area of the spectra of two images, we can easily mix low frequency properties (e.g. shadows) and high frequency properties (e.g. edges) of two images. The same operation in the spatial domain would be tough if not impossible.
  
  [![enter image description here](https://i.sstatic.net/lIdmn.png)](https://i.sstatic.net/lIdmn.png)
  
  > **Comments:**
  > **username_2**: It may be also worth mentioning graph neural networks. Some of them exploit the convolution theorem.  
  > **username_1**: thanks for the hint, not an expert on graph networks but I'll check!  
  
  ---
  
  ### Answer by username_3 on 2022-01-25. Score: 0
  
  In the Fourier analysis, an image is represented by frequencies and phases. This allows to filter or transform certain frequencies or phases.
  
  For example, Alan Oppenheim and Jae Lim have shown that phases contain information about the object contours in the image. [Their paper](https://ieeexplore.ieee.org/document/1456290) was published in 1981, so it is poorly scanned, and the images there are not clear. Better examples are in the following books:
  
  1. Danilo P. Mandic, Vanessa Su Lee Goh. Complex Valued Nonlinear Adaptive Filters (2009)
  2. Igor Aizenberg. Complex-Valued Neural Networks with Multi-Valued Neurons (2011)
  
  Here is an example from the second book.
  
  Consider two images, "Lenna" and "Airplane",
  
  [![enter image description here](https://i.sstatic.net/P6ULV.png)](https://i.sstatic.net/P6ULV.png)
  
  Take their Fourier transform, swap magnitudes and phases of their Fourier spectra, and then perform the inverse Fourier transform. The result is as follows.
  
  Magnitudes from the "Airplane", and phases from the "Lenna":
  
  [![enter image description here](https://i.sstatic.net/hgtIP.png)](https://i.sstatic.net/hgtIP.png)
  
  Magnitudes from the "Lenna", and phases from the "Airplane":
  
  [![enter image description here](https://i.sstatic.net/KI6lB.png)](https://i.sstatic.net/KI6lB.png)
  
  So, the information about objects is contained mainly in the phase part of the spectra.
  
  This is what happens when amplitudes are replaced by the constant 1:
  
  [![enter image description here](https://i.sstatic.net/n31QE.png)](https://i.sstatic.net/n31QE.png)
  
  This can be used to design algorithms for image processing.
  
  More examples of applying Fourier transform to image processing is shown in the lectures of Guillermo Sapiro where he makes really impressive things with images of HIV: [part 1](https://www.coursera.org/learn/image-processing/lecture/QAjqp/2-image-processing-and-hiv-part-i-duration-23-51-optional-breaks-at-12-37-and-18), [part 2](https://www.coursera.org/learn/image-processing/lecture/7Uy43/2-image-processing-and-hiv-part-ii-duration-16-30). That Prof.'s algorithms were used in Mars rovers.
  
  A very good explanation of spectral techniques is given in the book, Wilhelm Burger, Mark J. Burge. Digital Image Processing. An Algorithmic Introduction Using Java (2016).
  
  
  ---
  
  