Answers.Body: <p>Based on my experience, I would say that the standard notation is just to have a regular function, and specify that it applies element wise. For example, a common notation for activation functions is <span class="math-container">$\sigma$</span>, so e.g. you could represent the activations of a regular dense layer as <span class="math-container">$\sigma(W x + b)$</span> where <span class="math-container">$x, b$</span> are vectors and <span class="math-container">$W$</span> is a matrix. I've never seen a special notation for specifying that the function <span class="math-container">$\sigma$</span> is applied element wise.</p>
  <p>As you suggest in the question, if the function to be applied element wise is linear, then you can use either hadamard product, e.g. <span class="math-container">$a \circ x$</span> or the diag function <span class="math-container">$\text{diag}(a)x$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-23T09:09:55.580Z
Answers.Text: Thank you, great answer, and this does match what I have seen in the literature: a scalar function applied to a vector is implicitly element-wise. Thinking about it, I can now see why there isn't an operation like the one I asked for. There are still some cases where it would be useful (updated question), so I'll leave the question open on the slim chance there has been a notation for it somewhere.
Answers.UserDisplayName: user7834
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-23T09:31:54.197Z
Answers.Text: Thinking more sensibly, I think the "operator" is the just words "element wise".
Answers.UserDisplayName: user7834
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-18T09:55:10.477Z
Answers.LastActivityDate: 2021-10-18T09:55:10.477Z
Answers.UserDisplayName: Taw
Answers.Body: <p>The mathematical notation for complex tensorial expressions always tries to balance complexity and precision. More precise notation - the one that explicitly spells all the indices - becomes extremely convoluted very quickly. My favorite example illustrating it is from physics -- the Standard Model Lagrangian is written shortly on T-shirts and coffee mugs as:</p>
  <p><span class="math-container">$$ \mathcal{L} = - \frac{1}{4} F_{\mu \nu} F^{\mu \nu} + (i \bar{\psi} \hat{D} \psi + \bar{\psi}_i y_{ij} \psi_j \phi + h.c.) + |D_\mu \phi|^2 - V(\phi) $$</span></p>
  <p>But if you try to expand all the indices in all the objects above - then it <a href="https://www.symmetrymagazine.org/article/the-deconstructed-standard-model-equation" rel="nofollow noreferrer">barely fits on a page</a>.</p>
  <p>On the other hand, more succinct notation always leads to ambiguities in interpretations. Your example <span class="math-container">$f(x_i)$</span> can be read as:
  <span class="math-container">$$f(x_0, x_1, \dots, x_N)\quad \text{or as}\quad f(x_0),f(x_1), \dots, f(x_N)$$</span><br />
  One way to implicitly resolve this ambiguity is to show that the index <span class="math-container">$i$</span> &quot;escapes&quot; the argument brackets:</p>
  <p><span class="math-container">$$a_i = f(x_i)\quad \text{or e.g.} \quad \sum_if(x_i)$$</span></p>
  <p>This can only be interpreted as <span class="math-container">$f(x_i)$</span> being element-wise. Also, at least in my opinion, using <span class="math-container">$x_i$</span> with index and <span class="math-container">$x$</span> without index in the same expression is extremely confusing.</p>
  <p>And, of course, the best way to resolve these ambiguities is to state them explicitly. For example, I've seen authors using square brackets <span class="math-container">$f[x_i]$</span> or capital letters <span class="math-container">$F(x_i)$</span> the vector-argument functions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-19T19:25:39.297Z
Answers.Text: Hi Kostya thank you for your answer. Sorry, I think my question may not be clear, f(x_i) is not a vector function, it is a simple scalar function. f: R->R ; a: R^n->R^n . Unless I miss something, I have to have a variable index and unindexed in the same expression, because I am relating behaviour an the variable to its components? If I just write a_i = f(x_i) it looks like x is a free variable.
Answers.UserDisplayName: user7834
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-19T19:31:07.643Z
Answers.Text: I have edited my question to follow your advice regarding capital letters.
Answers.UserDisplayName: user7834
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-19T18:23:03.227Z
Answers.LastActivityDate: 2021-05-19T18:23:03.227Z
Answers.UserDisplayName: Kostya
Body: <p>I am looking for the standard notation to define element-wise / Hadamard-style functions, if there is one.</p>
  <p>That is to say, if the operator I am looking for were represented by a hexagon ⬡, I could use it as such:</p>
  <p><span class="math-container">$$A(x) = \underset{i}{\Large{⬡} } f(x_i)$$</span></p>
  <p><span class="math-container">$$A : \mathbb{R}^n \rightarrow \mathbb{R}^n$$</span>
  <span class="math-container">$$f : \mathbb{R} \rightarrow \mathbb{R}$$</span></p>
  <p>It is very convenient to define such functions explicitly because I want to manipulate them: <span class="math-container">$B \circ A$</span> . It seems to me that the following notation is correct: <span class="math-container">$A_i(x) = f(x_i)$</span> but I worry it is nonstandard and confusing.</p>
  <p>My functions are non-linear so I cannot simply apply them directly to the array as a vector.</p>
  <p>As stated in an answer, this is unnecessary when a function is strictly scalar because it is implied to apply element-wise. There are still some situations I would hope to have it:</p>
  <p><span class="math-container">$$\underset{ij}{\Large{⬡} } e^{M_{ij}} \ne e^M$$</span></p>
  <p>The answers suggest to me the best option would be something like these:</p>
  <p><span class="math-container">$$A(M) = \text{for each } i,j: e^{M_{ij}}$$</span>
  <span class="math-container">$$A(M) = \text{element-wise}: e^{M_{ij}}$$</span></p>
  <p>The question is now closed in the negative, but I would welcome a new answer. Would be nice to find something like <span class="math-container">$\forall$</span>.</p>
  <p>Related:</p>
  <ul>
  <li><a href="https://math.stackexchange.com/questions/159005/notations-for-array-operations">https://math.stackexchange.com/questions/159005/notations-for-array-operations</a></li>
  <li><a href="https://math.stackexchange.com/questions/20412/element-wise-or-pointwise-operations-notation">https://math.stackexchange.com/questions/20412/element-wise-or-pointwise-operations-notation</a></li>
  </ul>
  
Tags: <notation>
Title: What is the correct notation for an operation that applies to each element of an array independently?
UserDisplayName: user7834
ThreadText: # What is the correct notation for an operation that applies to each element of an array independently?
  
  (Asked by: username_0 on 2021-05-19)
  
  I am looking for the standard notation to define element-wise / Hadamard-style functions, if there is one.
  
  That is to say, if the operator I am looking for were represented by a hexagon ⬡, I could use it as such:
  
  $$A(x) = \underset{i}{\Large{⬡} } f(x\_i)$$
  
  $$A : \mathbb{R}^n \rightarrow \mathbb{R}^n$$
  $$f : \mathbb{R} \rightarrow \mathbb{R}$$
  
  It is very convenient to define such functions explicitly because I want to manipulate them: $B \circ A$ . It seems to me that the following notation is correct: $A\_i(x) = f(x\_i)$ but I worry it is nonstandard and confusing.
  
  My functions are non-linear so I cannot simply apply them directly to the array as a vector.
  
  As stated in an answer, this is unnecessary when a function is strictly scalar because it is implied to apply element-wise. There are still some situations I would hope to have it:
  
  $$\underset{ij}{\Large{⬡} } e^{M\_{ij}} \ne e^M$$
  
  The answers suggest to me the best option would be something like these:
  
  $$A(M) = \text{for each } i,j: e^{M\_{ij}}$$
  $$A(M) = \text{element-wise}: e^{M\_{ij}}$$
  
  The question is now closed in the negative, but I would welcome a new answer. Would be nice to find something like $\forall$.
  
  Related:
  
  * <https://math.stackexchange.com/questions/159005/notations-for-array-operations>
  * <https://math.stackexchange.com/questions/20412/element-wise-or-pointwise-operations-notation>
  
  **Tags:** notation
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-10-18. Score: 1 | Accepted answer
  
  Based on my experience, I would say that the standard notation is just to have a regular function, and specify that it applies element wise. For example, a common notation for activation functions is $\sigma$, so e.g. you could represent the activations of a regular dense layer as $\sigma(W x + b)$ where $x, b$ are vectors and $W$ is a matrix. I've never seen a special notation for specifying that the function $\sigma$ is applied element wise.
  
  As you suggest in the question, if the function to be applied element wise is linear, then you can use either hadamard product, e.g. $a \circ x$ or the diag function $\text{diag}(a)x$.
  
  > **Comments:**
  > **username_0**: Thank you, great answer, and this does match what I have seen in the literature: a scalar function applied to a vector is implicitly element-wise. Thinking about it, I can now see why there isn't an operation like the one I asked for. There are still some cases where it would be useful (updated question), so I'll leave the question open on the slim chance there has been a notation for it somewhere.  
  > **username_0**: Thinking more sensibly, I think the "operator" is the just words "element wise".  
  
  ---
  
  ### Answer by username_2 on 2021-05-19. Score: 1
  
  The mathematical notation for complex tensorial expressions always tries to balance complexity and precision. More precise notation - the one that explicitly spells all the indices - becomes extremely convoluted very quickly. My favorite example illustrating it is from physics -- the Standard Model Lagrangian is written shortly on T-shirts and coffee mugs as:
  
  $$ \mathcal{L} = - \frac{1}{4} F\_{\mu \nu} F^{\mu \nu} + (i \bar{\psi} \hat{D} \psi + \bar{\psi}\_i y\_{ij} \psi\_j \phi + h.c.) + |D\_\mu \phi|^2 - V(\phi) $$
  
  But if you try to expand all the indices in all the objects above - then it [barely fits on a page](https://www.symmetrymagazine.org/article/the-deconstructed-standard-model-equation).
  
  On the other hand, more succinct notation always leads to ambiguities in interpretations. Your example $f(x\_i)$ can be read as:
  $$f(x\_0, x\_1, \dots, x\_N)\quad \text{or as}\quad f(x\_0),f(x\_1), \dots, f(x\_N)$$  
  One way to implicitly resolve this ambiguity is to show that the index $i$ "escapes" the argument brackets:
  
  $$a\_i = f(x\_i)\quad \text{or e.g.} \quad \sum\_if(x\_i)$$
  
  This can only be interpreted as $f(x\_i)$ being element-wise. Also, at least in my opinion, using $x\_i$ with index and $x$ without index in the same expression is extremely confusing.
  
  And, of course, the best way to resolve these ambiguities is to state them explicitly. For example, I've seen authors using square brackets $f[x\_i]$ or capital letters $F(x\_i)$ the vector-argument functions.
  
  > **Comments:**
  > **username_0**: Hi username_2 thank you for your answer. Sorry, I think my question may not be clear, f(x_i) is not a vector function, it is a simple scalar function. f: R->R ; a: R^n->R^n . Unless I miss something, I have to have a variable index and unindexed in the same expression, because I am relating behaviour an the variable to its components? If I just write a_i = f(x_i) it looks like x is a free variable.  
  > **username_0**: I have edited my question to follow your advice regarding capital letters.  
  
  ---
  
  