text:
\section{Introduction}

A reinforcement learning (RL) agent learns how to map states to actions in order to maximize a long-term cumulative award signal in a given environment. Figure \ref{fig:RL_model} shows the various artifacts of an RL algorithm. An RL problem is defined by a quartet of $(S, A, P_a, R_a)$, where $S$ is a set of states or the state space; $A$ is a set of actions or the action space available to influence $S$; $P_a(s, s') = Pr(s_{t+1}=s'|s_t=s, a_t=a)$ is the transition probability which is the probability that action ${a}$ in state ${s}$ at time ${t}$ will lead to state ${s'}$ at time ${t+1}$; and finally, ${R_{a}(s,s')}$ is the immediate reward signal received after transitioning from state ${s}$ to state ${s'}$, due to action ${a}$. An RL agent training involves recognizing an optimal policy function, $\pi^{*}: a \gets s$,  from a corpus of $\{(s_i, a_i)\}$ to maximize the long-term cumulative reward, $\sum R_{a}$. The reward function, $R_a$, is defined ab initio  for an efficient goal accomplishment. The transition probability or state-action mapping is defined by the environment dynamics. In many real-life use cases, the agent cannot directly sense the effect of its actions on the environment. This is particularly true when the state-action relationship cannot be modeled by either closed-form analytical expressions \cite{kohl2004policy, liang2018gpu, zhu2019dexterous} or explicit rules as in the popular games such as Chess \cite{lai2015giraffe}, Go \cite{singh2017learning, silver2016mastering}, and Atari \cite{mnih2013playing}. This challenge is well documented in the literature \cite{thrun1993issues, zhu2018deep}. To address this challenge for RL model training \cite{van2012reinforcement, mnih2015human}, simulation-based action models \cite{sutton1991dyna} play a pivotal role. The optimal choice of simulation parameters such as the training action space is a non-trivial challenge because of the curse of dimensionality \cite{rl_dimensionality}, non-linearity \cite{rl_nonlinearity}, and non-uniform action set \cite{jin2021towards}.

Training data valuation \cite{jia2019towards, fryer2021shapley} and associated artisanal software engineering efforts constitute a large part of the machine learning (ML) life cycle (or MLOps). Yet, most research and development efforts \cite{molnar2020interpretable} focus on algorithms and infrastructure. Production-grade MLOps needs to handle data lifecycle management (DLM) \cite{polyzotis2018data} challenges including: fairness and bias in labeled datasets \cite{goel2019deep}, data quality \cite{crawford2019excavating}, limitations of benchmarks \cite{rocca2020putting}, and reproducibility concerns \cite{pineau2021improving}. For RL, the DLM challenges are further compounded due to complexities arising from non-linear state-action interactions, partially-observable processes, non-isometric action spaces \cite{non_isometric}, and strong domain specificity \cite{domain_specific_rl} of action models. With the recent emphasis on ML explainability, traditional supervised learning and deep learning research communities are actively working on systematic data-driven frameworks \cite{ghorbani2019data, weerts2019human} for training data valuation. We need equivalent frameworks for RL to streamline conflict-prone training action selection \cite{cavanagh2014conflict, osinski2020simulation, hashmy2020wide}. Depending on the agent-environment interactions, different training actions have different relative contributions to the RL agent performance. Some training actions are indispensable because of their unique positions in the parametric space. Other dispensable actions have different relative contributions to the reward function. \emph{Remarkably, a high-cardinality training action space in many cases leads to lower cumulative reward than a well-designed training action space.} To the best of authors' knowledge, there is hardly any data-driven tool for the evaluation of training action space for RL. Such a tool leads to superior RL agent performance, lower model training and maintenance cost, and strong multi-disciplinary collaboration \cite{agnostic}.

This paper proposes a shapley-inspired \cite{shapley_value} algorithm to categorize and rank training action sets. It also assists in recognizing cut-off cardinality for the training action space to reduce unnecessary exploration and ensure polynomial time complexity. While Section \ref{sec:algorithm} describes the algorithms for RL training action space evaluation, Section \ref{case_study} illustrates the effectiveness of the algorithms in a specific case study. Section \ref{related_work} discusses the relevant work. Finally, Section \ref{conclusion} presents a summary with possible future directions.

\begin{figure}
  \centering
  \fbox{\includegraphics[width=0.85\linewidth]{figures/RL_model.JPG}}
  \caption{The schematic diagram for a typical reinforcement learning algorithm}
  \label{fig:RL_model}
\end{figure}

\section{Algorithm}
\label{sec:algorithm}

The proposed framework provides a shapley-inspired methodology \cite{sundararajan2020many} to efficiently filter out unnecessary training actions and discover a near-optimal action set. Figure \ref{fig:framework} shows the framework consists of two algorithms and how these two algorithms interact with each other to design a near-optimal RL model by training action space filtering and evaluation. The first algorithm computes cut-off cardinality for the training action space. The cut-off cardinality is defined as the minimium number of action points needed in a training action set to enable high-fidelity predictions within an acceptable computational complexity and an error bound. The second algorithm takes action points only above the cut-off cardinality and categorize action points into two classes: $\{dispensable, indispensable\}$ and ranking dispensable training action points based on the corresponding cumulative rewards. Indispensable action points are absolute essential action points for the RL training to achieve the desired goal. On the other hand, dispensable action points are non-essential action points that can be discarded. However, the RL agent might perform sub-optimally without a disposable action point. By ranking disposable training action points based on the corresponding cumulative rewards, the relative contributions of different training action points are estimated.

 \begin{figure}
  \centering
  \fbox{\includegraphics[width=0.95\linewidth]{figures/framework.png}}
  \caption{The proposed training action selection framework for reinforcement learning }
  \label{fig:framework}
\end{figure}

\subsection{Cut-off Cardinality Computation for Training Action Points }

RL deals with compilation of a quartet $(S, A, P_a, R_a)$ and computation of an optimal policy, $\pi^{*}$ which maximizes the long-term cumulative reward. The search space for a shapley-inspired technique needs to span across the power set for the training action space which has exponential complexity with training action points. To circumvent this problem, the proposed algorithm, as discussed in Algorithm \ref{alg:cardinality}, uses Monte Carlo method to compute the cut-off cardinality. The inputs to this algorithm include an RL algorithm, the power set for the training action space, the corresponding state space, the transition probability, an acceptable performance margin, and an acceptable number of iterations. The output is the cut-off cardinality. The algorithm uses an iterative procedure to train the RL agent with different numbers of training actions, starting from all available training actions to only one training action. If for a certain cardinality, all the cumulative rewards do not improve by $\epsilon$ within $n$ iterations, then we set the status to failure, and assign the cut-off cardinality to that cardinality plus one. The cut-off cardinality signifies computational efficiency --- a larger cut-off cadinality means lower computational overhead and vice versa.

\begin{algorithm}
\caption{Monte Carlo method for cut-off cardinality computation for the training action space }\label{alg:cardinality}
\begin{algorithmic}
\Require The RL algorithm, $\mathcal{R}$,  the power set for the training action space, $\mathcal{P}(A)$, the corresponding state space, $\mathcal{P}(S|A)$,  transition probability ${P}$, an acceptable performance margin, $\epsilon$, and an acceptable number of iterations, $n$.
\Ensure Cut-off cardinality, $\vert{A}\vert_{cutoff}$

\For{$k \gets \vert{A}\vert$ to 1}
    \Repeat
       \State $\pi^{*} \gets \mathcal{R}(S, A, P_a, R_a)$
       \If{ the cumulative reward improves by $\epsilon$ with $n$ iterations }
            \State $status \gets success$
        \Else
            \State $status \gets failure$
        \EndIf
    \Until{all necessary training action sets for a given cardinality is exhausted.}
    \If{all statuses for a given cardinality, $k$, are $failures$}
        \State $\vert{A}\vert_{cutoff} \gets k+1$
    \EndIf
\EndFor
\end{algorithmic}
\end{algorithm}

\subsection{Categorization and Ranking of Training Actions}

Algorithm \ref{alg:gen} offers a principled procedure for training action selection. As for inputs, it takes the combinations of training state-action pairs above the cut-off cardinality, determined by Algorithm \ref{alg:cardinality}, the RL algorithm, the reward function, the threshold condition, and the acceptable number of iterations. As for outputs, it predicts a categorization vector which classifies training action points into two classes: \{\emph{dispensable}, \emph{indispensable}\} and ranks dispensable training actions based on the corresponding cumulative reward. At its core, Algorithm \ref{alg:gen} determines whether the policy determined from the given state-action pairs achieves the threshold condition within the acceptable number of iterations. The algorithm uncovers the indispensability of some training action sets for a given RL task. A training action set is deemed to be indispensable: \emph{iff} upon removal of that training action point, the RL agent fails to accomplish the goal within a finite number of iterations \cite{jiang2017contextual, zhou2021provably, jiang2018open}. All other action points are dispensable. The dispensable action points are rank listed in $\mathcal{L}$ based on the corresponding cumulative reward. Essentially, $\mathcal{L}$ informs the performance impact of different disposable action points on the RL agent.

\begin{algorithm}
\caption{Data-driven categorization and ranking for training action space evaluation }\label{alg:gen}
\begin{algorithmic}
\Require All combinations of training state-action pairs upto cut-off cardinality: ${\{(s_i,a_i)\}}$, the RL algorithm, $\mathcal{R}$, the reward function, the threshold condition, and the acceptable number of iterations.
\Ensure Categorization vector, $\mathcal{V}$, which classifies training action points into two classes: \{\emph{dispensable}, \emph{indispensable}\}. Rank order list of dispensable training actions based on the corresponding cumulative reward.
\Repeat
\For {$<s_i,a_i> $}
  \State $\pi^{*} \gets  \mathcal{R}(S, A, P_a, R_a) $
    \If{Threshold condition is $never$ satisfied $within$ the acceptable number of iterations}
        \State $\mathcal{V}(A - a_i)  \gets indispensable$
    \Else
        \State $\mathcal{V}(A - a_i)  \gets dispensable$
    \EndIf
\EndFor
\Until{all necessary training action sets are evaluated.}
\State Rank List, $\mathcal{L}  \gets$ dispensable actions sorted in order of the corresponding cumulative rewards
\end{algorithmic}
\end{algorithm}

\subsection{Action Shapley}

The proposed framework adopts Action Shapley which introduces a formulation for the problem of equitable training action valuation in reinforcement learning. Reinforcement learning has three key building blocks. The first building block is the training action space, $A$, the corresponding state space $S$, and a pre-assigned reward function, $R_{a}$.  The second building block is a reinforcement learning algorithm, $\mathcal{R}$, which is treated as a black box. It takes $(S, A, P_a, R_a)$ and computes an optimal policy function, $\pi^{*}$. The third building block is the performance measure or the cumulative sum of reward, $\sum R_{a}$. The goal of the framework is to compute the valuation for each training action point, $\phi_{i}(A, \mathcal{R}, \sum R_{a})$ as a function of these three building blocks. A Shapley technique is useful for action valuation because it satisfies nullity, symmetry, and linearity \cite{shorrocks1999decomposition}. From game theory, $\phi$ can be expressed as shown in Equation \ref{shapley}:

\begin{equation}
\label{shapley}
\phi_{i}=C\sum_{S \subseteq A - \{i\} } \frac{\sum R_{a}(S \cup \{i\}) - \sum R_{a}(S)}{\binom{n-1}{\vert{S}\vert}}
\end{equation}

where,  the sum is over all subsets of $A$ not containing $i$ and $C$ is an arbitrary constant. We call $\phi_{i}$ the Action Shapley value of action $a_{i}$. Equation \ref{shapley} suggests Action Shapley computation is with exponential time complexity w.r.t. the training action points. This exponential computational complexity warrants the need for the cut-off cardinality discussed in Algorithm \ref{alg:cardinality}.

\section{Case Study}
\label{case_study}

In this section, we design a resource tuning example (in the cloud) to illustrate the effectiveness of the proposed algorithm for the training action space evaluation. We evaluate the performance of an MDP \cite{howell2000line}-based RL agent, as shown in Figure \ref{fig:RL_model}. The state-action mapping and transition probability are modeled using time-series auto-regression \cite{ostrom1990time} after a PCA-based dimensionality reduction \cite{raunak2019effective} with time complexity $\mathcal{O}(n\log{}n)$, where $k$ is the number of principal components. The choice of the algorithms is purely driven by the nature of the training dataset. A more complex non-linear dataset warrants more complex sequential models such as long short-term memory (LSTM) \cite{lstm_tsf} or attention based models \cite{attention_tsf}. For action updates, WLOG, an RL-based PID controller \cite{pid_controller, gheisarnejad2020intelligent} is used. Similar to time-series modeling, action update can be conducted by other policy learning algorithms such as SARSA \cite{sarsa_sutton}.

As shown in Table \ref{table:particular}, the objective of the RL agent for this case study is to quickly reduce high CPU utilization below a pre-assigned threshold for a given workload. In most infrastructure/cloud resource tuning technologies \cite{zhang2017intelligent}, CPU utilization represents a key metric. Therefore, the state space for this RL case study consists of virtual machine CPU utilization (\%) metrics and the action space is defined by the VM resource set points, i.e., (\# of vCPUs, Memory Size (GB)). The RL agent uses AWS boto3 SDK \cite{aws_ec2} to manipulate actions and AWS CloudWatch \cite{aws_cloudwatch} for state space monitoring. The reward is defined as the number of time steps required by the agent to accomplish the objective multiplied by $-1$. The negative reward per time step was meant to push the agent to accomplish the task as fast as possible.

\begin{table}[t]
\caption{Relevant case study for cloud resource tuning}
\label{table:particular}
\begin{center}
\begin{tabular}{ll}
\toprule
\multicolumn{1}{c}{\bf RL Artifact}  &\multicolumn{1}{c}{\bf Description}
\\ \hline \\
  \emph{State}   & CPU utilization metrics     \\
  \emph{State Statistics}   & Median value of CPU utilization   \\
  \emph{Threshold}  & 90\% of CPU utilization  \\
  \emph{Action}   & Resource configuration set points: (\# of vCPUs, Memory Size (GB))  \\
  \emph{Reward} & Negative of total time steps required to satisfy the threshold condition \\
  \emph{Parametric Boundary} & Polygon defined by the parametric endpoints\\
  \emph{Initial Action} & $(6, 14)$: Arbitrarily assigned WLOG\\
  \emph{Error Margin} &  $5\%$\\
  \emph{Acceptable Steps} &  $400$\\
  \emph{Computational Complexity} &  Polynomial time\\
\bottomrule
\end{tabular}
\end{center}
\end{table}

The training data for this case study was generated internally \cite{aws_data} with an open source library, \emph{stress} (https://linux.die.net/man/1/stress). It uses a rectangular workload. The peak of the workload uses the stress command: \emph{sudo stress --io 4 --vm 2 --vm-bytes 1024M --timeout 500s}. Essentially, the peak is running 4 \emph{I/O} stressors and $2$ VM workers spinning on malloc with $1024$ MB per worker for $500$ s. The simulated rectangular workload has a time period of $600$ s: a high-stress phase of $500$ s is followed by an inactive phase of $100$ s. The RL training action space is spanned by the power set drawn from the five pairs of EC2 configuration set points as shown in Table \ref{table:action_space}. Using Algorithm \ref{alg:cardinality}, we notice that the power set below the cut-off cardinality of $4$ produces trivial and unstable results. Therefore, WLOG, the analysis in this paper has been focused on six training action sets as shown in Table \ref{table:differential_reward}. That amounts to 81.25\% reduction in the search space. For each EC2 instance in the training action space, the corresponding state metrics, i.e., CPU utilization (\%) are shown in Figure \ref{fig:time_series}. The training data was collected for a $24$ hour period with $1$ minute sampling interval. Using Monte Carlo method, the optimal step-size for action update is identified to be $0.1$ in both \# of vCPUs and Memory Size (GB) dimensions.

\begin{table}[t]
\caption{Five different AWS EC2 resource pairs used in training action space}
\label{table:action_space}
\begin{center}
\begin{tabular}{lll}
\toprule
\multicolumn{1}{c}{\bf EC2 Type}  &\multicolumn{1}{c}{\bf no of vCPUs}    &\multicolumn{1}{c}{\bf Memory Size (GB)}
    \\ \hline \\
     \emph{small t3a}    &2 &2 \\
     \emph{medium t3a}   &2 &4 \\
     \emph{large t3a}    &2 &8 \\
     \emph{xlarge t3a}   &4 &16 \\
     \emph{2xlarge t3a}  &8 &32 \\
\bottomrule
\end{tabular}
\end{center}
\end{table}

\begin{table}[t]
\caption{Training action categorization based on the valuation vector}
\label{table:action_categorization}
\begin{center}
\centering
\begin{tabular}{ll}
\toprule
\multicolumn{1}{c}{\bf Training Action }  &\multicolumn{1}{c}{\bf Category}
\\ \hline \\
    \emph{small t3a}         &dispensable \\
    \emph{medium t3a}        &dispensable \\
    \emph{large t3a}         &indispensable \\
    \emph{xlarge t3a}        &indispensable \\
    \emph{2xlarge t3a}       &indispensable\\
\bottomrule
\end{tabular}
\end{center}
\end{table}

With this set up, different RL models are developed with different training action sets and the corresponding rewards and ranks are noted in  Table \ref{table:differential_reward}. Remarkably, a high cardinality training action set does not guarantee the best reward: the agent with all training actions,\emph{<small t3a, medium t3a, large t3a, xlarge t3a, 2xlarge t3a>}, does not yield the highest reward. In fact, the action set with \emph{<medium t3a, large t3a, xlarge t3a, 2xlarge t3a>} yields the highest reward. This pattern could be attributed to the state-action interaction in this particular case study. First, the parametric distances between different EC2 instance pairs are not uniform. While the Euclidean distance between \emph{small t3a} and \emph{medium t3a} is equal to $2$, that between \emph{xlarge t3a} and \emph{2xlarge t3a} is $16.5$. The non-uniform spacing for training action space is a considerable deterrent \cite{manjanna2018reinforcement} for RL adoption. Second, in this case study, the transient CPU utilization (\%) patterns have undergone a material change from \emph{xlarge t3a} (max ~100\%) to \emph{2xlarge t3a} (max ~73\%). This indicates the strong influence of \emph{2xlarge t3a} for the given RL task. Indeed, we noticed \emph{2xlarge t3a} to be an indispensable training action. As shown in Table \ref{table:action_categorization}, a categorization of training actions can be inferred based on the valuation vector, $\mathcal{V}$, as described in Algorithm \ref{alg:gen}: two \emph{dispensable} training actions are uncovered to be \emph{small t3a, medium t3a} and three \emph{indispensable} training actions to be \emph{large t3a, xlarge t3a, 2xlarge t3a}.

\begin{table}[t]
\caption{Rewards and ranks for different training actions. Some training action sets fail to satisfy the objective. Therefore, the rewards and ranks for them are noted as \emph{none}}.
\label{table:differential_reward}
\begin{center}
\begin{tabular}{lll}
\toprule
\multicolumn{1}{c}{\bf Training Action Set}  &\multicolumn{1}{c}{\bf Reward}  &\multicolumn{1}{c}{\bf Rank}
\\ \hline \\
 \emph{<small t3a, medium t3a, large t3a, xlarge t3a, 2xlarge t3a>}   &  $-21$  & $3$    \\
   \emph{<medium t3a, large t3a, xlarge t3a, 2xlarge t3a>}  & $-13$   & $1$   \\
  \emph{<small t3a, large t3a, xlarge t3a, 2xlarge t3a>}    &  $-16$ & $2$  \\
\emph{<small t3a, medium t3a, xlarge t3a, 2xlarge t3a>}  & \emph{none} & \emph{none} \\
\emph{<small t3a, medium t3a, large t3a, 2xlarge t3a>}  & \emph{none}  & \emph{none}\\
 \emph{<small t3a, medium t3a, large t3a, xlarge t3a>}   &  \emph{none} & \emph{none} \\
\bottomrule
\end{tabular}
\end{center}
\end{table}

\begin{figure}
  \fbox{\includegraphics[width=0.9\linewidth]{figures/time_series.JPG}}
  \centering
  \caption{CPU utilization (\%) responses on five different EC2 instances (Table \ref{table:action_space}) from 2PM-UTC 9/1/2021-2PM-UTC 9/2/2021 at 1 minute granularity. This training data was generated internally \cite{aws_data}. The median CPU utilization values (\%) are noted to be equal to \{95\%, 95.5\%, 99.5\%, 100\%, 72.5\%\}}.
  \label{fig:time_series}
\end{figure}

\begin{figure}
     \centering
     \begin{subfigure}[b]{0.8\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/RL_all.JPG}

     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.8\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/RL_all_but_1.JPG}

     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.8\textwidth}
         \centering
         \includegraphics[width=\textwidth]{figures/RL_all_but_2.JPG}

     \end{subfigure}
        \caption{Examples of dispensable actions. (a) RL loop with all training actions, \emph{<small t3a, medium t3a, large t3a, xlarge t3a, 2xlarge t3a>}. The reward is noted to be $-21$. (b) RL loop with a training action set of \emph{<medium t3a, large t3a, xlarge t3a, 2xlarge t3a>}. The reward is noted to be $-13$. (c) RL loop with a training action set of \emph{<small t3a, large t3a, xlarge t3a, 2xlarge t3a>}. The reward is noted to be $-16$. Both \emph{small t3a} and \emph{medium t3a} are noted to be dispensable actions. }
    \label{fig:dispensable}
\end{figure}

\begin{figure}
  \fbox{\includegraphics[width=\linewidth]{figures/RL_all_but_3.JPG}}
  \caption{RL loop with \emph{<small t3a, medium t3a, xlarge t3a, 2xlarge t3a>} action space. The agent could never accomplish the goal, therefore, \emph{large t3a} is an indispensable element in the training action space. Similarly, \emph{xlarge t3a} and \emph{2xlarge t3a} are two indispensable elements in the training action space.}
  \label{fig:indispensable}
\end{figure}

Figure \ref{fig:dispensable} shows the RL loop action with three training actions: \emph{<small t3a, medium t3a, large t3a, xlarge t3a, 2xlarge t3a>}, \emph{<medium t3a, large t3a, xlarge t3a, 2xlarge t3a>}, and \emph{<small t3a, large t3a, xlarge t3a, 2xlarge t3a>}.

\begin{itemize}
    \item For the first training action set of \emph{<small t3a, medium t3a, large t3a, xlarge t3a, 2xlarge t3a>}, the reward is $-21$. The left subplot in Figure \ref{fig:dispensable}(a) shows how the recommended action is evolving with time from an \emph{arbitrary} initial point of (6, 14). The recommended points are superimposed on the training action parameter space to illustrate their relative position with respect to the parametric boundary which is defined by the trapezium with vertices: \emph{\{(2,2), (2,8), (8,32), (8,2)\}} in the (\# of vCPUs, Memory Size (GB)) space. The right subplot in Figure \ref{fig:dispensable}(a) shows how the median CPU utilization (\%) comes below the \emph{90\%} threshold in $21$ steps leading to $-21$ in reward. The blue dots represent the median CPU utilizations for different training set points.
    \item For the second training action set of \emph{<medium t3a, large t3a, xlarge t3a, 2xlarge t3a>}, the reward is $-13$ as shown in Figure \ref{fig:dispensable}(b).
    \item For the third training action set of \emph{<small t3a, large t3a, xlarge t3a, 2xlarge t3a>}, the reward is $-16$  as shown in Figure \ref{fig:dispensable}(c).
\end{itemize}

As shown in Table \ref{table:differential_reward}, the reward scores can indeed be used for ranking different training action sets leading to a data-driven approach for training action selection. As shown in Figure \ref{fig:indispensable}, \emph{large t3a} is an indispensable element in the training action space for the given RL agent. Without this training action, the RL agent fails to accomplish the goal of bringing the CPU utilization below the critical threshold. Similar observations can be made about \emph{xlarge t3a} and \emph{2xlarge t3a}.

\section{Related Work}
\label{related_work}

Shapley value was postulated in a classic paper in game theory \cite{gillies1953some} and influences the field of economics significantly \cite{roth1988shapley}. Various shapley-inspired techniques have been applied to model diverse problems including voting \cite{gelman2004standard}, resource allocation \cite{song2018comprehensive} and bargaining \cite{kravitz1984number}. Recent years saw increased application of shapley-inspired methodologies in machine learning feature importance evaluation and training data valuation \cite{ghorbani2019data, fryer2021shapley}. To the best of our knowledge, Shapley value has not been used to quantify training action valuation in a reinforcement learning context. The application of shapley-inspired technique for reinforcement learning is a challenging problem. First, the relationship between state-action pairs and transition probabilities are expensive to model. Also, often the RL training action points are not uniformly positioned. Finally, we have purposefully focused on a small sample cloud infrastructure use case instead of cliched RL use cases \cite{gym}. That is because the problem of optimal action selection has relatively more expensive in the cloud infrastructure space where the state-action relationship is only partially observable.

\section{Conclusion}
\label{conclusion}
This paper proposes a data-driven methodology for training action space evaluation for RL. The methodology offers a principled framework for training action space categorization and ranking within a finite computational time. It unleashes a strategy for superior model performance and lower modeling cost. Additionally, the proposed methodology is completely agnostic of use cases and machine learning algorithms. Therefore, it is a general-purpose methodology extensible to  different domains including distributed computing, network traffic control, healthcare, automatic locomotion, building management system, and industrial controls, and different machine learning algorithms such as PCA, Autoencoder, ARIMA, LSTM, Transformer, PID, SARSA, DQN, and many others. For the next phase of the development for this data-agnostic methodology, we are planning to contribute a general RL design library to relevant open source projects \cite{ray, gym, katib}.

\bibliographystyle{unsrt}


text:
\section{Introduction}

Maxwell has unified the laws of electricity and magnetism in a consistent way
into a set of four equations, and he calculated the speed of electromagnetic
waves and found that it was $300,000km/sec$, which was the same of light. This
forced Maxwell to ponder about the nature of light: he concluded that light is
a form of electromagnetic wave. For the explanation the spectra of black-body
radiation, Planck firstly proposed that emission of black-body was energy
quantization with value of $\hbar \omega$, and he introduced the Planck
constant $\hbar$ \cite{s1}. In photoelectric effect experiment, Einstein only
considered the energy conservation and made use of Planck's $\hbar \omega$ to
propose a quanta concept of light \cite{s2}, and successfully explaining
photoelectric effect phenomena, It was the beginning of Quantum Physics.

Despite claims that massless particles cannot be localized in space-time
\cite{s3,s4}, physicists have pondered over the problem of photon localization
and the related problem of the photon wave function for almost $80$ years now,
beginning with the work of Landau and Peierls \cite{s5}. An extensive review
of the photon localization problem was presented by Keller \cite{s6}. The
problem of photon localization is closely related to the widely studied
problem of the photon position operator. Quantum optics has long struggled to
define a quantum wavefunction for photons that would be in agreement with
innumerable experiments revealing the possibility of photon localization. The
photon wave function appears in both of these descriptions and that it
provides a very convenient concept to unify the two points of view. A very
useful mathematical tool in this analysis is the Riemann-Silberstein vector
\cite{s7,s8,s9,s10,s11}. Applications of the $RS$ vector to many physical
problems were reviewed in a very thorough paper by Keller \cite{s6}.

After Dirac discovered the relativistic equation for a particle with spin 1/2
\cite{s12}, much work was done to study spinor and vectors within the Lorentz
group theory. Any quantity which transforms linearly under Lorentz
transformations is a spinor. For that reason spinor quantities are considered
as fundamental in quantum field theory and basic equations for such quantities
should be written in a spinor form. A spinor formulation of Maxwell equations
was studied by Laporte and Uhlenbeck \cite{s13}, also see Rumer \cite{s14}. In
1931, Oppenheimer \cite{s15} proposed to consider the Maxwell theory of
electromagnetism as the wave mechanics of the photon. They introduced a
complex 3-vector wave function satisfying the massless Dirac-like equations.
In this paper, we shall give the relativistic spinor equation for photon with
spin $1$ and mass $0$.

\section{Relativistic spinor equation of free photon}

Dirac derived the Dirac equation by factorizing Einstein's dispersion relation
such that the field equation becomes the first order in time derivative
\cite{s16}. Namely, he factorized the relativistic dispersion relation
employing four by four matrice

\begin{equation}
E^{2}-c^{2}\vec{p}\hspace{0.05in}^{2}-m_{0}^{2}c^{4}=(E-c\vec{p}\cdot
\vec{\alpha}-m_{0}c^{2}\beta)(E+c\vec{p}\cdot \vec{\alpha}+m_{0}c^{2
\beta),\label{1
\end{equation}
where $\vec{\alpha}$ and $\beta$ are Dirac matrices. For a photon, mass
$m_{0}=0$, equation (\ref{1}) becomes
\begin{equation}
E^{2}-c^{2}\vec{p}\hspace{0.05in}^{2}=(E-c\vec{\alpha}\cdot \vec{p
)(E+c\vec{\alpha}\cdot \vec{p}),\label{2
\end{equation}
i.e.,
\begin{equation}
E-c\vec{\alpha}\cdot \vec{p}=0,\label{3
\end{equation}
canonical quantization equation (\ref{3}), we have
\begin{equation}
i\hbar \frac{\partial}{\partial t}\psi=-ic\hbar \vec{\alpha}\cdot \vec
{\bigtriangledown}\psi=H\psi,\label{4
\end{equation}
where the photon's Hamiltonian operator $H=-ic\hbar \vec{\alpha}\cdot
\vec{\bigtriangledown}$, and $\psi$ is spiron wavefunction of photon. For the
proper Lorentz group $L_{p}$, the irreducibility representations of spin $1$
photon is $D^{10}$, $D^{01}$ and $D^{\frac{1}{2}\frac{1}{2}}$, respectively,
and the dimension of irreducibility representations are corresponding to
three, three and four. We choose photon's spiron wave function as the basis
vector of three dimension irreducibility representations, it is
\begin{equation}
\psi=\left(
\begin{array}
[c]{c
\psi_{1}\\
\psi_{2}\\
\psi_{3
\end{array}
\right)  ,\label{5
\end{equation}
and $\vec{\alpha}$ matrices are taken as
\begin{equation}
\alpha_{x}=\left(
\begin{array}
[c]{ccc
0 & 0 & 0\\
0 & 0 & -i\hbar \\
0 & i\hbar & 0
\end{array}
\right)  ,\alpha_{y}=\left(
\begin{array}
[c]{ccc
0 & 0 & i\hbar \\
0 & 0 & 0\\
-i\hbar & 0 & 0
\end{array}
\right)  ,\alpha_{z}=\left(
\begin{array}
[c]{ccc
0 & -i\hbar & 0\\
i\hbar & 0 & 0\\
0 & 0 & 0
\end{array}
\right)  ,\label{6
\end{equation}
they are Hermitian matrices
\begin{equation}
\vec{\alpha}^{\dag}=\vec{\alpha},\label{7
\end{equation}
and photon's Hamiltonian operator is Hermitian
\begin{equation}
{H}^{\dag}={H}.\label{8
\end{equation}

\section{The spin operators of photon}

In the following, we shall prove the selection of $\vec{\alpha}$ matrices are
reasonable, they confirm that equation (4) is photon's spiron equation of spin
$1$. equation (\ref{4}) can be written as
\begin{equation}
i\hbar \frac{\partial}{\partial t}\psi=c(\vec{p}\cdot \vec{\alpha})={H\psi
},\label{9
\end{equation}
where
\begin{equation}
{H}=c\vec{p}\cdot \vec{\alpha},\label{10
\end{equation}
The orbital angular momentum of photon satisfy
\begin{align}
\frac{d}{dt}{L_{x}} &  =\frac{1}{i\hbar}[{L_{x}},{H}]\nonumber \\
&  =c(\alpha_{y}{p_{z}}-\alpha_{z}{p_{y}})\nonumber \\
&  =c(\vec{\alpha}\times \vec{p})_{x},\label{11
\end{align}
i.e.,
\begin{equation}
\lbrack \vec{L},{H}]=i\hbar c(\vec{\alpha}\times \vec{p}),\label{12
\end{equation}
the equation (\ref{12}) is shown that the orbital angular momentum of photon
isn't conservation, but the total angular momentum of free photon should be
conservative. So, photon should have an intrinsic angular momentum, i.e., spin
angular momentum $\vec{s}$, the total angular momentum of photon $\vec{J}$ is
\begin{equation}
{\vec{J}}={\vec{L}}+{\vec{s}},\label{13
\end{equation}
and $\vec{J}$ is conservative
\begin{equation}
\lbrack{\vec{J}},{H}]=0,\label{14
\end{equation}
with equations (\ref{12}) and (\ref{14}), we have
\begin{equation}
\lbrack \vec{s},{H}]=-[{\vec{L}},{H}]=-i\hbar c(\vec{\alpha}\times \vec
{p}),\label{15
\end{equation}
i.e.,
\begin{align}
\lbrack{s}_{x},{H}] &  =[{s}_{x},c\vec{\alpha}\cdot \vec{p}]=-i\hbar
c(\vec{\alpha}\times \vec{p})_{x}\nonumber \\
&  =i\hbar c(\alpha_{z}{p}_{y}-\alpha_{y}{p}_{z}),\label{16
\end{align}
or
\begin{align}
\lbrack{s}_{x},c\alpha_{x}p_{x}+c\alpha_{y}p_{y}+c\alpha_{z}p_{z}] &
=c[s_{x},\alpha_{x}]p_{x}+c[s_{x},\alpha_{y}]p_{y}+c[s_{x},\alpha_{z
]p_{z}\nonumber \\
&  =i\hbar c(\alpha_{z}{p}_{y}-\alpha_{y}{p}_{z}),\label{17
\end{align}
there are commutation relations
\begin{equation}
\lbrack s_{x},\alpha_{x}]=0,\hspace{0.2in}[s_{x},\alpha_{y}]=i\hbar \alpha
_{z},\hspace{0.2in}[s_{x},\alpha_{z}]=-i\hbar \alpha_{y},\label{18
\end{equation}
it is easy to summarize that
\begin{equation}
\lbrack s_{i},\alpha_{j}]=i\hbar \epsilon_{ijk}\alpha_{k},\label{19
\end{equation}
with equation (\ref{18}), we can calculate the $s$ matrices, let the $s_{x}$
matrix is
\begin{equation}
s_{x}=\left(
\begin{array}
[c]{ccc
a_{11} & a_{12} & a_{13}\\
a_{21} & a_{22} & a_{23}\\
a_{31} & a_{32} & a_{33
\end{array}
\right)  ,\label{20
\end{equation}
with the commutation relation
\begin{equation}
\lbrack s_{x},\alpha_{y}]=i\hbar \alpha_{z},\hspace{0.2in}[s_{x},\alpha
_{z}]=-i\hbar \alpha_{y},\label{21
\end{equation}
we get
\begin{equation}
s_{x}=\left(
\begin{array}
[c]{ccc
a & 0 & 0\\
0 & a & -i\hbar \\
0 & i\hbar & a
\end{array}
\right)  ,\label{22
\end{equation}
let the $s_{y}$ matrix is
\begin{equation}
s_{y}=\left(
\begin{array}
[c]{ccc
b_{11} & b_{12} & b_{13}\\
b_{21} & b_{22} & b_{23}\\
b_{31} & b_{32} & b_{33
\end{array}
\right)  ,\label{23
\end{equation}
with the commutation relation
\begin{equation}
\lbrack s_{y},\alpha_{x}]=-i\hbar \alpha_{z},\hspace{0.2in}[s_{y},\alpha
_{z}]=i\hbar \alpha_{x},\label{24
\end{equation}
we get
\begin{equation}
s_{y}=\left(
\begin{array}
[c]{ccc
b & 0 & i\hbar \\
0 & b & 0\\
-i\hbar & 0 & b
\end{array}
\right)  ,\label{25
\end{equation}
let the $s_{z}$ matrix is
\begin{equation}
s_{z}=\left(
\begin{array}
[c]{ccc
c_{11} & c_{12} & c_{13}\\
c_{21} & c_{22} & c_{23}\\
c_{31} & c_{32} & c_{33
\end{array}
\right)  ,\label{26
\end{equation}
with the commutation relation
\begin{equation}
\lbrack s_{z},\alpha_{x}]=i\hbar \alpha_{y},\hspace{0.2in}[s_{z},\alpha
_{y}]=-i\hbar \alpha_{x},\label{27
\end{equation}
we get
\begin{equation}
s_{z}=\left(
\begin{array}
[c]{ccc
c & -i\hbar & 0\\
i\hbar & c & 0\\
0 & 0 & c
\end{array}
\right)  .\label{28
\end{equation}

In the following, we should calculate the eigenvalues of $s_{x}$, $s_{y}$,
$s_{z}$.\newline The $s_{x}$ eigenvalue problem $s_{x}\psi=\lambda_{1}\psi$
is
\begin{equation}
\left(
\begin{array}
[c]{ccc
a & 0 & 0\\
0 & a & -i\hbar \\
0 & i\hbar & a
\end{array}
\right)  \left(
\begin{array}
[c]{c
\psi_{1}\\
\psi_{2}\\
\psi_{3
\end{array}
\right)  =\lambda_{1}\left(
\begin{array}
[c]{c
\psi_{1}\\
\psi_{2}\\
\psi_{3
\end{array}
\right)  ,\label{29
\end{equation}
therefore the characteristic equation is
\begin{equation}
\left \vert
\begin{array}
[c]{ccc
a-\lambda_{1} & 0 & 0\\
0 & a-\lambda_{1} & -i\hbar \\
0 & i\hbar & a-\lambda_{1
\end{array}
\right \vert =0,\label{30
\end{equation}
i.e.,
\begin{equation}
(a-\lambda_{1})[(a-\lambda_{1})^{2}-\hbar^{2}]=0,\label{31
\end{equation}
when $a=0$, the roots $\lambda_{1}$ are
\begin{equation}
\lambda_{1}=\pm \hbar,\label{32
\end{equation}
the $s_{y}$ eigenvalue problem $s_{y}\psi=\lambda_{2}\psi$ is
\begin{equation}
\left(
\begin{array}
[c]{ccc
b & 0 & i\hbar \\
0 & b & 0\\
-i\hbar & 0 & b
\end{array}
\right)  \left(
\begin{array}
[c]{c
\psi_{1}\\
\psi_{2}\\
\psi_{3
\end{array}
\right)  =\lambda_{2}\left(
\begin{array}
[c]{c
\psi_{1}\\
\psi_{2}\\
\psi_{3
\end{array}
\right)  ,\label{33
\end{equation}
therefore the characteristic equation is
\begin{equation}
\left \vert
\begin{array}
[c]{ccc
b-\lambda_{2} & 0 & i\hbar \\
0 & b-\lambda_{2} & 0\\
-i\hbar & 0 & b-\lambda_{2
\end{array}
\right \vert =0,\label{34
\end{equation}
i.e.,
\begin{equation}
(b-\lambda_{2})[(b-\lambda_{2})^{2}-\hbar^{2}]=0,\label{35
\end{equation}
when $b=0$, the roots $\lambda_{2}$ are
\begin{equation}
\lambda_{2}=\pm \hbar,\label{36
\end{equation}
the $s_{z}$ eigenvalue problem $s_{z}\psi=\lambda_{3}\psi$ is
\begin{equation}
\left(
\begin{array}
[c]{ccc
c & -i\hbar & 0\\
i\hbar & c & 0\\
0 & 0 & c
\end{array}
\right)  \left(
\begin{array}
[c]{c
\psi_{1}\\
\psi_{2}\\
\psi_{3
\end{array}
\right)  =\lambda_{3}\left(
\begin{array}
[c]{c
\psi_{1}\\
\psi_{2}\\
\psi_{3
\end{array}
\right)  ,\label{37
\end{equation}
therefore the characteristic equation is
\begin{equation}
\left \vert
\begin{array}
[c]{ccc
c-\lambda_{3} & -i\hbar & 0\\
i\hbar & c-\lambda_{3} & 0\\
0 & 0 & c-\lambda_{3
\end{array}
\right \vert =0,\label{38
\end{equation}
i.e.,
\begin{equation}
(c-\lambda_{3})[(c-\lambda_{3})^{2}-\hbar^{2}]=0,\label{39
\end{equation}
when $c=0$, the roots $\lambda_{3}$ are
\begin{equation}
\lambda_{3}=\pm \hbar.\label{40
\end{equation}
By calculation, we obtain the spin matrices of photon, they are
\begin{equation}
s_{x}=\left(
\begin{array}
[c]{ccc
0 & 0 & 0\\
0 & 0 & -i\hbar \\
0 & i\hbar & 0
\end{array}
\right)  ,s_{y}=\left(
\begin{array}
[c]{ccc
0 & 0 & i\hbar \\
0 & 0 & 0\\
-i\hbar & 0 & 0
\end{array}
\right)  ,s_{z}=\left(
\begin{array}
[c]{ccc
0 & -i\hbar & 0\\
i\hbar & 0 & 0\\
0 & 0 & 0
\end{array}
\right)  .\label{41
\end{equation}
Obviously, these matrices are photon's spin matrices, which describe the spin
$s=1$. On the one hand, their eigenvalues are $\pm \hbar$. On the other hand,
the matrices square is
\begin{equation}
{\vec{s}}\hspace{0.05in}^{2}=s_{x}^{2}+s_{y}^{2}+s_{z}^{2}=2\left(
\begin{array}
[c]{ccc
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{array}
\right)  \hbar^{2}=s(s+1)\hbar^{2}\left(
\begin{array}
[c]{ccc
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{array}
\right)  ,\label{42
\end{equation}
i.e., the spin $s$ is
\begin{equation}
s=1.\label{43
\end{equation}
Comparing (\ref{6}) with (\ref{41}), we find
\begin{equation}
s_{x}=\alpha_{x},\hspace{0.2in}s_{y}=\alpha_{y},\hspace{0.2in}s_{z}=\alpha
_{z}.\label{44
\end{equation}

\section{The helicity of photon}

By studying the helicity of photon, we can obtain the photon left-handed and
right-handed. The helicity is defined as the projection of spin in the
momentum direction, it is
\begin{equation}
h=\frac{\vec{s}\cdot \vec{p}}{|\vec{p}|}=\frac{\vec{\alpha}\cdot \vec{p}
{|\vec{p}|},\label{45
\end{equation}
and
\begin{equation}
\vec{\alpha}\cdot \vec{p}=\alpha_{x}p_{x}+\alpha_{y}p_{y}+\alpha_{z
p_{z}=\left(
\begin{array}
[c]{ccc
0 & -ip_{z} & ip_{y}\\
ip_{z} & 0 & -ip_{x}\\
-ip_{y} & ip_{x} & 0
\end{array}
\right)  ,\label{46
\end{equation}
the $\vec{\alpha}\cdot \vec{p}$ eigenvalue problem $\vec{\alpha}\cdot \vec
{p}\psi=\lambda \psi$ is
\begin{equation}
\left(
\begin{array}
[c]{ccc
0 & -ip_{z} & ip_{y}\\
ip_{z} & 0 & -ip_{x}\\
-ip_{y} & ip_{x} & 0
\end{array}
\right)  \left(
\begin{array}
[c]{c
\psi_{1}\\
\psi_{2}\\
\psi_{3
\end{array}
\right)  =\lambda \left(
\begin{array}
[c]{c
\psi_{1}\\
\psi_{2}\\
\psi_{3
\end{array}
\right)  ,\label{47
\end{equation}
therefore the characteristic equation is
\begin{equation}
\left \vert
\begin{array}
[c]{ccc
-\lambda & -ip_{z} & ip_{y}\\
ip_{z} & -\lambda & -ip_{x}\\
-ip_{y} & ip_{x} & -\lambda
\end{array}
\right \vert =0,\label{48
\end{equation}
i.e.,
\begin{equation}
\lambda(\bar{p}^{2}-\lambda^{2})=0,\label{49
\end{equation}
the roots $\lambda$ are
\begin{equation}
\lambda=|\vec{p}|,\hspace{0.15in}-|\vec{p}|,\label{50
\end{equation}
and the helicity $h$ are
\begin{equation}
\hbar=+1,\hspace{0.15in}-1.\label{51
\end{equation}
When $\lambda=+1$ the photon is called right-handed photon, and when
$\lambda=-1$ the photon is called left-handed photon.

\section{The probability conservation equation of photon}

In the following, we should give the probability density and probability
conservation equation of photon.\newline The hermitian conjugate of (\ref{4})
is
\begin{equation}
-i\hbar \frac{\partial \psi^{\dag}}{\partial t}=i\hbar c\vec{\nabla}\psi^{\dag
}\cdot \vec{\alpha},\label{52
\end{equation}
multiplying (\ref{52}) by $\psi$, there is
\begin{equation}
-i\hbar \frac{\partial \psi^{\dag}}{\partial t}\psi=i\hbar c\vec{\nabla
\psi^{\dag}\cdot \vec{\alpha}\psi,\label{53
\end{equation}
multiplying (\ref{4}) by $\psi^{+}$, there is
\begin{equation}
i\hbar \psi^{\dag}(\frac{\partial \psi}{\partial t})=-i\hbar c\psi^{\dag
\vec{\alpha}\cdot \vec{\nabla}\psi,\label{54
\end{equation}
taking the difference, we get
\begin{equation}
i\hbar(\psi^{\dag}\frac{\partial \psi}{\partial t}+\frac{\partial \psi^{\dag
}{\partial t}\psi)+i\hbar c\psi^{\dag}\vec{\alpha}\cdot(\vec{\nabla
\psi)+i\hbar c\vec{\nabla}\psi^{\dag}\cdot \vec{\alpha}\psi=0,\label{55
\end{equation}
or
\begin{equation}
\frac{1}{c}\frac{\partial}{\partial t}(\psi^{\dag}\psi)+\psi^{\dag}\vec
{\alpha}\cdot(\vec{\nabla}\psi)+(\vec{\nabla}\psi^{\dag})\cdot \vec{\alpha
\psi=0,\label{56
\end{equation}
i.e.,
\begin{equation}
\frac{\partial \rho}{\partial t}+\nabla \cdot \vec{J}=0,\label{57
\end{equation}
where
\begin{equation}
\rho=\psi^{\dag}\psi,\hspace{0.25in}\vec{J}=c\psi^{\dag}\vec{\alpha
\psi,\label{58
\end{equation}
are the probability and probability current density of photon, respectively.

\section{The plane wave of free photon}

We have the spiron equation of free photon
\begin{equation}
i\hbar \frac{\partial}{\partial t}\psi=H\psi,\label{59
\end{equation}
where
\begin{equation}
H=c\hspace{0.02in}\vec{{\alpha}}\cdot \vec{p},\psi=\left(
\begin{array}
[c]{c
\psi_{1}\\
\psi_{2}\\
\psi_{3
\end{array}
\right)  ,\label{60
\end{equation}
since $\frac{\partial H}{\partial t}=0$ and $[\vec{p},H]=0$, the energy $E$
and momentum $\vec{p}$ of photon are conserved quantity. Their common
eigenstate is
\begin{equation}
\psi_{E,\vec{p}}(\vec{r},t)=u(\vec{p})e^{i(\vec{p}\cdot \vec{r}-Et)/\hbar
},\label{61
\end{equation}
where
\begin{equation}
u(\vec{p})=\left(
\begin{array}
[c]{c
u_{1}(\vec{p})\\
u_{2}(\vec{p})\\
u_{3}(\vec{p})
\end{array}
\right)  ,\label{62
\end{equation}
substituting equations (\ref{61}) and (\ref{62}) into (\ref{59}), we have
\begin{equation}
c\hspace{0.02in}\vec{{\alpha}}\cdot \vec{p}\hspace{0.1in}u(\vec{p
)=E\hspace{0.02in}u(\vec{p}),\label{63
\end{equation}
i.e.,
\begin{equation}
\left(  c\hspace{0.01in}\alpha_{x}p_{x}+c\hspace{0.01in}\alpha_{y
p_{y}+c\hspace{0.01in}\alpha_{z}p_{z}\right)  \left(
\begin{array}
[c]{c
u_{1}\\
u_{2}\\
u_{3
\end{array}
\right)  =E\left(
\begin{array}
[c]{c
u_{1}\\
u_{2}\\
u_{3
\end{array}
\right)  ,\label{64
\end{equation}
and expanding (\ref{64}), we get
\begin{equation}
\left(
\begin{array}
[c]{ccc
0 & -ic\hspace{0.01in}p_{z} & ic\hspace{0.01in}p_{y}\\
ic\hspace{0.01in}p_{z} & 0 & -ic\hspace{0.01in}p_{x}\\
-ic\hspace{0.01in}p_{y} & ic\hspace{0.01in}p_{x} & 0
\end{array}
\right)  \left(
\begin{array}
[c]{c
u_{1}\\
u_{2}\\
u_{3
\end{array}
\right)  =E\left(
\begin{array}
[c]{c
u_{1}\\
u_{2}\\
u_{3
\end{array}
\right)  ,\label{65
\end{equation}
or
\begin{align}
Eu_{1}+icp_{z}u_{2}-icp_{y}u_{3} &  =0,\label{66}\\
icp_{z}u_{1}-Eu_{2}-icp_{x}u_{3} &  =0,\label{67}\\
icp_{y}u_{1}-icp_{x}u_{2}+Eu_{3} &  =0,\label{68
\end{align}
therefore the characteristic equation is
\begin{equation}
\left \vert
\begin{array}
[c]{ccc
E & icp_{z} & -icp_{y}\\
icp_{z} & -E & -icp_{x}\\
\hspace{0.05in}icp_{y} & -icp_{x} & E
\end{array}
\right \vert =0,\label{69
\end{equation}
the roots $E$ are
\begin{equation}
E_{1}=+c|\vec{p}|,\hspace{0.2in}E_{2}=-c|\vec{p}|.\label{70
\end{equation}
From equations (\ref{67}) and (\ref{68}), we have
\begin{align}
icp_{z}p_{y}u_{1}-Ep_{y}u_{2}-icp_{x}p_{y}u_{3} &  =0,\label{71}\\
icp_{y}p_{z}u_{1}-icp_{x}p_{z}u_{2}+Ep_{z}u_{3} &  =0,\label{72
\end{align}
taking the difference of equations (\ref{71}) and (\ref{72}), we get
\begin{equation}
(Ep_{y}-icp_{x}p_{z})u_{2}+(Ep_{z}+icp_{x}p_{y})u_{3}=0,\label{73
\end{equation}
i.e.,
\begin{equation}
\frac{u_{2}}{u_{3}}=-\frac{Ep_{z}+icp_{x}p_{y}}{Ep_{y}-icp_{x}p_{z
},\label{74
\end{equation}
substituting (\ref{74}) into (\ref{66}), there is
\begin{equation}
\frac{u_{1}}{u_{3}}=\frac{ic(p_{y}^{2}+p_{z}^{2})}{Ep_{y}-icp_{x}p_{z
},\label{75
\end{equation}
by (\ref{74}) and (\ref{75}), we have
\begin{equation}
\frac{u_{1}}{u_{2}}=-\frac{ic(p_{y}^{2}+p_{z}^{2})}{Ep_{z}+icp_{x}p_{y
},\label{76
\end{equation}
the $u(p)$ can be written as
\begin{equation}
u(p)=\left(
\begin{array}
[c]{c
u_{1}\\
u_{2}\\
u_{3
\end{array}
\right)  =N\left(
\begin{array}
[c]{c
ic(p_{y}^{2}+p_{z}^{2})\\
-(Ep_{z}+icp_{x}p_{y})\\
Ep_{y}-icp_{x}p_{z
\end{array}
\right)  ,\label{77
\end{equation}
where $N$ is normalization constant, it can be obtained by the normalization
\begin{align}
u^{\dag}(p)u(p) &  =N^{2}\left(
\begin{array}
[c]{ccc
-ic(p_{y}^{2}+p_{z}^{2}) & -(Ep_{z}-icp_{x}p_{y}) & (Ep_{y}+icp_{x}p_{z})
\end{array}
\right)  \left(
\begin{array}
[c]{c
ic(p_{y}^{2}+p_{z}^{2})\\
-(Ep_{z}+icp_{x}p_{y})\\
Ep_{y}-icp_{x}p_{z
\end{array}
\right)  \nonumber \\
&  =2E^{2}N^{2}(p_{y}^{2}+p_{z}^{2})=1,\label{78
\end{align}
i.e.,
\begin{equation}
N=\sqrt{\frac{1}{2E^{2}(p_{y}^{2}+p_{z}^{2})}},\label{79
\end{equation}
then
\begin{equation}
u(p)=\sqrt{\frac{1}{2E^{2}(p_{y}^{2}+p_{z}^{2})}}\left(
\begin{array}
[c]{c
ic(p_{y}^{2}+p_{z}^{2})\\
-(Ep_{z}+icp_{x}p_{y})\\
Ep_{y}-icp_{x}p_{z
\end{array}
\right)  ,\label{80
\end{equation}
The plane wave solution of free photon is
\begin{equation}
\psi_{E,\vec{P}}(\vec{r},t)=\sqrt{\frac{1}{2E^{2}(p_{y}^{2}+p_{z}^{2})
}\left(
\begin{array}
[c]{c
ic(p_{y}^{2}+p_{z}^{2})\\
-(Ep_{z}+icp_{x}p_{y})\\
Ep_{y}-icp_{x}p_{z
\end{array}
\right)  e^{i(\vec{p}\cdot \vec{r}-Et)/\hbar}.\label{81
\end{equation}

\section{The spin wave function of photon}

From equations (\ref{41}) and (\ref{42}), we can find $\vec{s}\hspace
{0.05in}^{2}$ commute with $s_{x}$, $s_{y}$ and $s_{z}$, we should calculate
the common eigenstate of $\vec{s}\hspace{0.05in}^{2}$ and $s_{z}$, they are
\begin{equation}
\vec{s}\hspace{0.05in}^{2}\chi_{\mu}=2\hbar^{2}\chi_{\mu},\label{82
\end{equation
\begin{equation}
s_{z}\chi_{\mu}=\mu \hbar \chi_{\mu},\label{83
\end{equation}
where the common eigenstate $(\chi_{\mu})^{T}=(\varphi_{1},\varphi_{2
,\varphi_{3})$, the equation (\ref{83}) can be written as
\begin{equation}
\left(
\begin{array}
[c]{ccc
0 & -i\hbar & 0\\
i\hbar & 0 & 0\\
0 & 0 & 0
\end{array}
\right)  \left(
\begin{array}
[c]{c
\varphi_{1}\\
\varphi_{2}\\
\varphi_{3
\end{array}
\right)  =\mu \left(
\begin{array}
[c]{c
\varphi_{1}\\
\varphi_{2}\\
\varphi_{3
\end{array}
\right)  ,\label{84
\end{equation}
therefore the characteristic equation is
\begin{equation}
\left \vert
\begin{array}
[c]{ccc
-\mu & -i & 0\\
i & -\mu & 0\\
0 & 0 & -\mu
\end{array}
\right \vert =0,\label{85
\end{equation}
i.e.,
\begin{equation}
-\mu(\mu^{2}-1)=0,\label{86
\end{equation}
the roots $\mu$ are
\begin{equation}
\mu_{1}=0,\hspace{0.1in}\mu_{2}=1,\hspace{0.1in}\mu_{3}=-1,\label{87
\end{equation}
substituting $\mu_{1}=0$ into (\ref{84}), we get
\begin{equation}
\left \{
\begin{array}
[c]{c
-i\varphi_{2}=0\\
i\varphi_{1}=0
\end{array}
\right.  ,\label{88
\end{equation}
i.e.,
\begin{equation}
\varphi_{1}=\varphi_{2}=0,\hspace{0.2in}\varphi_{3}\neq0,\label{89
\end{equation}
the normalization spin wave function is
\begin{equation}
\chi_{0}=\left(
\begin{array}
[c]{c
0\\
0\\
1
\end{array}
\right)  ,\label{90
\end{equation}
substituting $\mu_{2}=1$ into (\ref{84}), we get
\begin{equation}
\left \{
\begin{array}
[c]{c
-i\varphi_{2}=\varphi_{1}\\
i\varphi_{1}=\varphi_{2}\\
\varphi_{3}=0
\end{array}
\right.  ,\label{91
\end{equation}
we can take
\begin{equation}
\varphi_{1}=-1,\hspace{0.2in}\varphi_{2}=-i,\hspace{0.2in}\varphi
_{3}=0,\label{92
\end{equation}
the normalization spin wave function is
\begin{equation}
\chi_{1}=-\frac{1}{\sqrt{2}}\left(
\begin{array}
[c]{c
1\\
i\\
0
\end{array}
\right)  ,\label{93
\end{equation}
substituting $\mu_{3}=-1$ into (\ref{84}), we get
\begin{equation}
\left \{
\begin{array}
[c]{c
-i\varphi_{2}=-\varphi_{1}\\
i\varphi_{1}=-\varphi_{2}\\
\varphi_{3}=0
\end{array}
\right.  ,\label{94
\end{equation}
we can take
\begin{equation}
\varphi_{1}=1,\hspace{0.2in}\varphi_{2}=-i,\hspace{0.2in}\varphi
_{3}=0,\label{95
\end{equation}
the normalization spin wave function is
\begin{equation}
\chi_{-1}=\frac{1}{\sqrt{2}}\left(
\begin{array}
[c]{c
1\\
-i\\
0
\end{array}
\right)  ,\label{96
\end{equation}
and these spin wave functions satisfy the normalization condition
\begin{equation}
\sum_{\alpha}\chi_{\mu}^{\ast}(\alpha)\chi_{\mu^{\prime}}(\alpha)=\delta
_{\mu \mu^{\prime}}.\label{97
\end{equation}

\section{The spiron wave equation of non-free photon}

In view of the above, we have given the spiron wave equation of free photon.
In the following, we should give the spiron wave equation of non-free
photon.\newline

For the non-free particle, the Einstein's dispersion relation is
\begin{equation}
(E-V)^{2}=c^{2}\vec{p}\hspace{0.05in}^{2}+m_{0}^{2}c^{4},\label{98
\end{equation}
factorizing (\ref{98}), we obtain
\begin{equation}
(E-V)^{2}-c^{2}\vec{p}\hspace{0.05in}^{2}-m_{0}^{2}c^{4}=(E-V-c\vec{p
\cdot \vec{\alpha}-m_{0}c^{2}\beta)(E-V+c\vec{p}\cdot \vec{\alpha}+m_{0
c^{2}\beta).\label{99
\end{equation}
For photon, $m_{0}=0$, equation (\ref{99}) becomes
\begin{equation}
(E-V)^{2}-c^{2}\vec{p}\hspace{0.05in}^{2}=(E-V-c\vec{p}\cdot \vec{\alpha
})(E-V+c\vec{p}\cdot \vec{\alpha})=0,\label{100
\end{equation}
or
\begin{equation}
(E-V-c\vec{p}\cdot \vec{\alpha})=0,\label{101
\end{equation}
canonical quantization equation (\ref{101}), we have
\begin{equation}
i\hbar \frac{\partial}{\partial t}\psi=-ic\hbar \vec{\alpha}\cdot \vec{\nabla
}\psi+V\psi,\label{102
\end{equation}
the potential energy of photon in medium is\cite{s17}
\begin{equation}
V=\hbar \omega(1-n),\label{103
\end{equation}
the spiron equation of photon in medium is
\begin{equation}
i\hbar \frac{\partial}{\partial t}\psi=-ic\hbar \vec{\alpha}\cdot \vec{\nabla
}\psi+\hbar \omega(1-n)\psi,\label{104
\end{equation}
by the method of separation variable
\begin{equation}
\psi(\vec{r},\vec{t})=\psi(\vec{r})f(t),\label{105
\end{equation}
the equation (\ref{104}) becomes
\begin{equation}
i\hbar \frac{\partial}{\partial t}\psi(\vec{r})f(t)=[-ic\hbar \vec{\alpha
\cdot \vec{\nabla}+\hbar \omega(1-n)]\psi(\vec{r})f(t),\label{106
\end{equation}
or
\begin{equation}
\frac{i\hbar \frac{\partial}{\partial t}f(t)}{f(t)}\psi^{\dag}(\vec{r
)\psi(\vec{r})=\psi^{\dag}(\vec{r})[-ic\hbar \vec{\alpha}\cdot \vec{\nabla
}+\hbar \omega(1-n)]\psi(\vec{r}),\label{107
\end{equation}
or
\begin{equation}
\frac{i\hbar \frac{\partial}{\partial t}f(t)}{f(t)}=\frac{\psi^{\dag}(\vec
{r})[-ic\hbar \vec{\alpha}\cdot \vec{\nabla}+\hbar \omega(1-n)]\psi(\vec{r
)}{\psi^{\dag}(\vec{r})\psi(\vec{r})}=E,\label{108
\end{equation}
we have
\begin{equation}
f(t)=f_{0}e^{-iEt/\hbar},\label{109
\end{equation}
and
\begin{equation}
\psi^{\dag}(\vec{r})[-ic\hbar \vec{\alpha}\cdot \vec{\nabla}+\hbar
\omega(1-n)]\psi(\vec{r})=E\psi^{\dag}(\vec{r})\psi(\vec{r}),\label{110
\end{equation}
i.e.,
\begin{equation}
\lbrack-ic\hbar \vec{\alpha}\cdot \vec{\nabla}+\hbar \omega(1-n)]\psi(\vec
{r})=E\psi(\vec{r}),\label{111
\end{equation}
where $E$ is photon total energy. Equation (\ref{111}) is time-dependent
spiron equation of photon, and the general spiron equation of photon in
arbitrary potential energy $V$ is
\begin{equation}
\lbrack-ic\hbar \vec{\alpha}\cdot \vec{\nabla}+V]\psi(\vec{r})=E\psi(\vec
{r}).\label{112
\end{equation}

\section{Conclusion}

In this paper, we have proposed the spiron equation of free and non-free
photon, and give the spin operator and spin wave function of photon. We can
further study two and multi-photon spin wave function, and can obtain
multi-photon entanglement state. We calculate the helicity of photon and prove
there are left-handed and right-handed photon. By the spiron equation of
non-free photon, we can study the quantum property of photon in medium. We
think the spiron equation of free and non-free photon can be widely used in
the future.

redpajama_set_name: RedPajamaArXiv