extfieldsofstudy:
Economics
Mathematics
Computer Science
Medicine

text:
Vast Volatility Matrix Estimation using High Frequency Data for Portfolio Selection

Portfolio allocation with gross-exposure constraint is an effective method to increase the efficiency and stability of selected portfolios among a vast pool of assets, as demonstrated in Fan et al (2008). The required high-dimensional volatility matrix can be estimated by using high frequency financial data. This enables us to better adapt to the local volatilities and local correlations among vast number of assets and to increase significantly the sample size for estimating the volatility matrix. This paper studies the volatility matrix estimation using high-dimensional high-frequency data from the perspective of portfolio selection. Specifically, we propose the use of"pairwise-refresh time"and"all-refresh time"methods proposed by Barndorff-Nielsen et al (2008) for estimation of vast covariance matrix and compare their merits in the portfolio selection. We also establish the concentration inequalities of the estimates, which guarantee desirable properties of the estimated volatility matrix in vast asset allocation with gross exposure constraints. Extensive numerical studies are made via carefully designed simulations. Comparing with the methods based on low frequency daily data, our methods can capture the most recent trend of the time varying volatility and correlation, hence provide more accurate guidance for the portfolio allocation in the next time period. The advantage of using high-frequency data is significant in our simulation and empirical studies, which consist of 50 simulated assets and 30 constituent stocks of Dow Jones Industrial Average index.

Introduction
The mean-variance efficient portfolio theory by Markowitz (1952Markowitz ( , 1959 has profound impact on modern finance. Yet, its applications to practical portfolio selection face a number of challenges. It is well known that the selected portfolios depend too sensitively on the expected future returns and volatility matrix (Klein and Bawa, 1976;Best and Grauer, 1991;Chopra and Ziemba, 1993). This leads to the puzzle postulated by Jagannathan and Ma (2003) why no short-sale portfolio outperforms the efficient Markowicz portfolio. See also De Roon, et al. (2001) on the study of optimal no-short sale portfolio on emerging market. The sensitivity on the dependence can be effectively addressed by the introduction of the constraint on the gross exposure of portifolios (Fan et al., 2008b). In particular, Fan et al. (2008b) shows, with non-asymptotic inequalities, that for a range of gross exposure constraint parameters, the actual risk of an empirically selected optimal portfolio, the actual risk of the theoretically optimal portfolio, and the estimated risk of an empirically selected optimal portfolio are in fact close. The accuracy depends only on the gross exposure parameter and the maximum componentwise estimation error of expected returns and covariance matrix -there is little error accumulation effect. The results are demonstrated also by both simulation and empirical studies. This gives not only a theoretical answer to the puzzle postulated by Jagannathan and Ma (2003) but also paves a way for optimal portfolio selection in practice.
The second challenge of the implementation of Markowitz's portfolio selection theory is the intrinsic difficulty of the estimation of the large volatility matrix. This is well documented in the statistics and econometrics literature even for the static large covariance matrix (Johnstone, 2001;Bickel and Levina, 2008;Fan, et al., 2008a;Lam and Fan, 2009;Rothman et al., 2009). The additional challenge comes from the time-varying nature of a large volatility matrix. For a short and medium holding period (one day or one week, say), the expected volatility matrix in the near future can be very different from the average of the expected volatility matrix over a long time horizon (the past one year, say). As a result, even if we know exactly the realized volatility matrix in the past, the bias can still be large. This calls for a stable and robust portfolio selection. The portfolio allocation under the gross exposure constraint provides a needed solution. To reduce the bias of the forecasted expected volatility matrix, we need to shorten the learning period to better capture the dynamics of the time-varying volatility matrix, adapting better to the local volatility and correlation. But this is at the expense of a reduced sample size. The wide availability of high-frequency data provides sufficient amount of data for reliable estimation of the volatility matrix.
Recent years have seen dramatic developments in the study of high frequency data in integrated volatility. Statisticians and econometricians have been focusing on the interesting and challenging problem of volatility estimation in the presence of market microstructure noise and asynchronous tradings, which are the style features of high-frequency financial data. The progresses are very impressive with a large literature. Assuming the price processes follow Brownian semimartingales to satisfy the no-arbitrage based characterizations (Delbaen and Schachermayer, 1994), if there were no market microstructure noise, and if the processes are observed synchronously on grids that become denser, classical results in stochastic calculus show that the realized variance and realized covariance are consistent estimators of the quadratic variation and quadratic co-variation of two price processes; see for example Karatzas and Shreve (2000) and Jacod and Shiryaev (2003). When directly applied to high-frequency financial data, however, Andersen et al. (2000) show that the realized variance exhibits a large positive bias when the sampling frequency gets higher, through their famous signature plots; Epps (1979) documented that the correlation estimates based on the realized covariances tend to be biased toward zero when sampled at high frequencies. The recent developments have enabled us to understand much better the signature plots and Epps effect. Analytical explanations of how the market microstructure noise and asynchronization may affect the estimates and ways to correct for the biases have been given. In particular, in the one dimensional case when the focus is on estimation of integrated volatility, Aït-Sahalia, et al. (2005) discussed a subsampling scheme; Zhang, et al. (2005) proposed a two-scale estimate which was extended and improved by Zhang (2006) to multiple scales; Fan and Wang (2007) separated jumps from diffusions in presence of market microstructural noise using a wavelet method; the robustness issues are addressed by Li and Mykland (2007); the realized kernel methods are proposed and thoroughly studied in Barndorff-Nielsen et al. (2009a,b) ;Jacod, et al. (2009) proposed a preaveraging approach to reduce the market microstructral noise; Xiu (2008) demonstrated that a simple quasi-likelihood method achieves the optimal rate of convergence for estimating integrated volatility. For estimation of integrated covariation, the non-synchronized trading issue was first addressed by Hayashi and Yoshida (2005) in absence of the microstructural noise; the kernel method with refresh time idea was first proposed by Barndorff-Nielsen et al. (2008); Zhang (2009) extend the two-scale method to study the integrated covariation using a previous tick method; Wang, et al. (2009) aggregate daily integrated volatility matrix via a factor model; Aït-Sahalia, et al. (2010) extend the quasi-maximum likelihood method; Kinnebrock et al. (2009) extend the pre-averaging technique.
The aim of this paper is to study the volatility matrix estimation using high-dimensional highfrequency data from the perspective of financial engineering. Specifically, our main topic is how to extract the covariation information from high-frequency data for asset allocation and how effective they are. Two particular strategies are proposed for handling the non-synchronized trading: "pairwise-refresh" and "all-refresh" schemes. The former retains much more data points and estimates covariance matrix componentwise, which is usually not semi-positive definite, whereas the latter retains far less data points and the resulting covariance matrix is usually semi-positive definite. As a result, the former has a better componentwise estimation error and is better in controlling risk approximation mentioned in the first paragraph of the introduction. However, the merits between the two methods are not that simple. In implementation, quadratic programming algorithms require the estimated covariance matrix to be semi-positive definite. Therefore, we need to project the estimate of covariance matrix based on the "pairwise-refresh" scheme onto the space of the semi-positive definite matrices. However, the projections distort the accuracy of the elementwise estimation. As a result, the pairwise-refresh scheme does not have much more advantage than the all-refresh method, though the former is very easy to implement. However, both methods significantly outperform the methods based on low frequency data, since they adapt better to the time-varying volatilities and correlations. The comparative advantage is more dramatic when there are rapid changes of the volatility matrix over time. This will be demonstrated in both simulation and empirical studies.
As mentioned in the introduction and demonstrated in Section 2, the accuracy of portfolio risk relative to the theoretically optimal portfolio is governed by the maximum elementwise estimation error. How does this error grow with the number of assets? Thanks to the concentration inequalities derived in this paper, it grows only at the logarithmic order of the number of assets. This gives a theoretical endorsement why the portfolio selection problem is feasible for vast portfolios.
The paper is organized as follows. Section 2 gives an overview of portfolio allocation using high-frequency data. Section 3 studies the volatility matrix estimation using high-frequency data from the perspective of asset allocation, where the analytical results are also presented. How well our idea works in simulation and empirical studies can be found in Sections 3 and 4, respectively. Conclusions are given in Section 5. Technical conditions and proofs are relegated to the appendix.

Problem Setup
Consider a pool of p assets, with log-price processes X s ) T the vector of the log-price processes at time s. Suppose they follow an Itô process, namely, where W t is the vector of p-dimensional standard Brownian motions. The drift vector µ t and the instantaneous variance S t can be stochastic processes and are assumed to be bounded and independent of W t .
A given portfolio with the allocation vector w at time t and a holding period τ has the log-return where w T 1 = 1 and with E t denoting the conditional expectation given the history up to time t. Let w + be the propotion of long positions and w − be the proposition of the short positions. Then, w 1 = w + + w − is the gross exposure of the portfolio. To simplify the problem, following Jagannathan and Ma (2003) and Fan et al. (2008b) and other papers in the literature, we consider only the risk optimization problem. In practice, the expected return constraint can be replaced by the constraints of sectors or industries, to avoid unreliable estimates of the expected return vector. For a short-time horizon, the expected return is usually negligible. Following Fan et al. (2008b), we consider the following risk optimization under gross exposure constraints: where c is the total exposure allowed. Note that using w + − w − = 1, the problem (4) puts equivalently the constraint on the proportion of the short positions: w − ≤ (c − 1)/2.
Problem (4) involves the conditional expected volatility matrix (3) in the future. Unless we know exactly the dynamic of the volatility process, this is usually unknown, even if we observed the entire continuous paths up to the current time t. As a result, we rely on the approximation even with ideal data that we were able to observe the processes continuously without error. The typical approximation is for an appropriate window width h and we estimate t t−h S u du based on the historical data at the time interval [t − h, t].
The approximation (5) holds reasonably well when τ and h are both small. This relies on the continuity assumptions: local time-varying volatility matrices are continuous in τ . The approximation is also reasonable when both τ and h are large. This relies on the stationarity assumption so that both quantity will be approximately ES u , when the stochastic volatility matrix S u is stationary. The approximation is not good when τ is small whereas h is large as long as S u is time varying, whether or not the stochastic volatility S u is stationary or not. In other words, when the holding time horizon τ is short, as long as S u is time varying, we can only use a short time window [t − h, t] to estimate Σ t,τ . The recent arrivals of high-frequency data make this problem feasible.
The approximation error in (5) can not usually be evaluated unless we have a specific parametric model on the stochastic volatility matrix S u . However, this is at the risk of model misspecifications and nonparametric approach is usually preferred for high-frequency data. With p 2 elements are approximated, which can be in the order of hundred of thousands or millions, a natural question to ask is whether these errors accumulate and whether the result (risk) is stable. The gross-exposure constraint gives a stable solution to the problem as shown in Fan et al. (2008b).

Risk approximations with gross exposure constraints
The utility of gross-exposure constraint can easily be seen through the following inequality. Let Σ t,τ be an estimated covariance matrix and be estimated risk of the portfolio. Then, for any portfolio with gross-exposure w 1 ≤ c, we have whereσ i,j and σ i,j are respectively the (i, j) elements ofΣ t,τ and Σ t,τ , and is the maximum componentwise estimation error. The risk approximation (7) reveals that there is no error accumulation effect when gross exposure c is moderate.
From now on, we drop the dependence of t and τ whenever there is no confusion. This facilitates the notation. Fan et al. (2008b) showed further that the risks of optimal portfolios are indeed close. Let w opt = argmin w T 1=1, ||w|| 1 ≤c R(w),ŵ opt = argmin w T 1=1, ||w|| 1 ≤cR (w) be respectively the theoretical (oracle) optimal allocation vector we want and the estimated optimal allocation vector we get. Then, R(w opt ) is the theoretical minimum risk and R(ŵ opt ) is the actual risk of our selected portfolio, whereasR(ŵ opt ) is our perceived risk, which is the quantity known to financial econometricians. They showed that with a p = |Σ − Σ| ∞ , which usually grows slowly with the number of assets p. These reveal that the three relevant risks are in fact close as long as the gross-exposure parameter c is moderate and the maximum componentwise estimation error a p is small.
The above risk approximations hold for any estimate of covariance matrix. It does not even requireΣ a semi-positive definite matrix. This facilitates significantly the method of covariance matrix estimation. In particular, the elementwise estimation methods are allowed. In fact, since the approximation errors in (9), (10) and (11) are all controlled by the maximum elementwise estimation error, it can be advantageous to use elementwise estimation methods. This is particularly the case for the high-frequency data where trading are non-synchronized. The synchronization can be done pairwisely or for all assets. The former retains much more data than the latter, as shown in the next section.
3 Estimation of Covariance Matrix Using High Frequency Data

All-refresh method and pairwise-refresh method
Estimating high-dimensional volatility matrix using high-frequency data is a challenging task. One of the challenges is the non-synchronicity of trading. Several synchronization schemes have been proposed. The refresh time method is proposed in Barndorff-Nielsen et al. (2008) and the previous tick method is proposed in Zhang (2009). The former uses more efficiently the available data and will be used in this paper.
The idea of refresh time is to wait until all assets are traded at least once at time v 1 (say) and then use the last price traded before or at v 1 of each asset as its price at time v 1 . This obtains one synchronized price vector at time v 1 . The clock now starts again. Wait until all assets are traded at least once at time v 2 (say) and again use the previous tick price of each asset as its price at time v 2 . This yields the second synchronized price vector at time v 2 . Repeat the process until all available trading data are synchronized. Clearly, the process discards a large portion of the available trades: After each synchronization, we always wait until the slowest stock to trade once. But this is the most efficient synchronization scheme. We will refer this synchorization scheme as the "all-refresh time" (The method is called all-refresh method for short). Barndorff-Nielsen et al. (2008) advocate the kernel method to estimate integrated volatility matrix after synchronization, but this can also be done by using other methods. The advantage of the all-refresh method is that the estimated covariance matrix can be made semi-positive definite.
A more efficient method to use the available sample is the pairwise refresh time scheme, which synchronizes the trading for each pair of assets separately (The method is called pairwise-refresh method for short). This retains far more data points, but we have to estimate the covariance matrix elementwise. The resulting covariance matrix is not necessarily semi-positive definite. Thanks to the gross exposure constraint, this is still applicable to the portfolio selection problems, as long as the elementwise estimation error is small. See (7) -(11). The pairwise-refresh scheme makes far more efficient use of the rich information in high-frequency data, and enables us to estimate each element in the volatility matrix more precisely, which helps improve the efficiency of the selected portfolio. We will study the merits of these two methods.
The pairwise estimation method allows us to use a wealth of univariate integrated volatility estimators, such as the two-scale realized volatility (TSRV) , the multi-scale realized volatility (MSRV) (Zhang, 2006), the wavelet method (Fan and Wang, 2007), the realized kernel method (Barndorff-Nielsen et al., 2009a,b), the pre-averaging approach (Jacod, et al., 2009) and the QMLE method (Xiu, 2008). For any given two assets with log-price processes X (i) t and X (j) t , with pairwise-refresh times, the synchronized prices of X With the univariate estimate of the integrated volatilities < X (i) + X (j) > and < X (i) − X (j) >, the integrated covariantion can be estimated aŝ In particular, the diagonal elements are estimated by the method itself. When the TSRV is used, this results in the two-scale realized covariance (TSCV) estimate (Zhang, 2009).

Pairwise refresh method and TSCV
We now focus on the pairwise estimation method. To facilitate the notation, we reintroduce it.
We consider two log price processes X and Y that satisfy where cor(B . For the two processes X and Y , consider the problem of estimating X, Y T with T = 1. Denote by T n the observation times of X and S m the observation times of Y . Denote the elements in T n and S m by {τ n,i } n i=0 and {θ m,i } m i=0 respectively, in an ascending order (τ n,0 and θ m,0 are set to be 0). We assume that the actual log-prices are not observable, but are observed with microstructure noises: where X o and Y o are the observed transaction prices in the logarithmic scale, and X and Y are the latent log prices govern by the stochastic dynamics (13). We assume that the microstructure noise ǫ X i and ǫ Y i processes are independent of the X and Y processes and that Note that this assumption is mainly for the simplicity of presentation; as we can see from the proof, one can for example easily replace the Gaussian assumption with the sub-Gaussian assumption without affecting our results.
The pairwise refresh time V = {v 0 , v 1 , · · · , vñ} can be obtained by setting v 0 = 0, and whereñ is the total number of refresh times in the interval (0, 1]. The actual sample times for the two individual processes X and Y that correspond to the refresh times are which is really the previous-tick measurement.
We study the property of the TSCV based on the asynchronous data: where Zhang (2009), the optimal choice of K has order K = O(ñ 2/3 ), J can be taken to be a constant such as 1. In the following analysis, we consider the specific case when J = 1 (orn J =ñ) andn K = O(ñ 1/3 ).
When either the microstructure error or the asynchronicity exists, the realized covariance is seriously biased. An asymptotic normality result in Zhang (2009) reveals that TSCV can simultaneously remove the bias due to the microstructure error and the bias due to the asynchronicity. However, this result is not adequate for our application to the vast volatility matrix estimation. The maximum componentwise estimation error a p depends on the number of assets p. To understand its impact on a p , we need to establish the concentration inequality. In particular, for a sufficiently for two positive constants a and C, then We will show in the next section that the result indeed holds with a = 1/2 and n replaced by the minimum subsample size. Hence the impact of the number of assets is limited, only of the logarithmic order.

Concentration Inequalities
Inequality (17) requires the conditions on both diagonal elements and off-diagonal elements. Technically, they are treated differently. For the diagonal cases, the problem corresponds to the estimation of integrated volatility and there is no issue of asynchronicity. TSCV (16) reduces to TSRV , which is explicitly given by where As shown in Zhang, et al. (2005), the optimal choice of K has order K = O(n 2/3 ) and J can be taken to be a constant such as 1. Again, for the TSRV, in the following analysis, we will consider the specific case when J = 1 (orn J = n) andn K = O(n 1/3 ).
To facilitate the reading, we relegate the technical conditions and proofs to the appendix. The following two results establish the concentration inequalities for the integrated volatility and integrated covariation.
Theorem 1. Let X process be as in (13), and n be the total number of observations for the X process during the time interval (0,1]. Under Conditions 1-4 in section A.1, for x ∈ [0, cn 1/6 ], for positive constants c and C. A set of candidate values for c and C are given in (49) for the case when the TSRV parameters are chosen according to Condition 5.
Theorem 2. Let X and Y processes be as in (13), andñ be the total number of refresh times for the processes X and Y during time interval (0,1]. Under Conditions 1-5 in section A.1, for x ∈ [0, cñ 1/6 ], for positive constants c and C. A set of candidate values for c and C are given in (52) for the case when the TSCV parameters are chosen according to Condition 5.

Error rates on risk approximations
Having had the above concentration inequalities, we can now readily give an upper bound of the risk approximations. Consider the p log-price processes as in Section 2.1. Suppose the processes are observed with the market microstructure noises. Letñ (i,j) be the observation frequency obtained by the pairwise-refresh method for two processes X (i) and X (j) andñ * be the observation frequency obtained by the all-refresh method. Clearly,ñ (i,j) is typically much larger thanñ * . Hence, most elements are estimated more accurately using the pairwise-refresh method than using the all-refresh method. On the other hand, for less liquidly traded pairs, its observation frequency of pairwiserefresh time can not be an order of magnitude larger thanñ * .
Using (18), an application to Theorems 1 and 2 to each element in the estimated integrated covariance matrix yields whereñ min = min i,jñ (i,j) be the minimum number of observations of the pairwise-refresh time.
Note that based on our proofs which don't rely on particular properties of pairwise-refresh times, our results of Theorem 1 and Theorem 2 are applicable to all-refresh method as well, with the observation frequency of the pairwise-refresh times replaced by that of the all-refresh times. Hence, using the all-refresh time scheme, we have Clearly,ñ min is larger thanñ * . See Figure 2. Hence, the pairwise refresh method gives a somewhat more accurate estimate in terms of the maximum elementwise estimation error.

Projections of estimated volatility matrices
The risk approximations (9)-(11) hold for any solutions to (8) whether the matrixΣ is positive semi-definite or not. However, convex optimization algorithms typically require the positive semidefiniteness of the matrixΣ. Yet, the estimates based on the elementwise estimation sometimes can not satisfy this and even the one from all-refresh method can have the same problem if TSRV is used. This leads to the issue of how to project a symmetric matrix onto the space of positive semi-definite matrices.
There are two intuitive methods for projecting a p×p symmetric matrix A onto the space of positive semi-definite matrices. Consider the singular value decomposition: where Γ is an orthogonal matrix, consisting of p eigenvectors. The two intuitive appealing projection methods are where λ + j is the positive part of λ j and where λ − min is the negative part of the minimum eigenvalue. For both projection methods, the eigenvectors remain the same as those of A. When A is positive semi-definite matrix, we have obviously that A 1 = A 2 = A.
In applications, we apply the above transformations to the estimated correlation matrix A rather than directly to the volatility matrix estimateΣ. The correlation matrix A has diagonal elements of 1. The resulting matrix under the projection method (23) apparently still satisfies this property, whereas the one under the projection method (22) does not. As a result, the projection method (23) keeps the integrated volatility of each asset intact.
In the simulation and empirical studies, we applied both projections. It turns out that there is no significant difference between the two projection methods in terms of result. We decided to apply only the projection (23) in all numerical studies, as it keeps the individual volatility estimate intact.

Comparisons between pairwise-refresh and all-refresh methods
The pairwise-refresh method keeps far richer information in the high-frequency data than the allrefresh method. See Figure 2. Thus, it is expected to estimate each element more precisely. Yet, the estimated correlation matrix is typically not positive semi-definite. As a result, projection (23) can distort the accuracy of elementwise estimation. On the other hand, the all-refresh method is typically positive semi-definite or nearly so. The property (12) typically entails the positive semi-definiteness property, as long as the volatility estimator for X, X is always nonnegative. For example, using the realized kernel method as the building block, the positive semi-definite version can easily be obtained. Therefore, the projection (23) has less impact on the all-refresh method than on the pairwise-refresh method.
Risk approximations (9)-(11) are only the upper bounds. The upper bounds are controlled by a p , which has rates of convergence govern by (20) and (21). While the average number of observations of pairwise-refresh time is far larger than the number of observationsñ * of the allrefresh time, the minimum number of observations of pairwise-refresh timeñ min is not much larger thanñ * . Therefore, the upper bounds (20) and (21) are approximately of the same order. This together with the distortion due to projection do not leave much advantage for the pairwise-refresh method.

Simulation Studies
In this section, we simulate the market trading data using a reasonable stochastic model. As the latent prices and dynamics of simulations are known, our study on the risk profile is facilitated. It is a good tool to verify our theoretical results and to quantify the finite sample behaviors. In particular, we would like to demonstrate that high frequency data based approaches have a better risk profile than those based on the low frequency data.
Throughout this paper, the risk is referring to the standard deviation of portfolio's returns.
To avoid ambiguity, we call R(w opt ) the theoretical optimal risk or oracle risk, R n (ŵ opt ) the perceived optimal risk, and R(ŵ opt ) the actual risk of the perceived optimal allocation.

Design of Simulations
A slightly modified version of the simulation model in Barndorff-Nielsen et al. (2008) is used to generate the latent price processes of p traded assets. It is a multivariate factor model with stochastic volatilities. Specifically, the latent log-prices X given by N β . The integrated quadratic variation and covariation are given by The analytic formula for the conditional covariance matrix Σ t,τ in (3) can be found for our model, but we decide not to report it for brevity.
The number of assets p is taken to be 50. Slightly modified from Barndorff-Nielsen et al. (2008), the parameters is set to be (µ (i) , β To model the non-synchronicity, p independent Poisson processes with intensitive parameters λ 1 , λ 2 , · · · , λ p are used to simulate the trading times of the assets. Motivated by the US equity trading dataset (the total number of seconds in a common trading day of the US equity is 23400), we set the trading intensity parameters λ i 's to be 0.02i × 23400 for i = 1, 2, · · · , 50, meaning that the average numbers of trading times for each asset are spread out in the arithmetic sequence of the interval [468, 23400].

An oracle investment strategy and risk assessment
An oracle investment strategy is usually a decent benchmark for other portfolio strategies to be compared with. There are several oracle strategies. The one we choose is to make portfolio allocation based on the covariance matrix estimated using latent prices at the finest grid (one per second). Latent prices are the noise-free prices of each asset at every time points (one per second), which are unobservable in practice and is available to us only in the simulation. Therefore, for each asset, there are 23400 latent prices in a normal trading day. We will refer to the investment strategy based on the latent prices as the oracle or latent strategy. This strategy is not available for the empirical studies.
The assessment of risk is based on the high-frequency data. For a given portfolio strategy, its risk is computed based on the latent prices at the finest grid (one per second) for the in-the-sample simulation studies; its risk is computed based on the latent prices at every 15 minutes for the out-of-sample simulation studies; whereas for the empirical studies, the observed prices at every 15 minutes are used to assess its risk. This mitigates the influence of the microstructure noises. For the empirical study, we do not hold positions overnight therefore are immune to the overnight price jumps (we will discuss the details in Section 5).

In-sample Risk Approximation and Optimal Allocation
Based on the past h = 1 day, the latent prices (at the finest grid) based estimated TSCV covariance matrix (called latent covariance for short) is regarded as the true covariance matrix. There are several methods for estimating covariance matrix based on observed non-synchronized highfrequency data with microstructure noise. In particular, we employ all-refresh method based TSCV covariance matrix (called all-refresh TSCV covariance), all-refresh method based realized kernel covariance matrix (called all-refresh RK covariance, for short), and pairwise-refresh method based TSCV covariance matrix (called pairwise-refresh TSCV covariance). The all-refresh RK covariance is included since it is positive semi-definite and there is no distortion effect due to projection. The latent covariance serves as the oracle covariance matrix from which the actual portfolio risk of any portfolio is computed. The conditioning number of the latent covariance of the p = 50 assets ranges from 192.27 to 226.46, with median 210.34, across 100 simulations. The medians of the minimum and maximum eigenvalues are respectively 0.0004 and 0.0838. For the all-refresh RK approach, the bandwidth of the realized kernel H is chosen to be 1, which gives the best risk profile in our numerical analysis.
The efficiencies of using the rich high-frequency data between pairwise-refresh and all-refresh methods are contrasted. In particular, for each realization, we compute the median number of pairwise-refresh times median i,j (ñ (i,j) ), the minimum number of pairwise-refresh times M min = min i,j (ñ (i,j) ) (see (20)), and the number of all-refresh timesñ * (see (21)). The distributions of these three numbers are summarized in Figure 2. It is clear that the pairwise refresh scheme uses far more data on average, yet minimum number of pairwise-refresh time is not appreciably larger than that of refresh time.
To gain insights on the risk approximations, we consider 4 specific portfolios of the p = 50 assets with the weight vectors Their daily risks are computed based on various covariance estimators and are compared with the actual risk, which is computed based on the latent price. This is done across 100 simulations. The medians, robust standard deviation (defined as interquartile range divided by 1.35) and other characteristics are summarized in Table 1.
From the result, we can see that both the all-refresh TSRV and pairwise-refresh TSRV methods, especially all-refresh TSRV method, have a tendency to underestimate the risk in comparison with the latent risks, while all-refresh RK method has a tendency to overestimate the risk. In terms of the absolute risk difference from the oracle, for 3 out of the 4 portfolios, pairwise-refresh TSRV method outperforms the all-refresh TSRV method. The same relationship can be observed when we turn to the L 1 norm of the absolute covariance difference (a p ) as well. The RK method outperforms the TSRV method. These are in line with our expectation.
We now study the problem of the optimal portfolio allocation under gross exposure constraints. The optimal allocation vectors are computed based on the latent covariance, all-refresh TSCV covariance, all-refresh RK covariance and pairwise-refresh TSCV covariance and their actual risks are computed based on the latent covariance matrix. The medians of these actual risks against the gross exposure parameter c are depicted in Figure 3.
Firstly, the all-refresh RK method outperforms the two TSRV methods when the gross exposure is below 3.7. The pairwise-refresh TSRV method outperforms the all-refresh TSRV method where the gross exposure is smaller than 1.2. That agrees with what we expected since the smaller the gross exposure is, the tighter the bound (9) on the risk difference. It is obvious that the pairwiserefresh TSRV method gives an estimated covariance matrix with higher element-wise accuracy than the all-refresh TSRV method, therefore the former outperforms the latter where the bound is the tightest (gross exposure below 1.2) for this simulation design.
Secondly, all the methods produce an upward-sloping risk curve up to some point and an almost flat curve beyond that (the curve is clipped). This is mainly due to the fact that we use only the intra-day data for 1 trading day, which does not result in sufficient amount of data to yield a stable estimate of the 50 × 50 covariance matrix. As the result, the estimated covariance matrix can be ill-conditioned. As c increases, the selected portfolios become increasingly unstable. When c reaches 5 or so, the selected portfolio becomes basically a randomly selected portfolio. Hence, their actual risks become larger and flat afterwards.

Out-of-sample Optimal Allocation
One of the main purposes of this paper is to investigate the comparative advantage of the high frequency based methods against the low frequency based method (especially in the context of portfolio investment). Hence, it is essential for us to run the following out-of-sample investment strategy test which includes both the high frequency and low frequency based approaches. Moreover, since in the empirical studies, we do not know the latent asset prices, the out-of-sample test should be designed so that it can also be conducted in the empirical studies.
We simulate the prices of 50 traded assets using the model (24) and (25) with microstructure noise for the duration of 200 trading days (numbered as day 1, day 2, ..., day 200) and record all the tick-by-tick trading times and trading prices of the assets. We assume that there are no overnight jumps for asset prices, meaning one trading day's closing price of an asset is always the same as the next trading day's opening price of that asset.
We start investing 1 unit of capital into the pool of assets with low frequency and high frequency based strategies from day 101 (the portfolios are bought at the opening of day 101). For the low frequency strategy, we use the previous 100 trading days' daily closing prices to compute the sample covariance matrix and make the portfolio allocation accordingly with the gross exposure constraints. For the all-refresh high frequency strategies, we use the previous h = 10 trading days' tick-by-tick trading data, use all-refresh time to synchronize the trades of the assets before applying realized kernel and TSCV to estimate the integrated volatility matrix and make the portfolio allocation, while for the pairwise-refresh high frequency strategy, we use pairwise-refresh times to synchronize each pair of assets and apply TSCV to estimate the integrated covariance for the corresponding pair. With the projection technique (23), the resulting TSCV integrated volatility matrix can always be transformed to a positive semi-definite matrix which facilitates the optimization.
We run two investment strategies. In the first strategy, the portfolio is held for τ = 1 trading day before we re-estimate the covariation structure and adjust the portfolio weights accordingly. The second strategy is the same as the first one except for the fact that the portfolio is held for τ = 5 trading days before rebalance.
In the investment horizon (which is from day 101 to day 200 in this case), we record the 15minute portfolio returns based on the latent prices of the assets, the variation of the portfolio weights across 50 assets, and other relevant characteristics. While it appears that 100 trading days is short, calculating 15-minute returns increases the size of the relevant data for computing the risk by a factor of 26.
We study those portfolio features for a whole range of gross exposure constraint c from c = 1, which stands for the no-short-sale portfolio strategy, to c = 3. This is usually the relevant range of gross exposure for investment purpose.
The standard deviations and other characteristics of the strategy for τ = 1 are presented in Table 2 (the case τ = 5 is very similar, therefore omitted). The standard deviations represent the actual risks of the strategy. As we only optimize the risk profile, we should not look significantly on the returns of the optimal portfolios. They can not even be estimated accurately with such a short investment horizon. Figures 4 and 5 provides graphical details to these characteristics for both τ = 1 and τ = 5.   For both holding lengths τ = 1 and τ = 5, the all-refresh TSRV and pairwise-refresh TSRV approaches outperform significantly the low frequency one in terms of risk profile for the whole range of the gross exposure constraint. This supports our theoretical results and intuitions. The shorter estimation window allows these 2 high frequency approaches to deliver consistently better results than the low frequency one. The low-frequency strategy outperforms significantly the equalweight portfolio (see Figure 4 and Figure 5). Slightly surprising is the fact that the low frequency approach also outperforms the all-refresh RK approach. We believe it must be due to the instability of the estimated realized kernel covariance matrix.
All the risk curves attain their minimum around c = 1.2 (see Figure 4 and Figure 5), which falls into our expectation again, since that must be the point where the marginal increase in estimation error outpaces the marginal decrease in specification error. This, coupled with the result we get in the empirical studies section, will give us some guidelines about what gross exposure constraint to use in investment practice.
Firstly, the pairwise method outperforms the all-refresh method, as expected. Secondly, the risk of the low frequency approach only increases at a mild speed as the gross exposure constraint increases. A possible explanation is that only 50 assets is considered, therefore the estimation error accumulation effect is not dominating as badly as we were afraid it would be, given the low frequency covariance sampling window is the previous 100 trading days. Another possible reason could be that as the data is generated by a stationary stochastic model, the low frequency approach may be able to capture some of the stationarity within the model.
In terms of portfolio weights, neither the low frequency nor the high frequency optimal no-shortsale portfolios are well diversified with all approaches assigning a concentrated weight of around 20% to one individual asset. Their portfolio risks can be improved by relaxing the gross-exposure constraint (Figure 4 and Figure 5).

Empirical Studies
The risk minimization problem (6) has important applications in asset allocation. We demonstrate its application in the stock portfolio investment in the 30 Dow Jones Industrial Average (DJIA) constituent stocks (will be called the 30 DJIA stocks for short).
The Dow Jones Industrial Average is one of the several stock market indices created by Charles Dow, the editor of Wall Street Journal and a co-founder of Dow Jones and Company. It is an index that shows how 30 large, publicly-owned companies based in the United States have traded during a standard trading session in the stock market. We make the portfolio allocation to the constituents of the index as of Sep 30, 2008 (The individual components of the DJIA are occasionally changed as market conditions warrant.) To make asset allocation, we use the high frequency data of the 30 DJIA stocks from Jan 1, 2008 to September 30, 2008. These stocks are highly liquid. The intensity of trading for each given trading day is summarized by the maximum, minimum and median number of trades among these 30 stocks. The distributions of these summary statistics across those 9 months (189 trading days) are summarized in Figure 6. The period covers the birth of financial crisis in 2008.
At the end of each holding period of τ = 1 or τ = 5 trading days in the investment period (from May 27, 2008to Sep 30, 2008, the covariance of the 30 stocks is estimated according to various estimators. They are the sample covariance of the last 100 trading days' daily return data (lowfrequency), the all-refresh TSCV estimator of the last 10 trading days, the all-refresh RK estimator of the last 10 trading days (the bandwidth of the realized kernel H is chosen to be 1 since the risk profile for H = 1 outperforms other alternative choices of H), and the pairwise-refresh TSCV   estimator of the last 10 trading days. These estimated covariance matrices are used to construct optimal portfolios with various exposure constraints. For τ = 5, we do not count the overnight risks of the portifolio. The reason that the overnight price jumps are often due to the arrival of news and are irrelevant of the topic of our studies. The standard deviations and other characteristics of these portfolio returns for τ = 1 are presented in Table 3 together with the characteristics of an equally weighted portfolio of the 30 DJIA stocks rebalanced daily. The standard deviations represent the actual risks. The risk is computed based on the 15 minutes returns. Figure 7 and Figure 8 provide the graphical details to these characteristics for both τ = 1 and τ = 5.  Figures 7 and 8 reveal that in terms of the portfolio's actual risk, the all-refresh TSRV and pairwise-refresh TSRV strategies perform at least as well as the low frequency based strategy when the gross exposure is small and outperform the latter significantly when the gross exposure is large. Both facts support our theoretical results and intuitions. Given 10 times the length of covariance estimation window, the low frequency approach still cannot perform better than the high frequency TSRV approaches, which affirms our belief that the high frequency TSRV approaches can significantly shorten the necessary covariance estimation window and capture better the shortterm time-varying covariation structure (or the "local" covariance). These results, together with the ones presented in the simulation section, lend strong support to the above statement.
Again the fact that the all-refresh RK strategy is outperformed by the low frequency strategy could be due to the instability of the estimated realized kernel covariance matrix.
As the gross exposure constraint increases, the portfolio risk of the low frequency approach increases drastically relative to the ones of the high frequency TSRV approaches. The reason could be a combination of the fact that the low frequency approach does not produce a well-conditioned estimated covariance due to the lack of data and the fact that the low frequency approach can only attain the long run covariation but cannot capture well the "local" covariance dynamics. The portfolio risk of the high frequency TSRV approaches increased only moderately as the gross exposure constraint increases. From financial practitioner's standpoint, that is also one of the comparative advantages of high frequency TSRV approaches, which means that investors do not need to be much concerned about the choice of the gross exposure constraint while using the high frequency TSRV approaches. It can be seen that both the low frequency and high frequency optimal no-short-sale portfolios are not diversified enough. Their risk profiles can be improved by relaxing the gross-exposure constraint to around c = 1.2, i.e. 10% short positions and 110% long positions are allowed. The no-short-sale portfolios under all approaches have the maximum portfolio weight of 22% to 50%. As the gross exposure constraint relaxes, the pairwise-refresh TSRV approach has its maximum weight reaching the smallest value around 30% to 34% while the low frequency approach goes down to only around 40%. That is another comparative advantage of the high frequency approach in practice as a portfolio with less weight concentration is always considered more preferable by most of the investors.
Another interesting fact is that the equally weighted daily-rebalanced portfolio of the 30 DJIA stocks carries an annualized return of only −10% while DJIA went down 13.5% during the same period (May 27, 2008to Sep 30, 2008, giving an annualized return of -38.3%. The cause of the difference is that we intentionally avoided holding portfolios overnight, hence not affected by the overnight price jumps. In the turbulent financial market of May to September 2008, that means our portfolio strategies are not affected by the numerous sizeable downward jumps. Those jumps are mainly caused by the news of distressed economy and corporations. The moves could deviate far from what the previously held covariation structure dictates.

Conclusion
We advocate the portfolio selection with gross-exposure constraint (Fan et al., 2008b). It is less sensitive to the error of covariance estimation and is immune to the noise accumulation. The outof-sample portfolio performance depends on the expected volatility in the holding period. It is at best approximated and the gross-exposure constraints help reducing the error accumulation in the approximations.
Two approaches are proposed for the use of high-frequency data to estimate the integrated covariance: "all-refresh" and "pairwise-refresh" methods. The latter retains far more data on average and hence estimates more precisely element by element. Yet, the pairwise-refresh estimates are typically not positive semi-definite and projections are needed for the convex optimization algorithms. The projection distorts somewhat the performance of the pairwise-refresh strategies.
The use of high frequency financial data increases significantly the available sample size for volatility estimation, and hence shortens the time window for estimation, adapts better to local covariations. Our theoretical observations are supported by the empirical studies and simulations, in which we demonstrate convincingly that the high-frequency based strategies outperform the low-frequency based one in general.
With the gross-exposure constraint, the impact of the size of the candidate pool for portfolio allocation is limited. We derive the concentration inequalities to demonstrate this theoretically. Simulation and empirical studies also lend further support to it.

A.1 Conditions
The following conditions are needed. For simplicity, we state the conditions for integrated covariation (Theorem 2). The conditions for integrated volatility (Theorem 1) are simply the ones with Y = X.
Condition 3. The observation times are independent with the X and Y processes. The synchronized observation times for the X and Y processes satisfy sup 1≤j≤ññ ·(v j − v j−1 ) ≤ C ∆ ≤ ∞, wherẽ n is the observation frequency and V = {v 0 , v 1 , · · · , vñ} is the set of refresh times of the processes X and Y .
Conditions 1 and 4 are imposed for simplicity. They can be removed at the expenses of lengthier proofs. For a short horizon and high-frequency, whether Condition 1 holds or not has little impact on the investment. For estimating integrated volatility, the synchronized time becomes observation time {τ n,j } and Condition 3 and 5 becomes sup 1≤j≤n n · (τ n,j − τ n,j−1 ) ≤ C ∆ < ∞ and 1 2 · n 1/3 ≤n K ≤ 2 · n 1/3 .

A.2 Lemmas
We need the following three lemmas for the proof of Theorems 1 and 2. In particular, Lemma 2 is exponential type of inequality for any dependent random variables that have a finite moment generation function. It is useful for many statistical learning problems. Lemma 3 is a concentration inequality for the realized volatility based on discretely observed latent process.
Lemma 2. For a set of random variables X i , i = 1, · · · , K, if when |θ| ≤ C 1 , for some two positive constants C 1 and C 2 , then Proof. By the Markov inequality, for 0 ≤ θ ≤ C 1 , we have Taking θ = x/(2C 2 ), we have For a small constant ξ > 0 to be specified later, let where a ξ = −ξx 2 0 and b ξ = 2ξx 0 . Then g ξ (x) is a continuously differentiable increasing convex function on [0, ∞). It follows from the Markov inequality and the convexity that, for w which is further bounded by 4g ξ (x) −1 if we can show that 4 is a common bound for {Eg ξ (w|X i |)}.
Therefore, continued from (30), we have This completes the proof of the lemma.
Lemma 3. (A Concentration Inequality for Realized Volatility Based on Latent Process) For a one dimensional process X t following (1) that satisfies Conditions 1-2, when one observes X t at times v i , i = 1, · · · , n, and the observation frequency satisfies Condition 3 (see (26)), then, for x ∈ [0, c √ n ], is the realized volatility based on the discretely observed X process; the constants c and C can be taken as in (34).
Proof. LetX t = X t − X 0 = t 0 σ s dW s . By time-change for martingales, (see, for example, Theorem 4.6 of Karatzas and Shreve (2000)), if τ t = inf{s : [X] s ≥ t} where [X] s is the quadratic variation process, then B t :=X τt is a Brownian-motion w.r.t. {F τt } 0≤t≤∞ . We then have that Note further that for any t, [X] t is a stopping time w.r.t. {F τs } 0≤s≤∞ , and the process exp θ(B 2 s − s) is a sub-martingale for any θ. By the optional sampling theorem, using [X] u ≤ C 2 σ u (bounded stopping time), we have Therefore, we have that, under Conditions 2 and 3, where Z ∼ N (0, 1) and It follows from the law of iterated expectations and (32) that where Z ∼ N (0, 1). Repeating the process above, we obtain By Lemma 1, we have for |θ| ≤ By Lemma 2, we have,

A.3 Proof of Theorem 1
We first prove the results conditional on the set of observation times V. Recall notation introduced in sections 3.2 and 3.3. Let n be the observation frequency. For simplicity of notation, without ambiguity, we will write τ n,i as τ i and σ (X) t as σ t . Denote the TSRV based on the unobserved latent process by where [X, X] Then, from the definition, we have, where 1 , and (X τ jK+l − X τ (j−1)K+l ) 2 , for l = 0, 1, · · · , K − 1.
Note that we have assumed thatn K = n−K+1 K is an integer above, to simplify the presentation.
Recall that we consider the case when J = 1, orn J = n. We are interested in The key idea is to compute the moment generating functions for each terms in (37) and then to use Lemma 2 to conclude.
For the first term in (37), since V (l) k is a realized volatility based on discretely observed X process, with observation frequency satisfying sup 1≤i≤n Kn K · (τ iK+l − τ (i−1)K+l ) ≤ C ∆ , we have, by (33) in Lemma 3, for |θ| ≤ For the second term in (37), we have obtained in (33) that The third term in (37) can be ignored because it has an upper bound that goes to zero sufficiently fast as n grows:n by Condition 5.
We introduce an auxiliary sequence a n that grows with n in a moderate rate to facilitate our presentation in the following. In particular, we can set a n = n 1/12 .
Let us now deal with R 1 , the fourth term in (37). Note that from the definition The first two terms in (41) are not the sum of independent variables. But they can be decomposed into the sum of independent random variables and the moment generating functions can be computed. To simplify the argument without losing the essential ingredient, let us focus on the first term of (41). It can be decomposed as and the summands in each terms of the right-hand side are now independent. Therefore, we need only to calculate the moment generating function of ε i ε i−1 .
For two independent normally distributed random variables X ∼ N (0, σ 2 X ) and Y ∼ N (0, σ 2 Y ), it can easily be computed that where we have used the fact that log(1 − x) ≥ −2x when 0 ≤ x ≤ 1 2 .
Hence, by the independence, it follows that (we assume n is even to simplify the presentation) The second term in R 1 works similarly and have the same bound. For example, whenn K is even, one can have the following decomposation The last four terms are sums of independent χ 2 -distributed random variables and their moment generating functions can easily be bounded by using Lemma 1. Taking the term For the term R 2 , we have, where where in the second line we have again used the optional sampling theorem and law of iterated expectations as in the derivations of Lemma 3; Z denotes a standard normal random variable. The second term in R 2 works similarly and has the same bound. For the third term, we have E exp a n θ √n where we have used the Hölder's inequality above. The forth term works similarly and has the same bound.
Combining the results for all the terms (38) -(45) together, applying Lemma 2 to (37), we have, for the following set of parameters, the conditions for Lemma 2 are satisfied with C 1 = C 1,x √n K . and coefficients in the first two terms of (37) + 4 √n K √ n K + 2 √n K a n √ K + 2 √n K a n √ n controls coefficients in (41) + 4a nnK n + 4 a n coefficients in (43) , where the > is valid when n is big enough and Condition 5 is applied.
This completes the proof of the result conditional on the observation times. Theorem 1 is proved by noting that this conditional result depend only on the observation frequency n and not on the locations of the observation times as long as the Condition 3 is satisfied.
Note also that in the above proof, we have demonstrated by using a sequence a n that goes to ∞ at a moderate rate that, one can eliminate the impact of the small order terms on the choices of the constants, as long as the terms have their moment generating functions satisfy inequalities of form (27). We will use this technique again in the next subsection.
with by the same way as before (again assumeñ is even for the simplicity of presentation below): .
The other terms are of a smaller order of magnitude. By applying an añ sequence which grows moderately withñ as in the proof of Theorem 1 (we can set añ =ñ 1/12 ), we can see easily that their exact bounds don't have effect on our choice of C 1 , C 2 or ω. All we need to show is that the moment generating functions of these terms can indeed be suitably bounded as (27). To show this, first note that, for any positive number a and real valued b, by the optional sampling theorem (applied to sub-martingales exp(aB 2 s ) and exp(b∆yB s ) with stopping time [X] u ≤ C 2 σ u for real number∆y), we have, where F i is the information collected up to time v i . Inequality (50) holds when∆X i is replaced bỹ ∆Y i . Similarly, The inequalities (50) and (51) can be used to obtain the bounds we need. For example, by (51) and the law of iterated expectations, E exp{θ odd i∆ X i∆ Y i−1 } ≤ 1 1 − θ 2 C 4 σ C 2 ∆ /ñ 2 ñ/4 ≤ exp θ 2 C 4 σ C 2 ∆ /2ñ when |θ| ≤ñ √ 2C 2 σ C ∆ ; by independence, normality of the noise, the law of iterated expectations and (50), we have .
Similar results can be found for the other terms above, with the same techniques.
The second term in √n K R 1 works similarly and have the same bound. The other terms in √n K R 1 and the whole term of √n K R 2 are of order o P (1). Again, by using a sequence añ we can conclude immediately that their exact bounds won't matter in our choice of the constants and we only need to show that their moment generating functions are appropriately bounded as (27). The arguments needed to prove the inequalities of form (27) for each elements in these terms are similar to those presented in the above proofs, and are omitted here.
This completes the proof.
Note that the argument is not restricted to TSCV based on the pairwise refresh times -it works the same (only withñ replaced byñ * , the observation frequency of the all-refresh method) for the case when the synchronization scheme is chosen to be the all-refresh method, as long as the sampling conditions Condition 3-4 are satisfied.


extfieldsofstudy: Physics Economics

text:
Cross-correlation in financial dynamics

To investigate the universal structure of interactions in financial dynamics, we analyze the cross-correlation matrix C of price returns of the Chinese stock market, in comparison with those of the American and Indian stock markets. As an important emerging market, the Chinese market exhibits much stronger correlations than the developed markets. In the Chinese market, the interactions between the stocks in a same business sector are weak, while extra interactions in unusual sectors are detected. Using a variation of the two-factor model, we simulate the interactions in financial markets.

In recent years, there has been a growing interest of physicists in economic systems. Concepts and methods in physics have been applied to the study of financial time series [1][2][3][4][5][6][7]. Different models and theoretical approaches have been developed to describe the features of the financial dynamics [8][9][10][11][12][13][14][15][16][17][18][19]. Statistical properties of price fluctuations and correlations between different stocks are topics of interest, not only scientifically for understanding the complex structure and dynamics of the economy, but also practically for the asset allocation and portfolio risk estimation [20][21][22]. The probability distributions of stock prices in different stock markets show a universal nature and follow the "inverse cubic law" [23][24][25]. However, the statistical properties of correlations between different stocks seem less universal across different stock markets [26].
Unlike most traditional physical systems, where one derives correlations between subunits from their interactions, the underlying "interactions" for the stock markets are not known. Pioneering studies at the phenomenological level analyze cross-correlations between stocks by applying concepts and methods of the random matrix theory (RMT), which was developed in the context of complex quantum systems where the precise nature of the interactions between subunits is not known [27,28]. The properties of the empirical correlation matrix C of price returns are compared with those of a random matrix in which the price movements are uncorrelated [29,30]. This spectral property-focused method was first applied to developed markets such as the New York Stock Exchange (NYSE) in USA [29][30][31][32], and recently also to some emerging markets, e.g. the National Stock Exchange (NSE) in India [26].
In general, the bulk of the eigenvalue spectrum of the correlation matrix C of price returns shares universal properties with the Gaussian orthogonal ensemble of random matrices, while the largest eigenvalue of C which deviates significantly from the bulk represents the influence of the entire market on all stocks. This is true for both developed and emerging stock markets. For developed markets, other large eigenvalues can be associated with the conventionally identified business sectors, according to the compositions of the eigenvectors. These sectors are stable in time, in some cases, as long as 30 years [31,33]. But for emerging markets such as the NSE in Indian, the number of large eigenvalues are smaller than that of developed markets [26]. Although there exist often much more correlations between stocks in emerging markets in all, the less large eigenvalues may reflect less correlations between stocks in a same sector. It seems that strong interactions may emerge within groups of stocks as a financial market evolves over time.
Although there have been many studies of correlated price movements in stock markets, the researches on emerging markets are few and mostly on the synchronicity of price movements across stocks [34][35][36][37]. The Chinese stock market is an important emerging market. Although sharing certain common features with developed markets, it also exhibits unusual dynamic properties [7,38,39]. Especially, the anti-leverage effect in the Chinese market is unique according to up-to-date literatures [7,39]. In some sense, the Chinese market is a representative emerging market, and the Indian market is gradually approaching a developed one.
In this paper, we continue our comparative study of the Chinese market and developed markets as well as even the Indian market, with the RMT theory and using the data of the Shanghai Stock Exchange (SSE) of China. The Shanghai Stock Exchange was established in 1990, and it is now one of the most important emerging stock markets in the world. We have analyzed the daily closing prices of 259 stocks traded in the SSE from Jan., 1997 to Nov., 2007, and it corresponds to 2633 days. The data are obtained from "Wind Financial Database" (http://www.wind.com.cn). If the price of a stock is missing on a particular day, it is then assumed that the price remains the same as the preceding day [37]. In Ref. [26], it has been testified that the missing prices do not result in artifacts for the American market.
Let P i (t) be the price of a stock i = 1, . . . , N at time t, we then define the logarithmic price return of the ith stock over a time interval ∆t as Since different stocks may fluctuate at different levels, e.g., measured by the standard deviation of its return, we should make sure that the results are independent of the scale of the measurement. Therefore, we introduce the normalized return where σ i ≡ < R 2 i > − < R i > 2 is the standard deviation of R i , and < . . . > represents the time average. Finally, we compute the equal-time cross-correlation matrix C, whose element measures the correlation between the returns of stocks i and j. By definition, C is a real symmetric matrix with C ii = 1, and C ij is valued in the domain [−1, 1]. In Fig. 1 (a), the probability distribution P (C ij ) of the elements of the cross-correlation matrix C is displayed for the SSE (China), NSE (India) and NYSE (USA). The data for the NSE and NYSE are from the period 1996-2006 [26]. The mean value C ij of the elements for the SSE is 0.37, much larger than 0.22 and 0.20 for the NSE and NYSE respectively [26]. This result strongly supports the general belief that stock prices in emerging markets tend to be more correlated than developed ones [34]. In this sense, however, the Chinese market is much more 'emerging' than the Indian market. In addition, all elements of the matrix C of the SSE are positive, while there are a few negative ones for the NSE and NYSE. This also suggests that the correlation between stocks is stronger in the Chinese stock market.
If N time series of length T are mutually uncorrelated, the resulting cross-correlation matrix is called a Wishart matrix. Statistical properties of such random matrices are known [40,41]. In the limit N → ∞ and T → ∞ with Q ≡ T /N ≥ 1, the probability distribution P rm (λ) of the eigenvalue λ is given by for λ min,ran ≤ λ ≤ λ max,ran and 0 otherwise. The lower (upper) bound is given by For the data of the SSE in the period 1997-2007, for example, there are N = 259 stocks each containing T = 2632 returns. If there are no correlations between the stocks, the eigenvalues should be bounded between λ min,ran = 0.47 and λ max,ran = 1.73.
In developed stock markets such as those of USA and Japan, the bulk of the eigenvalue spectrum P (λ) of the cross-correlation matrix is similar to P rm (λ) of the Wishart matrix, but some large eigenvalues deviate significantly from the upper bound λ max,ran [29][30][31][32]. This is also true for emerging stock markets, but the number of the large eigenvalues is relatively few as it is reported for the NSE (India) [26]. The latter should be a characteristic of emerging markets. In Fig. 1 (b), the eigenvalue spectrum of the SSE (China) is displayed with the solid line. The bulk is similar to P rm (λ) of the Wishart matrix, and there are a few large eigenvalues. The number of the large eigenvalues is less than that of the NYSE (USA). To verify that these outliers are not an artifact of the finite length of the observation period, we shuffle the empirical time series of returns (i.e., randomly change the time sequence of returns), thereby destroying all the equal-time correlations, then compute a surrogate correlation matrix. The eigenvalue spectrum of this surrogate matrix matches exactly with P rm (λ). This is shown with the dotted line in Fig. 1 (b). Therefore, one confirms that the large eigenvalues are the genuine effect of equal-time correlations between stocks.
The presence of a well-defined bulk of the eigenvalue spectrum which agrees with P rm (λ) suggests that the contents of C are mostly random except for the large eigenvalues that deviate. As shown in Table I and Fig. 1 (b), the largest eigenvalue λ 0 of the cross-correlation matrix of the SSE (China) is 97.33, about 56 times as large as the upper bound λ max,ran of P rm (λ). However, λ 0 of the NSE (Indian) and NYSE (USA) is 46.67 and 42.50, only about 26 and 28 times as large as λ max,ran respectively. The correlation corresponding to the largest eigenvalue λ 0 is believed to be generated by interactions common for stocks in the entire market. Our results imply that this correlation of the market mode in the Chinese stock market is much stronger than that in other stock markets, including the NSE (Indian) which is usually considered to be an emerging market. Now let us look at the components u i (λ) of the eigenvectors of the first four largest eigenvalues. In the top sector of Fig. 2 (a), the eigenvector of the largest eigenvalue λ 0 shows a relatively uniform composition with all stocks participating in it. Actually, all the elements also have the same sign. It represents a common mechanism that affects all the stocks with the same bias. So the largest eigenvalue is associated with the market mode, i.e., the collective response of the entire market to the external information. The Chinese market shares this feature with other stock markets [26,29,31].
According to previous studies [31,33], the components of the eigenvectors of other large eigenvalues are localized, i.e., each of these eigenvectors is dominated by only a part of stocks. The dominating components of an eigenvector belong to similar or related business sectors. This is true for both developed markets and some emerging markets. However, the number of the large eigenvalues of the NYSE (USA) is obviously more than those of the NSE (India) [26,31,33]. That of the SSE (China) is even less. Especially, the dynamic effect of business sectors is hardly seen, as shown in the second, third and fourth sectors of Fig. 2 (a).
What interactions do control the large eigenvalues of the SSE? To answer it, we introduce a threshold u c to select the dominating components of the eigenvectors with large eigenvalues. In other words, we consider that only those components which satisfy |u i (λ)| ≥ u c contribute to the eigenvector of the eigenvalue λ. The results of u c = 0.08 is shown in Fig. 2 (b).
In the SSE (China), a company will be specially treated when its financial situation is abnormal. In this case, the word "ST" will then be added to the stock name as a prefix. The abnormal financial situation includes: the audited profits are negative in two successive accounting years, the audited net worth per share is less than its stock's par value in the recent accounting year etc. When the company has a negative audited profit for three successive accounting years, the word "*ST" will be added to its stock name and the stock may be also delisted. When the abnormal financial conditions are eliminated, the word "ST" or "*ST" will be removed.
For the SSE (China), we observe that the eigenvector of the second largest eigenvalue λ 1 is dominated by the stocks now with or used to be with "ST" or "*ST". When we increase the threshold u c , the proportion of the dominating stocks with "ST" and "*ST" rises. When u c ≥ 0.12, all dominating components are "ST" or "*ST" stocks. Detailed results are given in Table II. Therefore, the second largest eigenvalue corresponds to the so-called ST sector.
The dominating components of the eigenvector of the third largest eigenvalue λ 2 mainly are or used to be "Blue-chip" stocks, which are referred to be with stable and good performance, i.e., a reasonable positive profit in a period of time. When we increase the threshold u c , the proportion of the Blue-chip stocks steadily rises, as shown in Table II. With a similar approach, we examine the dominating components of the eigenvector of the forth largest eigenvalue λ 3 , and observe that they are mainly those companies registered in Shanghai with the real estate business. When we continue to investigate the next largest eigenvalues such as λ 4 and λ 5 of the SSE (China), we could not find clear characteristics. For the NYSE (USA), however, one could detect the dominating effect of the business sectors at least up to the 10th largest eigenvalue [31,33].
Why does the SSE (China) exhibit the usual sectors like the ST and Blue-chip sectors rather than the standard business sectors? Since the Chinese stock market is an emerging market, the companies are not operated strictly with the registered business. On the other hand, people seriously look at the performance of the companies. The fourth largest eigenvalue of the SSE may reflect the fact that Shanghai and especially its real estate business play important roles in China in the past years both economically and politically.
To better understand the structure of interactions in the SSE (China), we introduce a variation of the two-factor model of the market dynamics [26]. In our model, we assume that the normalized return at time t of the ith stock from the kth business sector can be decomposed into (i) a market factor r m (t), that contains information common to all stocks; (ii) a business sector factor r k g (t), representing dynamic effects exclusive to stocks in the kth business sector; (iii) a profit factor r p (t), reflecting three categories of stocks: the "ST" sector, "Blue-chip" sector and "general" sector-i.e., the rest stocks except the ST and Blue-Chip stocks; and (iv) an idiosyncratic term η i (t), which corresponds to random variations unique for that stock. We thus obtain where the constants β i , γ k i , γ p i and σ i represent the relative strengths of the four factors respectively. Although the empirically distributions of returns show power-law tails and are not Levy stable, we assume that the normalized returns r k i follow a Gaussian distribution with a zero mean and a unit variance for simplicity. In other words, we choose r m (t), r k g (t), r p (t) and η i (t) to be Gaussian processes with a zero mean and a unit variance. The unit variance ensures that the relative strengths of the four factors satisfy the relation For each stock, we can independently assign γ k i , σ i and γ p i , and obtain β i from Eq. (7). But we should assure that all γ p i for the ST sector are greater than any γ p i for the Blue-chip sector. This condition may lead to the eigenvalue spectrum of the SSE (China), whose second and third largest eigenvalues represents the ST and Blue-chip sectors respectively. We practically choose γ k i , σ i and γ p i from a uniform distribution with a width δ and centered about the mean value γ, σ and γ p respectively.
With the above model, we generate N times series of length T for returns r k i , and simulate an artificial market with N stocks and K business sectors. These K business sectors are composed of n 1 , n 2 , . . ., n k stocks such that n 1 + n 2 + . . . + n k = N . Next we randomly select n ST stocks as "ST" stocks and n BC stocks as "Blue-Chip" stocks from the N stocks such that n ST + n BC + n G = N , where n G represents the number of the rest stocks after selecting ST and Blue-chip stocks. Finally, we construct the resultant cross-correlation matrix C and compute its eigenvalues and eigenvectors.
In order to describe the empirical properties of the SSE (China), we choose a large β, moderate γ p and small γ. This leads to a market with a strong market mode, moderate interactions between stocks in the ST or Blue-chip sectors, and a small effect of the business sectors. For example, we take N = 250 and T = 2500, and assign 5 sectors A, B, C, D and E, each with 50 stocks. We choose γ k i = 0.2, γ p i = 0.55 for the ST sector and γ p i = 0.40 for the Blue-Chip sector, and σ = 0.3, δ = 0.05, n ST = 40 and n BC =40. With these fixed parameters, one calculates β to be 0.75 and 0.84 for the ST and Blue-chip sectors respectively. In Fig. 3, the components of the eigenvectors of the first eight largest eigenvectors are displayed. The largest eigenvalue represents the market mode, and the components are distributed with uniform weights. For the second and third eigenvalues, we observe large components from different business sectors. When we introduce a threshold u c to select those dominating components which satisfy |u i | ≥ u c , we find that they are all ST stocks for the second largest eigenvalue and Blue-chip stocks for the third largest eigenvalue.
For the next largest eigenvalues, we detect the dynamic effect of the standard business sectors. But several business sectors may contribute to a single eigenvector. This behavior is also similar to that of the Chinese stock market. The eighth largest eigenvalue in Fig. 3 is the upper bound λ max,ran of the Wishart matrix which is derived from uncorrelated time series. In our numerical simulations, the cross-correlation matrix is essentially modified by the extra interactions between the stocks in the unusual sectors such as the ST and Blue-chip sectors.
In summary, we phenomenologically compute the correlations of stock price returns, and construct the cross-correlation matrix C. The average value C ij of the matrix elements is 0.37 for the SSE (China), much larger than 0.22 and 0.20 for the NSE (Indian) and NYSE (USA) respectively. It supports that the correlations in emerging markets are stronger than those in developed markets. In this sense, the NSE is obviously less emerging than the SSE.
We then diagonalize the matrix C and obtain the eigenvalue spectrum and eigenvectors. The large eigenvalues which go beyond the upper bound λ max,ran of the Wishart matrix reflect the correlations between stocks. The largest eigenvalue λ 0 corresponds to the market mode, which is induced by interactions common for all stocks in the stock market. The largest eigenvalue λ 0 of the SSE (China) is 97.33, much large than 46.67 of the NSE (Indian) and 42.50 of the NYSE (USA) respectively. This result is in agreement with that of the average value C ij . Other large eigenvalues represent the interactions between stocks in a same sector. For the NYSE, the correlations corresponding to standard business sectors are strong, and there are at least 10 such large eigenvalues. For emerging markets such as the SSE and NSE, the correlations between stocks in a same sector are weak, and there exist only a few large eigenvalues. Especially, the correlations within a standard business sector is almost invisible. For the SSE (China), we identify the large eigenvalues according to unusual sectors such as the ST and Blue-chip sectors.
Finally we apply a variation of the two-factor dynamic model to explain the collective behavior of the emerging stock markets. In our model, extra interactions within an unusual sector are introduced, and the correlations corresponding to the standard business sectors are then suppressed.