content_image/393184.jpg content_image/393186.jpg content_image/393185.jpg content_image/393187.jpg
12 × 12 lattice. The search is done in the mean field space with the local density n$_{i}$ and the three compo- nents of local spin density ⃗ S$_{i}$ as variational parameters. We have used both the conjugate gradient method and the simulated annealing method to perform the search and used more than 100 random initial configurations for the local density and local spin density. We find the search always converge to the two configurations shown in Fig.1(up to global spin rotations and point group op- erations) for large enough value of U t . Thus, although the quarter-filled system with positive hopping integral has no nesting property, it has the same magnetic ordering pattern in the strong correlation regime as the negative hopping system. 

The mean field equations for the collinear state is given by 

<img src='content_image/393184.jpg'>

in which the sum is limited in the magnetic Brillouin zone and ξ$_{k}$ = − 2 t (cos k$_{x}$ + cos k$_{y}$ + cos(k$_{x}$ + k$_{y}$)) − µ is the dispersion on the triangular lattice. Here we have used the convention for the momentum that k = k$_{x}$ ⃗ b$_{1}$ + k$_{x}$ ⃗ b$_{2}$ . In this convention, the first Brillouin zone of the trian- gular lattice is given by (k$_{x}$ , k$_{y}$) ∈ [ − π, π ] ⊗ [ − π, π ] and the magnetic Brillouin zone of the collinear state is given by [0 , π ] ⊗ [ − π, π ]. $_{E}$± k = ξ$_{k}$ + ξ$_{k}$$_{+}$$_{Q}$$_{3}$ $_{±}$√ ( ξ$_{k}$ − ξ$_{k}$$_{+}$$_{Q}$$_{3}$ ) $^{2}$+( Um ) 2 2 is quasiparticle energy in the magnetic ordered state. 

In the non-coplanar insulating state, the magnetic Brillouin zone reduces further into the region given by [0 , π ] ⊗ [0 , π ]. The mean field equation for the non- coplanar state is given by 

in which W n k = 1 $_{2}$√$_{3}$ ∑ α,β φ n ∗ α ( k ) M $_{α,β}$φ n $_{β}$( k ) , α, β, n = 1 · · · 8 and the sum is limited in the magnetic Brillouin zone. φ n $_{α}$( k ) is the α − th component of the eigenvector for the mean field Hamiltonian with eigenvalue E n $_{k}$. The mean field Hamiltonian matrix is of the following form 

here I is a 2 × 2 identity matrix, M$_{1}$ = − Um 3 ( 1 √$_{2}$ √$_{2}$ − 1 ) , M$_{2}$ = − Um 3 ( 1 − 1+ $_{i}$√$_{3}$ √$_{2}$ − $^{1}$− $_{i}$√$_{3}$ √$_{2}$ − 1 ) . The matrix M is 

<img src='content_image/393186.jpg'>

<img src='content_image/393185.jpg'>

given by 

Fig.2 shows the order parameter m as a function of U/ | t | at zero temperature for both signs of the hopping integral. For negative t , the magnetic order exist for all non-zero U as a result of the nesting property of the system. While for positive t , the magnetic order exists only for U/ | t | > U$_{c}$/ | t | = 7 . 2. For both sign of t , the non- coplanar state is slightly more stable than the collinear state at zero temperature. 

Fig.3 shows the mean field phase diagram at finite tem- perature for both signs of the hopping integral. At fi- nite temperature, the gapless Fermion excitation in the collinear state will contribute more entropy than the gapped Fermion excitation in the non-coplanar insulating state. We find for sufficiently large U/ | t | , the collinear state will become more stable than the non-coplanar state above a critical temperature at which a first order transition between the two occurs. In the phase diagram, the first order transition line between the collinear state and the non-coplanar state end at a critical end point on the magnetic-paramagnetic transition line. For positive t , the critical end point locates at U/ | t | = 7 . 9, while for negative t , it locates at a smaller value of U/ | t | = 5 . 5. Thus for both sign of the hopping integral, the collinear state is stable only in the strong correlation regime. 

It is interesting to note that the transition between the collinear state and non-coplanar state, though is of first order in nature, has almost no observable effect on the peak position and intensity of the magnetic Bragg peak. As we have explained above, the non-coplanar state has the same sets of magnetic Bragg peak as the collinear state, if the three domains of the latter as mixed with equal probability. In Fig.4, we plot the change of the magnitude of the order parameter at the collinear - non- coplanar phase transition as a function of U/ | t | . We find the change to be always positive but its absolute value is always less than 1 . 5% of the ordered moment for all value of U/ | t | and for both signs of t . Thus we conclude that the neutron scattering experiments are not sensitive to the collinear-non-coplanar transition. 

However, as the system opens a full gap abruptly on the quasiparticle spectrum at the collinear-non-coplanar transition, the longitudinal resistivity is expected to in- crease dramatically. Below the transition point, the sys- tem will enter the true insulating state. Thus in this sys- tem, the metal-insulator transition occurs in a two-step 

<img src='content_image/393187.jpg'>

