It's not like that as you are thinking It's correct and it will be
$a = \frac{20}{3} \frac{m}{s^2}$ only. Let's find out how.
First I'll go with the second diagram
For this system the equation for both the masses will be as:-
For the block of mass $m_1$ (which is in this case 4kg)
$m_1 g-T = m_1 a_1$ $----eq 1$
{took the positive direction downwards}
And For the block of mass $m_2$ (which is in this case 2kg)
$T=m_2 a_2$ $-----eq 2$
{Took the RHS direction positive}
Hence we can say that this is the constraint motion so for this, the total work done by the tension in the system will be zero.
Workdone by the tension on 4 Kg block that's $m_1$ will be
$W_1 = \vec{T} . \vec{x_1}$
So the displacement and the tension are exactly in the same direction so the angle between the would be 180 degrees
And workdone will come out to be
$W_1 = T x_1$
Similarly for $m_2$ block with mass 2Kg the workdone by the tension will be:-
$W_1 = \vec{T} . 512
\vec{x_2}$
$W_1 = -T x_2$
So by the conclusion that the total workdone by the tension on the whole system will be zero then:-
$T x_1 - T x_2 = 0$
Cancelling out the tension on both sides we get
$x_1 = x_2$
So on differentiating both sides two times w.r.t time we will get the acceleration.
$a_1 = a_2$
So this means both the blocks accelerate with the same rate.
So by putting $a_1 = a_2 = a$
By solving all of the equations above we get
$m_1 g - T=m_1 a$
And for m_2
$T = m_2 a$
But in this case mass of $m_1$ is 4 Kg and the block with mass $m_2$ which is 2 Kg so by substituting the masses and acceleration due to gravity in $eq 1$ and $eq 2$ we get
$40-2a = 4a$
$40 = 6a$
$a=20/3 \frac{m}{s^2}$
Now as we know that according to the second law of motion the net force on the object will be equal to it's mass times acceleration. So from this equation the net force on block with mass $m_1$ (in this case it's 4kg) will be $m_1 g-T$. T will be negative just because it's acting in the opposite direction and we have fixed the downward direction as positive and anything which will happen in its opposite direction will be considered as negative.
And on the block with mass $m_2$ (in this case it's 2kg) only one force is acting that's tension so the net force on that block will be only T.
So by just substituting we will get the answer.
And man don't go with any shortcut formulae they are just made to make our life a bit easier (and this is one of it) they will not be valid everywhere just go through the with the concepts and pull out as much as you can.
This system has only three pulling forces they are the tension on both the blocks and the gravitational force acting on block with mass $m_1$ so according to your formula:-
$A = \frac{T-T+40}{6} m/s^2$
$A = \frac{20}{3} m/s^2$
It again comes out to be true
This formula for which you are struggling to find how it comes is like this:-
We assume the single block instead of assuming two blocks with the mass equal to the sum of masses of both the blocks and assuming that the system has only one block and calculate the net force on it, the tension here vanishes out because there are no threads connected anywhere because the system has only one block now.
And the formula comes out to be
$Fnet = ma$
Where here $m$ is the mass of that single block which has mass equal to sum of masses participating in the system.
That is $m = m_1+m_2+..... 512