Number of ways to pick $3$ balls from a box with $8$ balls if $3$ of the balls are identical and the other $5$ are all different 
A box contains 8 balls, of which 3 are identical and the remaining 5 are different from each other. 3 balls are to be picked out of the box; the order in which they are picked out does not matter. Find the number of different possible selections of 3 balls.

My Solution: I grouped them into cases
Case 1: All three balls are identical in which number of ways $= 1$
Case 2: Two of the balls are identical in which number of ways $= 5$
Case 3: One of the ball is from the $3$ identical ones in which number of ways $= 5 \cdot 4$ since there are $5$ ways to choose $1$ ball from the remaining $5$ balls and another $4$ ways to choose again from the remaining $4$ balls.
Case 4: I do not take from any of the $3$ identical balls so that leaves me with $5$ balls to choose $3$ from in which number of ways $= 5 \cdot 4 \cdot 3 = 60$.
Total number of ways $= 1+5+20+60 = 86$.
However, the answer given is $26$.

When the order of selection does not matter, you should be thinking in terms of combinations.
In what follows, suppose that the three identical balls are white and that the remaining balls are blue, green, red, orange, and yellow.
Since we wish to determine the number of distinguishable selections of three balls, your first two cases are correct.
Exactly one of the balls is one of the three identical ones:  We do not care about which of the identical balls we select.  We can select two of the other five balls in $\binom{5}{2}$ ways.  Hence, the number of such selections is
$$\binom{5}{2} = \frac{5!}{2!3!} = \frac{5 \cdot 4 \cdot 3!}{2 \cdot 1 \cdot 3!} = 5 \cdot 2 = 10$$
As lulu pointed out in the comments, this is half your answer since first selecting a blue ball and then selecting a green ball results in the same set of balls as first selecting a green ball and then selecting a green ball.
None of the balls is selected from among the three identical ones:  There are
$$\binom{5}{3} = \frac{5!}{3!2!} = \frac{5 \cdot 4 \cdot 3!}{3! \cdot 2 \cdot 1} = \frac{5 \cdot 4}{2} = 5 \cdot 2 = 10$$
such selections.  You counted each such selection six times, once for each of the $3!$ orders in which you could have picked the same three balls.
With these corrections, we obtain
$$1 + 5 + 10 + 10 = 26$$
distinguishable ways to select three of the eight balls in the box.
