In which base $b$ is $(374)_b$ a perfect square? If you convert this number to base 10, we can obtain the expression $$3b^2+7b+4 = (b+1)(3b+4).$$
Since $\gcd(b+1,3b+4) = 1$, we further conclude that both $b+1$ and $3b+4$ are perfect squares.
So the problem is equivalent to finding $b$ that satisfies the Diophantine equation $$3(b+1)^2+1 = L^2,$$ where $\gcd(b+1,L) = 1$.
I'm not sure how to find all solutions to the above equation.

You're off to a good start; if $b+1$ and $3b+4$ are both perfect squares, then
$$3b+4=x^2\qquad\text{ and }\qquad b+1=y^2,$$
for some integers $x$ and $y$, and hence
$$x^2-3y^2=1.$$
This is a Pell equation, and its solutions are well known. I suggest to start from the Wikipedia page to understand how to find all integral solutions. In particular there are infinitely many solutions.
One characterization of the integral solutions is that they are precisely the pairs of integers $(x,y)$ for which
$$x+y\sqrt{3}=\pm(2+\sqrt{3})^k,$$
for some integer $k$. Of course the choice of $\pm$ sign only changes the signs of $x$ and $y$, and the same is true if we replace $k$ by $-k$. So to find all solutions $b$ it suffices to consider $(2+\sqrt{3})^k$ with $k\geq0$. The first few solutions are:
$$\begin{array}{r|rr|rr}
k&x&y&b&(374)_b\\
\hline
0&1&0&\color{red}{-1}&\color{red}{0^2}\\
1&2&1&\color{red}{0}&\color{red}{2^2}\\
2&7&4&15&28^2\\
3&26&15&224&390^2\\
4&97&56&3135&5432^2\\
5&362&209&43680&75658^2
\end{array}$$
