Reading between the lines, it may be that the child's biological father and y'all's husband's own offspring-situation are part of the equation.
Y'all's father's children are the only part of the problem.