When using <PERSON> solvers to solve $\mathbf A \mathbf x = \mathbf b$ systems, singularity in $\mathbf A$ can be pretty benign as long as $\mathbf b$ is consistent. Consider applying the conjugate gradient (CG) algorithm to a symmetric positive semidefinite $\mathbf A$ (singular, but all non-zero eigenvalues still positive). Further assume $\mathbf b$ is in the range of $\mathbf A$, and we pick the initial guess to be $\mathbf x = \mathbf 0$. Then, the search vector/residual $\mathbf r = \mathbf b - \mathbf A \mathbf x$ will always be orthogonal to the nullspace of $\mathbf A$ (because in CG, search vectors/residuals are always drawn by applying $\mathbf A$ to previous residuals, and this multiplication by $\mathbf A$ can't generate any component in the nullspace). 768
The algorithm can never "sense" the singularity of $\mathbf A$, and will converge to the (unique!) $\mathbf x$ such that $\mathbf A \mathbf x = \mathbf b$ and $\mathbf x \perp \mathrm {null}(\mathbf A)$
A short demo in matlab:
clear all
close all
% Form positive semidefinite A.
N = 10;
[Q,~] = qr(rand(N));
D1 = linspace(1,N/2,N/2);
D2 = zeros(1,N/2);
D = diag([D1 D2]);
A = Q*D*Q';
kappa = cond(A) % infinite/singular
% Form random b in range(A).
b = A * rand(N,1);
% Solve A*x=b using CG.
x = pcg(A,b);
% Check properties.
residual = norm(A*x-b)
nullity = norm(x'*null(A))
Similar arguments can be made for other <PERSON> solvers, but I can't really speak to "classical" iterative solvers (Gauss-Seidel etc) .. I doubt they are as well-behaved.
There are preconditioning strategies (deflation) based entirely on this idea: find a basis for the eigenvectors of $\mathbf A$ that are somehow problematic to the solver, then "deflate"/project them out (change the operator such that the corresponding eigenvalues are mapped to zero), then apply the same projection to $\mathbf b$, and iterate away. For more information about deflation preconditioning (which involves convergence analysis for singular systems), I recommend:
<PERSON>, <PERSON>, et al. "A comparison of two-level preconditioners based on multigrid and deflation."
<PERSON>, <PERSON>, and <PERSON>. "Deflation and balancing preconditioners for <PERSON> subspace methods applied to nonsymmetric matrices." 768