Question: If $N$ and $N'$ are two closed hyperbolic 3-manifolds, then one would like to have an algorithm which determines whether or not $N$ and $N'$ are homeomorphic.

If $N$ and $N'$ are Haken, then such an algorithm is provided by Haken. The homeomorphism problem is solved in general by Sela's work on hyperbolic groups and Mostow rigidity. 

I was wondering though whether in light of Agol's work one can now give a "purely $3$-dimensional" solution to the homeomorphism problem. More precisely, Agol showed that given any hyperbolic 3-manifold there exists a finite cover with $b\_1>0$. So given $N$ and $N'$ I can find a finite cover $\tilde{N}$ with $b\_1(\tilde{N})>0$. I can take the subgroup $\pi\_1(\tilde{N})$ to be of the type "intersection of all kernels of homomorphisms to a fixed finite group". I can then use the `same' finite index subgroup to get a cover $\tilde{N}'$ with $b\_1(\tilde{N}')>0$. So now one can apply Haken to check whether $\tilde{N}$ and $\tilde{N}'$ are homeomorphic. To push this result down to $N$ and $N'$ it seems to me that one needs to calculate the mapping class group of $\tilde{N}$. This group is finite by Mostow rigidity, but are there algorithms for determining it? Are there any bounds on the size of the group?

Answer: 
There is a bound on the size of the mapping class group of a hyperbolic manifold, namely the volume divided by the minimal volume of a hyperbolic 3-orbifold ([0.03905...](http://annals.math.princeton.edu/2012/176-1/p04)). 

There is a (non-rigorous) algorithm to compute the isometry group by [Hodgson-Weeks](http://projecteuclid.org/euclid.em/1048515809), which is used in the program Snappea. 

**Addendum:** Another possible approach to you question would be to compute the Thurston norm of the cover (which in principle is possible using normal surface algorithms developed by Tollefson). Then one has a short exact sequence of the mapping class group action on the Thurston norm unit ball, and the kernel of this action. If there are fibered faces, then in principle one can compute both by solutions to the conjugacy problem in the mapping class group, which would avoid some of the difficulties involved with hierarchies and Waldhausen/Hemion solution to the homeomorphism problem. I should say though that this entire approach is likely to be computationally infeasible, especially compared to a program like snappea which works well in practice, or using Sela's algorithm as referred to in Henry's answer. So although understanding isometry groups of manifolds is an interesting question, I think your whole approach to the homeomorphism problem is quixotic, except possibly as a theoretical exercise.
The trouble is that computing the outer automorphism group is a very similar to the isomorphism problem, but actually a little harder. The only algorithm that I know of to compute the outer automorphism group of a hyperbolic-manifold group uses Sela's techniques.

The algorithm follows quickly from Theorem G of Dahmani--Groves' paper ('The isomorphism problem for toral relatively hyperbolic groups'. *Publ. Math. Inst. Hautes ́Etudes Sci.* No. 107 (2008), 211–290) which, combined with the immediately preceding remark, implies the following:

> 
> There is an algorithm with the following properties. It takes as input a finite presentation for a toral relatively hyperbolic group G [eg the fundamental groups of a hyperbolic 3-manifold]. It terminates if and only if Out(G) is finite. In case it terminates, it provides a finite list of homomorphisms which contains a representative of every conjugacy class of monomorphism from G to itself.
> 
> 
> 

In particular, the output list contains a representative of each outer automorphism class. (I don't think the list is *a priori* guaranteed to only contain monomorphisms.) Using the membership problem for G, one can check which maps on the list are surjective, hence automorphisms (since G is Hopfian). Finally, using the simultaneous conjugacy problem in G, one can check which maps on the list are conjugate, and reduce the list to exactly one representative for each conjugacy class.

It's conceivable that a conceptually easier algorithm (avoiding Sela's techniques) can be produced along the lines of Scott--Short's recent alternative solution to the homeomorphism problem (arXiv:1211.0264), but I haven't thought about this.