Q:

How does the speed of a DC motor depend on the size of the coil?

I have to build a simple electric motor by attaching a magnet to a battery, extending the terminals of the battery (with stiff wires so they could act as supports), and placing a coil of wire on top of it. I know that the more loops you have in the coil the stronger the magnetic field and therefore the faster the coil will rotate. However, how does the diameter of the magnetic coil affect speed of the wire coil rotating? A smaller diameter might make the magnetic field stronger, but the magnetic force which produces torque would also be be applied closer to the center of rotation, therefore creating less torque.

A:

The basic electric motor you described (see here and here for more complete descriptions than that provided in the question) is interesting and quite subtle to treat quantitatively, and I have not been able to find a good explanation of it on the Internet.  I will provide one below.  Even if the OP has moved beyond this problem, I hope others might find my answer helpful.
The experiment
The setup is a coil of wire, with its ends half-stripped, suspended above a permanent magnet and connected to a battery.  As soon as the coil is given a small push, it starts rotating rapidly, and continues to move for a long time (easily minutes).  The motor only rotates in one direction, even when restarted multiple times from rest, either clockwise or counterclockwise depending on how the coil was stripped.  The links above provide pictures and step-by-step descriptions of how to build the device.  But how does it work, exactly?
Qualitative explanation
The coil of wire acts as an electromagnet.  Magnets like to align themselves in parallel, so when the electromagnet is on, there is a torque on it due to the permanent magnet.  The torque is clockwise or counterclockwise, depending on which way the electromagnet (coil) has to rotate to align itself with the permanent magnet.  Say you completely stripped the insulation off of the coil wire where it touches the battery connectors.  As you rotate the coil though one revolution, the torque is clockwise half the time and counterclockwise the other half.  Here's a picture illustrating this for a clockwise rotation of the coil:

With the wire completely stripped at the ends, the coil behaves as a pendulum.  But now comes the twist: to make a motor, you strip the coating off of half of each wire leading to the coil.  The electromagnet will only be on for half the cycle.  If you are lucky, this will be exactly the half when the torque is clockwise, the coil will be given an extra push with each revolution, and it will keep rotating.  Note the direction of the curly arrows indicating torque in the picture below:

But when you think about it, you don't need to be this lucky.  If you strip the coating in such a fashion that the torque is, say, clockwise for 2/3 of the cycle and counterclockwise for 1/3 of the cycle, you will still on net deliver clockwise torque to the coil, producing clockwise rotation.  In the picture below, you can see there are more clockwise torque arrows than counterclockwise ones:

In fact, you need to be very unlucky to strip half the insulation from the wire in such a way that the clockwise and counterclockwise torques remain exactly balanced, as they are for the fully stripped wire.  The net torque (averaged over one rotation) will be either clockwise or counterclockwise, and the motor will rotate in that direction.
Quantitative model
The qualitative discussion above gives an intuition for how this motor works, but doesn't give an answer to questions such as that posted by OP (i.e., How is the speed of the motor affected by the size of the coil?).  In fact, it is not even obvious that the motor will settle down to a more-or-less steady speed: after all, it is given a "push" for only half the rotation period, and this push depends on the angle the coil makes with the magnet.  To gain further insight, it helps to examine a theoretical model.
Let's assume the coil is sufficiently small, and sufficiently close to the magnet, that the magnetic field is constant.  Then, the torque on the coil is,
$$\vec{N} = \vec{m} \times \vec{B},$$
where $\vec{B}$ is the constant magnetic field due to the permanent magnet and $\vec{m}$ is the magnetic moment of the coil, equal to the product of the coil's area, $\vec{A}$, its number of loops, $n$, and the current flowing through it, $J$:
$$\vec{m} = Jn\vec{A}.$$
The only nonzero component of the torque is,
$$N = -JnAB \sin \theta,$$
where $\theta$ is the angle between the magnetic moment of the loop and the vertical.  Because the wire connecting the coil to the battery terminals is only half-stripped, the current $J$ depends on $\theta$, too.  We'll assume the stripping is optimal, so that
$$J(\theta) = \begin{cases}
J_0 & \text{for $\theta \in (0, \pi)$,}\\
0   & \text{otherwise.}
\end{cases} \equiv J_0 \cdot \mathbb{I(\theta\in (0, \pi))}$$
where the last, concise expression is in terms of the indicator function $\mathbb{I}(\cdot)$.
The coil's rotation is opposed by various frictional forces.  If the rotation is fast, the dominant force slowing down the coil should be air drag, which can be approximately described as,
$$N = - \gamma \omega |\omega|$$
where $\omega$ is the angular velocity of the coil and $\gamma$ a constant with units $1/\mathrm{radian}$ that depends on the air density and on the coil's shape, but not on its angular velocity.
Putting all this together, the net torque on the loop is,
$$N = -J_0 A B n \,\mathbb{I}(\theta \in (0, \pi)) \sin \theta - \gamma \omega |\omega|.$$
Newton's second law for rotation is $N = I\frac{d\omega}{dt}$, where $I$ is the moment of inertia.  We can use it to write down equations of motion for the loop:
$$I\frac{d\omega}{dt} = -J_0 A B n \,\mathbb{I}(\theta \in (0, \pi)) \sin \theta - \gamma \omega |\omega|$$
$$\frac{d\theta}{dt} = \omega$$
Dynamical systems perspective
To understand the dynamics of this system it's instructive to examine its phase portrait: the motor's trajectories in ($\theta$, $\omega$) space.

The dynamics has only one attractor, the orbit plotted in red that moves through all angles $\theta$ at a roughly constant velocity $\omega$.  Any initial condition other than ($\theta = 0$, $\omega = 0$) flows to this attractor.  This means that regardless of the coil's initial position and the strength of the initial "kick" given to it (in fact, regardless of the kick's direction!), the motor will eventually rotate in its preferred direction, at a roughly constant angular velocity.
The Mathematica notebook used to generate this phase portrait can be downloaded here.
Speed of the motor
From examining the phase portrait, we know that the motor will settle into rotation at a particular angular velocity.  But what is this velocity?
We proceed in the spirit of Street Fighting Mathematics.  The problem has only two parameters, the drive strength $J_0 ABn/I$ with units of $\mathrm{radians/s^2}$, and the drag force constant $\gamma/I$ with units of $\mathrm{1/radians}$.  The only combination of these parameters with units of angular velocity is,
$$\omega_\mathrm{dimensional} \approx \sqrt{\frac{J_0 ABn}{\gamma}}.$$
Therefore, up to some constant that doesn't depend on the system's parameters, this must be the attractor velocity.
If you're too squeamish for a street fight, you can instead approach this problem via the "fast-motor approximation", $\omega \gg \int_0^T |\frac{d\omega}{dt}|\,dt$, where $T$ is the attractor period.  The meaning of the approximation is that the motor's velocity is approximately constant throughout the attractor, as is the case if the attractor is located far from the $\omega = 0$ line in phase space (i.e., if the motor is rotating quickly).  This approach is too long-winded to work through here, but it produces the overall constant dimensional analysis couldn't provide:
$$\omega_\mathrm{fast-motor} \approx \sqrt{\frac{2}{\pi}\frac{J_0 ABn}{\gamma}}.$$
Discussion
The analysis above suggests that the speed of the motor is proportional to the square root of the motor current $J_0$, of the loop area $A$, of the permanent magnet's field $B$ and of the number of coil windings $n$.  It decreases with the square root of the aerodynamic drag coefficient, $\gamma$.
A note of caution
The result derived above depends on some assumptions, and these assumptions may not apply in a particular case.  Most importantly, they will break down if you try to optimize the motor too aggressively by following the prescriptions of the previous paragraph.  Here are some examples of what I mean:

The current through the wire depends on the coil's resistance.  Much of this resistance comes from the poor contacts between the coil and the battery leads, and so is independent of the other device parameters.  But as you make the coil larger (increase $A$) or increase the number of loops $n$, the internal resistance of the coil will eventually dominate the device's resistance, and the current will start scaling as $J_0 \propto \frac{1}{n\sqrt{A}}$.  At this point, adding more windings to the coil or increasing its area will no longer increase the motor's speed.
The air drag coefficient $\gamma$ depends in a complicated way on the shape of the coil.  It almost certainly increases with $n$ and $A$.  (More specific comments from people who actually know hydrodynamics are encouraged!)
Especially for a slow motor, friction at the pivots (which we neglected) may be more important than air drag.
I assumed the coil is sufficiently small compared to the permanent magnet that the magnetic field can be taken to be constant.  As you increase $A$, at some point this assumption must break down: the fringe field of the magnet will become important.  The precise effects of this are difficult to analyze, but almost certainly the motor's speed would no longer increase with the square root of the coil area.

