Q:

Non-Linear Density Shell Problem

I'm trying to understand Newton's Shell Theorem (Third)
http://en.wikipedia.org/wiki/Shell_theorem
However this applies to a sphere of constant density. How is this formulated for sphere of varying density, e.g., a ball of gas bound together by gravity? 
Actually, this requires another question: how does the density of ball of gas bound by gravity diminish with radius? 
EDIT
as DMCKEE pointed out "You can't answer the problem ignoring thermodynamics", so I've removed that proviso.

A:

The shell theorem applies to spherically-symmetric mass distributions.  If you have a compressible fluid so that the density varies with depth, you can still apply the shell theorem (unless there is something else going on, for example net angular momentum that breaks the spherical symmetry).  Specifically, the gravitational field is the same as that of a point mass at the center of the distribution.
Here is a demonstration:
Newton's law for the gravitational field of a point mass is
$$F_g/m = GM/r^2$$
with $F_g/m$ the acceleration of a test particle and M the mass of the gravitating body.  This is linear in M, meaning that we can take two mass distributions and superimpose them, then find the gravitational field of this new mass distribution by superimposing the two associated individual gravitational fields.
Assume the shell theorem holds for constant density spheres.  Take two spheres, one slightly larger than the other, so their radii are $r$ and $r+dr$.  Give the larger sphere density $\rho$ and the smaller one density $-\rho$.  Superimpose them.  You get a thin shell of radius $r$ and thickness $dr$.  The shell is empty in the middle because the positive and negative density distributions are canceling there.  (It doesn't matter that there is no such thing as negative mass - the shell theorem is purely mathematical and so holds for negative numbers, and the final distribution we're dealing with has no negative mass; it was just a calculation trick.)  
By superimposing the two gravitational fields of the positive and negative density distributions, we see that the gravitational field of this shell is the same as that of a point mass at the center of the shell with the mass $4\pi r^2 dr \rho$.
The shell theorem thus holds for these thin shells.  A distribution with varying density at different radii is simply a lot of such shells, so the shell theorem holds for arbitrary $\rho(r)$ as long as $\rho$ is a function of $r$ alone.
As for the change in density as a function of radius, we need to balance the hydrostatic pressure with the gravitational attraction.
Take a little piece of the mass distribution with volume $dV$.  It is pulled down by a gravitational force
$$F_g = \frac{GM(r) dm}{r^2}$$
$dm$ is $\rho dV$ and $M(r)$ is the mass located closer to the center of the distribution than the point in question, so
$$M(r) = \int_0^r 4\pi x^2 \rho(x) dx$$
This is balanced by the force from pressure
$$F_p = \nabla P dV$$
We assume there is a relationship between pressure and density so
$$\nabla P = \frac{\partial P}{\partial r} = \frac{d P}{d\rho}\frac{d \rho}{dr}$$
Equating the gravitational and pressure forces gives, after some algebra
$$4\pi G r^2\rho = \frac{d}{dr} \left( \frac{r^2}{\rho} \frac{dP}{d\rho} \frac{d\rho}{dr} \right)$$
The solution depends on $\rho(P)$, and is subject to the boundary condition $P(R) = 0$ with $R$ the radius of the distribution, as well as the condition $M(R) = M$, the total mass.  We will probably find $R = \infty$.
Thus, once we specify a relationship between $\rho$ and $P$ based on the thermodynamics of the system involved, we can integrate the differential equation to find $\rho(r)$.

