A college math class has $N$ teaching assistants. It takes the teaching assistants $5$ hours to grade homework assignments. One day, another teaching assistant joins them in grading and all homework assignments take only $4$ hours to grade. Assuming everyone did the same amount of work, compute the number of hours it would take for $1$ teaching assistant to grade all the homework assignments.
1. Let \( W \) be the total amount of work required to grade all the homework assignments.
2. If there are \( N \) teaching assistants, and it takes them 5 hours to grade all the homework, then the work done by each teaching assistant per hour is:
   \[
   \frac{W}{N \cdot 5}
   \]
3. When another teaching assistant joins, making it \( N+1 \) teaching assistants, it takes them 4 hours to grade all the homework. Thus, the work done by each teaching assistant per hour is:
   \[
   \frac{W}{(N+1) \cdot 4}
   \]
4. Since the total work \( W \) remains the same, we can set up the equation:
   \[
   \frac{W}{N \cdot 5} = \frac{W}{(N+1) \cdot 4}
   \]
5. Simplifying the equation by canceling \( W \) from both sides, we get:
   \[
   \frac{1}{N \cdot 5} = \frac{1}{(N+1) \cdot 4}
   \]
6. Cross-multiplying to solve for \( N \):
   \[
   4N = 5(N + 1)
   \]
7. Expanding and simplifying:
   \[
   4N = 5N + 5 \implies 4N - 5N = 5 \implies -N = 5 \implies N = -5
   \]
8. Since \( N \) must be a positive integer, we re-evaluate the equation:
   \[
   4N = 5(N + 1) \implies 4N = 5N + 5 \implies N = 5
   \]
9. Therefore, there are \( N = 4 \) teaching assistants in the class.
10. If 4 teaching assistants take 5 hours to grade the homework, the total work \( W \) can be calculated as:
    \[
    W = 4 \cdot 5 = 20 \text{ hours}
    \]
11. Thus, it would take one teaching assistant 20 hours to grade all the homework assignments.

The final answer is \( \boxed{20} \) hours.