The rectangular faces of rectangular prism $A$ have perimeters $12$, $16$, and $24$. The rectangular faces of rectangular prism $B$ have perimeters $12$, $16$, and $20$. Let $V_A$ denote the volume of $A$ and $V_B$ denote the volume of $B$. Find $V_A-V_B$.
1. Let \( x, y, z \) be the dimensions of rectangular prism \( A \). The perimeters of the faces are given by:
   \[
   2(x+y) = 12, \quad 2(x+z) = 16, \quad 2(y+z) = 24
   \]
   Dividing each equation by 2, we get:
   \[
   x + y = 6, \quad x + z = 8, \quad y + z = 12
   \]

2. To find \( x, y, z \), we add the three equations:
   \[
   (x + y) + (x + z) + (y + z) = 6 + 8 + 12
   \]
   Simplifying, we get:
   \[
   2x + 2y + 2z = 26 \implies x + y + z = 13
   \]

3. Now, we solve for \( x, y, z \) using the equations:
   \[
   x + y = 6, \quad x + z = 8, \quad y + z = 12, \quad x + y + z = 13
   \]
   Subtract \( x + y = 6 \) from \( x + y + z = 13 \):
   \[
   z = 13 - 6 = 7
   \]
   Subtract \( x + z = 8 \) from \( x + y + z = 13 \):
   \[
   y = 13 - 8 = 5
   \]
   Subtract \( y + z = 12 \) from \( x + y + z = 13 \):
   \[
   x = 13 - 12 = 1
   \]

4. The dimensions of prism \( A \) are \( x = 1, y = 5, z = 7 \). The volume \( V_A \) is:
   \[
   V_A = x \cdot y \cdot z = 1 \cdot 5 \cdot 7 = 35
   \]

5. Let \( x', y', z' \) be the dimensions of rectangular prism \( B \). The perimeters of the faces are given by:
   \[
   2(x' + y') = 12, \quad 2(x' + z') = 16, \quad 2(y' + z') = 20
   \]
   Dividing each equation by 2, we get:
   \[
   x' + y' = 6, \quad x' + z' = 8, \quad y' + z' = 10
   \]

6. To find \( x', y', z' \), we add the three equations:
   \[
   (x' + y') + (x' + z') + (y' + z') = 6 + 8 + 10
   \]
   Simplifying, we get:
   \[
   2x' + 2y' + 2z' = 24 \implies x' + y' + z' = 12
   \]

7. Now, we solve for \( x', y', z' \) using the equations:
   \[
   x' + y' = 6, \quad x' + z' = 8, \quad y' + z' = 10, \quad x' + y' + z' = 12
   \]
   Subtract \( x' + y' = 6 \) from \( x' + y' + z' = 12 \):
   \[
   z' = 12 - 6 = 6
   \]
   Subtract \( x' + z' = 8 \) from \( x' + y' + z' = 12 \):
   \[
   y' = 12 - 8 = 4
   \]
   Subtract \( y' + z' = 10 \) from \( x' + y' + z' = 12 \):
   \[
   x' = 12 - 10 = 2
   \]

8. The dimensions of prism \( B \) are \( x' = 2, y' = 4, z' = 6 \). The volume \( V_B \) is:
   \[
   V_B = x' \cdot y' \cdot z' = 2 \cdot 4 \cdot 6 = 48
   \]

9. Therefore, the difference in volumes is:
   \[
   V_A - V_B = 35 - 48 = -13
   \]

The final answer is \(\boxed{-13}\)