Well, first of all, the answer to your question depends quite a
lot on whether your asking about RSA specifically, or public key
algorithms in general (of which RSA is one example). For raw
RSA, the core operations to convert between the plaintext $M$
and the ciphertext $C$ is: $C = (M^e) \bmod N$ $M = (C^d) \bmod
N$ where the private key is the pair $(N, d)$ and the public key
is the pair $(N, e)$. At this very high level, it is obvious
that the operation is symmetric, and there's no inherent reason
why one couldn't "encrypt" with the private key, and "decrypt"
with the public (although "encryption" isn't quite the correct
terminology if anyone can decrypt it). However, it is rarely
appropriate to use raw RSA as such; RSA has a homomorphic
property that can easily leak information. To avoid such
weaknesses, we generally perform padding before and after. So,
for RSA public key encryption, we usually do: $C = (Pad(M)^e)
\bmod N$ $M = Depad( (C^d) \bmod N )$ and for performing
signature operations, we do: $C =