Terminal app can access folders that it isn't allowed In the security and privacy pane I didn't allow the terminal app to access anything.

I didn't allow full disk access either.
However, it can show the contents of Desktop by doing cd Desktop, ls. Why?
My first question is: why would you do that to your terminal.app?
To answer your question. Terminal.app is automatically granted Full Disk Access by default and it does not show up in the Full Disk Access tab on the Security Pane.
You can see this by running the following in your term:
sudo sqlite3 /Library/Application\ Support/com.apple.TCC/TCC.db 'select * from access'
I hope it answers your question.