I was sharing an iCloud account, would he know about my photos? So I was sharing an  iCloud's account with person I know and lately I took some photos which I later regretted taking.   So I deleted them immediately from my iPhone and I deactivated my iCloud photo library, my photostream and my iCloud photo sharing. Then, I logged out and made my own new account.

I am worried that this person has access to my photos even though I deleted them, because I know it's possible to  recover deleted photos from iCloud.

It is possible for him to recover the photos? Should I be worried?
If you deleted the photos, then went into the Recently Deleted folder and permanently deleted them from your iPhone, followed by disabling iCloud Photo Library as you stated, then you should be fine.