My Macbook Air isn't outputting the correct resolution I bought a DELL P2416D with a 2560x1440 resolution but I didn't know that it wasn't compatible with OS X. 

I use a mini Display Port to HDMI adapter.

Also the Mac detected the monitor as a TV and so used the YPbPr color format but I already fixed that issue by forcing the RGB color format and I would like to know if I can force a higher resolution or if you know a better solution to my problem.
I ran into the same problem when I bought a Dell UltraSharp Monitor to use with a MacBook Pro.  It appears you may be using an older spec mDP to HDMI cable (Probably HDMI 1.2)


  Version 1.2 HDMI 1.2 was released August 8, 2005 and added the option
  of One Bit Audio, used on Super Audio CDs, at up to 8 channels. It
  also added the availability of HDMI type A connectors for PC sources,
  the ability for PC sources to implement only the sRGB color space
  while retaining the option to implement the YCbCr color space, and
  required HDMI 1.2 and later displays to allow low-voltage
  sources.


That would explain why it showed up as a TV using YPbPr color format. 

This particular monitor has a native DP interface - per Tetsujin, I would use that port instead of doing the conversion from mDP to HDMI.

If you want to go with HDMI, you will need an an mDP to HDMI 2.0 adapter and an HDMI 2.0 cable



My suggestion is to go native mDP to Display Port - it's much easier and you don't have to buy separate cables.  

As for your MBA, every model from 2010 onwards supported the 2550x1400 resolution.  So unless you are using a 2009 model, you should be able to get the full resolution you are look for with either of these solutions.