Grimshaw, who hosts Radio 1's breakfast show, replaces Louis Walsh, taking his seat beside show creator Simon Cowell and Cheryl Fernandez-Versini.
Pop star Rita Ora fills the fourth chair, jumping ship from BBC One's The Voice after just one series.
The singer previously appeared as a guest judge on The X Factor in 2012.
"I am excited to be coming back," said the star. "I had a ball here in 2012. This is a journey I can't wait to embark on to find some talented singers.
"It's the new generation taking over. Booooom."
Grimshaw added: "I love X Factor and have watched the series religiously since it started, so obviously I'm hugely excited to be joining the show.
"Music is my job and my passion and I'm really looking forward to seeing what talent we can find and nurture this year. I can't wait for judges' houses in Oldham!"
Ora and Grimshaw are close friends, and producers will be hoping their chemistry adds a new dynamic to the venerable TV talent show.
Cowell said: "The best-kept secret in TV is out!! But I am really happy with the new judging line-up for X Factor this year. It's going to be a lot of fun."
He said he had "met a lot of people" when looking for new judges. "We have always had great judges. And I will always be thankful for their passion and expertise," he said.
"I have no idea what the show will look like but I feel both excited and nervous. It's going to be very different! Embrace the madness..."
Dermot O'Leary and Louis Walsh are out.
Nick Grimshaw, Rita Ora and Olly Murs are in.
The new appointees for The X Factor suggest Simon Cowell is chasing a younger audience as the show enters its 12th year.
At the moment the programme has a broad reach, but only 8.9% of the viewers are in the 16-24 age range. That's more than The Voice UK but less than ITV's other reality behemoth I'm A Celebrity... Get Me Out Of Here!
Grimshaw and Ora certainly appeal to a younger audience than the judges they replace (sorry, Mel B). Add in the fact that Kiss FM presenters Rickie and Melvin are being tipped to host The Xtra Factor spin-off, and it seems like ITV is aiming for a more streetwise, credible feel.
Which all makes sense: The show's reputation lives or dies on the success of the winning acts. With younger audiences casting votes, the chances of producing another Little Mix or One Direction increase.
It is not the first time ITV has poached one of the BBC's talent show judges; Alesha Dixon was lured from Strictly Come Dancing to Britain's Got Talent in 2012.
Last week, Ora said she was enjoying being fought over by bosses of The X Factor and The Voice.
"It is very complimentary, of course," the pop star told the Evening Standard. "A massive compliment."
"Watching a bidding war going on makes me feel special."
X Factor is being given a thorough shake-up after ratings for last year's final dipped to a 10-year low.
An average audience of 9.1 million tuned in to see Croydon's Ben Haenow win the contest, the lowest since Steve Brookstein's victory in 2004.
Only Cowell and Fernandez-Versini are returning to their roles in 2015.
Host Dermot O'Leary will be replaced by former contestant Olly Murs and TV presenter Caroline Flack, while DJ Sarah Jane Crawford has also stepped down as presenter of spin-off show The Xtra Factor.
This year's arena auditions - which mark the contestants' first opportunity to perform for the panel - begin in Manchester on 6 July.
Pop star Rita Ora and BBC Radio 1 presenter Nick Grimshaw have signed up as judges for the 12th series of The X Factor, producers have confirmed.