The 22-year-old, who had a loan spell at Rotherham earlier this season, has agreed a deal with the Addicks until the summer of 2019.
Forster-Caskey scored nine goals in 82 appearances for the Seagulls after making his debut aged 16 in 2010.
"Jake is a real talent who has shown his quality in the Championship," boss Karl Robinson told the club website.
Former England Under-21 international Forster-Caskey worked under 36-year-old Robinson last season during two loan stints at former club MK Dons.
He is Charlton's second signing of the January transfer window, and his arrival at The Valley came after winger Ademola Lookman left the Addicks to join Premier League side Everton in an £11m deal.
Meanwhile, Brighton under-23 midfielder Jack Harper has moved to Spanish side Malaga for an undisclosed fee.
Find all the latest football transfers on our dedicated page.

The idea is to create a pyramid for club cricket that will raise standards on and off the pitch.
Andrew Watson, an executive director on the Yorkshire Cricket Board, was one of the key thinkers behind the decision.
"We had a national playing survey for three years," he said. "Stronger players and the stronger clubs wanted more competitive cricket and the other players from other leagues wanted more regional cricket, so they didn't have to travel so far. We engaged every league in Yorkshire with that."
Under the old system, there were two premier leagues - the North Yorkshire League and the South Durham League.
The Yorkshire league has been split into two to form Yorkshire League North and Yorkshire League South, with the Bradford League added in. There are then partner leagues beneath these and promotion and relegation are both possible.
The clubs are now coming to the end of their first year under this system. The top two teams from each of the premier leagues will feature in an end-of-season champion of champions play-off, with the final taking place in Abu Dhabi in October.
One long-standing club affected by this change is Pudsey Congs - former player and club spokesperson Ralph Middlebrook believes the move will bridge the gap between league cricketers and professionals, as clubs that win promotion to one of the premier leagues must also meet ECB ground standards.
"The standard of cricket is going to improve and the standard of pitches is going to improve," said Middlebrook.
"There are certain factors that are mandatory on clubs. You must have under-11s, under-13s, under-15s. You must have covers. You must have Clubmark, which is the ECB standard for everything, from child welfare through to development plans.
"It's going to give people who are keen to be professionals the chance to go to a club that's on the verge of the first-class game.
"And it's not just about excellence. It's about the other clubs lower down being able to find their own level."
He continued: "If you just want to be a local club, with people in the locality playing and supporting cricket in their area, you can do that. Not all clubs want to be Premier League clubs with the professional standards they would have to provide.
"If you're thinking of being a pro, you'd better go to a decent club with a decent pitch and see if you can hack it."
Ringing these changes could be difficult for sides with limited resources, but the Yorkshire Cricket Board have a support network in place for clubs facing the transition.
There are five development managers and 10 community officers in Yorkshire, and Cricket Development Groups can help clubs work towards Clubmark.
There are talks in place to look at implementing this system throughout the country.
"I think ECB and cricket nationally is watching Yorkshire," Watson explained. "We have got 12% of all cricket in the country, so we're the biggest output area. To reorganise the leagues as we've done has taken a lot of discussions - it's taken over 200 meetings to get from where we were.
"Most people have been watching Yorkshire to see how it fits, and we're now going round to other counties to offer assistance and help.
"Their initial responses have been a lot of questions about people and territories, because people are precious of what they've got.
"We're planning a blueprint now for the next two decades, so it's very important we do it right."
Subscribe to the BBC Sport newsletter to get our pick of news, features and video sent to your inbox.
League One side Charlton Athletic have signed midfielder Jake Forster-Caskey from Brighton for an undisclosed fee.
[NEXT_CONCEPT]
Amateur cricket in Yorkshire received a major overhaul in 2016 with the launch of the new Yorkshire Premier Cricket system.