Coles (3-54) removed Dan Christian and captain James Taylor at the end of the 48th over and bowled Luke Fletcher with the first delivery of the 50th.
Taylor (109) and Alex Hales (81) helped Notts post 335 all out at Trent Bridge.
Billings then smashed 118 not out off 89 balls after Daniel Bell-Drummond hit 73 as Kent reached 340-5.
Pace bowler Coles' hat-trick came in a remarkable end to the hosts' innings which saw them lose their final six wickets in 10 balls for just four runs.
However, Taylor's fine hundred, which included four boundaries and four sixes, Hales' 77-ball knock and Dan Christian's 59 off 36 balls had already helped Notts to a huge score before the clatter of wickets.
Kent then looked to be heading towards defeat with 106 runs required off the final 10 overs with five wickets in hand.
But England one-day batsman Billings was instrumental as Kent took 54 off the next three overs, before bringing up his hundred off 77 balls.
Alex Blake was equally as aggressive in his unbeaten 50 as the visitors raced to victory with eight deliveries to spare.
Kent move up to fourth in Group B, while leaders Notts have already qualified for the quarter-finals.
Matt Coles took a hat-trick for Kent before Sam Billings' magnificent century saw them beat Nottinghamshire by five wickets in the One-Day Cup.