As you reightly mentioned the star destroyers were there to keep the rebels in the firing range of the fully operational death star.
Even with the rebels being played there (and their secret service) they still had a fully operational (yepp that sounds like a pun compared to the ds) secret service. Thus any LARGE number of star destroyers that suddenly disappear would very probably raise a few concerns among the rebels if it is a TOO GOOD opportunity and thus nothing else but a trap.
Thus the emperor needed to use only a small number of star destroyers. Small enough to raise no suspicion and large enough to keep the rebels in range. For that he used star destroyers and vaders super star destroyer.
As we have seen the rebel fleet consisted of a good number of ships, BUT not that many capital ships. And a star destroyer vs. a few corvettes or a nebulon B frigate .... 963
is a fight going for the star destroyer (not to mention a super star destroyer!). Thus that fleet SHOULD have been enough to do both jobs. Keep the rebels in check and possibly even destroy them while raising no suspicion.
What he did not take into account was what his death and the destruction of the death star would cause in terms of disorder among the star destroyers AND that the rebels would take out (at least the super star destroyer) star destroyers by crashing into their bridges and destroying their overly exposed shield generators.
So all in all the empire had all the aces but still got beat.
THe only thing that is really strange there is one little detail. With how many Ties the star destroyers and the super star destroyer fielded.....the rebel fighters should have been overrun with no chance to counteract. Even with a better quality the numbers would have been enormously stacked against them.....alone 26 star destroyers (72 tie fighters / ship) would field 1,8k Tie Fighters !!!!! While the rebels had nowhere near that number of fighters.
I would count that part as continuity error as I can't fathom any reason why not at least 900 fighters were crushing the rebel fighters. But else it is clear that the risk of the rebels getting wind of the trap if more star destroyers disappeared from the screen was seen as too high (especailly as the rebels had also spies within imperial ranks....thus only the most trusted star destroyer crews were used). 963