In general, I would warn you that you aren't likely to get very good rigidity out of this system because by maintaining the quill mechanism and adding the length of the tool holder plus an extra joint (of any type) you are adding more cantilever to what sounds like already fairly sloppy tool. In addition, be warned that the manufacturer of the tool certainly won't endorse any project like this and any joint you implement yourself may fail, so you're going to be spinning a pretty heavy piece of metal pretty fast at your own risk. Personally, I wouldn't recommend it.
That said, you asked about ways to couple the spline shaft with the tool holder's cylindrical end, here are some options:
Firstly you could buy an adapter. Spline shaft adapters are made, but usually only for much larger shafts. In addition, I'm not sure if that spline profile will be standard or specific to your drill press manufacturer. Other people will make you a custom spline adapter, which may be expensive but would get the job done. Most reasonably equipped local machine shops could also make an adapter. The problem with connecting spline-to-spline is that with the cutting forces, you'll have significant axial and radial forces to resist. Splines are good at transmitting torque, and reasonable at resisting radial force, but don't resist axial forces at all. For that reason, pick a spline adapter design where the female spline rigidly cplamps to the spline with a thru-bolt, not one of the designs that only has a set screw.
Another strategy would be to try to develop as much clamping force on the outside of the spline shaft as you can, completely forgoing any contact with the radial faces of the spline. 215
To do this you could use a regular shaft coupling like this or this. Standard rigid shaft couplings with different sized bores at each end are rare though, so you may still need to get one made custom. It is common to find couplings where one end is a finished bore and the other end is unfinished to allow you to bore it out yourself, but I haven't found any small enough for your application. If you do decide to do this, the bore needs to be cut by a machinist on a lathe, don't attempt to use the drill press for it.
The last option, though I'm still not confident you could find parts small enough is a taper lock shaft coupling like the Fenner RM12 shown here or the Climax C600M-9 shown here. They have compatible sizes for 9 mm and 11 mm shafts, but don't seem to make it up to 16mm in one go, so you may still need one special bushing. These type of couplers work on a taper principle, much like a collet and produce probably the strongest and most rigid clamping force of any of the options here. Because these systems are modular (the inside tapered bushing works with multiple flanges) you are most likely to be able to find the sizes you need as stock parts. I have not searched exhaustively for suppliers that may stock your specific sizes.
Overall, I think that this is a very risky idea and may well result in failure or even injury. I suspect that to join such mismatched shafts you will need a part custom-machined, which as you know is not a cheap proposition. If you do need something custom machined, you might as well get a whole new shaft machined that has the 9mm spline profile and the tool holder itself, but by the time you pay for that, you probably could have bought a small used mill with much less hassle and risk.
There are of course other options like drilling a through hole radial to the shaft and inserting a bolt or pin, or welding the two shafts together, but neither will give you nearly the rigidity or concentricity you need, and they are likely even more dangerous that the other options. 215