How to serve previously uploaded video files in Django
Is there a way to load video files via Django but them serve them
  using a different server?

You should define what you mean by "different server". but I assume, you mean different project that is not written in Django.  
Since video files land in file system ( if you design so), you can access them as you want if different project is running on the same server. otherwise you would need some file sync between servers. if you want to distinguish which video file belongs to which object in db, I would insert the object name into filepath. 
if I didnot fully answer your question, let me know below
I'm developing a Django site which allows users to upload PDF, image and video files.  Django is able to serve the pdf and image files comfortably for my purposes but cannot cope with the video downloads.  Is there a way to load video files via Django but them serve them using a different server?