Just put a multi-layer Linear + ReLU neural network on top of these features.
You can read this paper by Karpathy:
http://www.cv-foundation.org/openaccess/content_cvpr_2014/html/Karpathy_Large-scale_Video_Classification_2014_CVPR_paper.html 41
The early-fusion / late-fusion etc. are relevant.