Question: I have seen that android does not allow installation of any apk when screen filters are enabled.Do they cause any security risks?

Answer: 
The problem is with the DRAW OVER APPS permission. Android will not let you change any system level changes or install anything while an app is overlaying your screen. The reasoning being since the app can hide a malicious request and have the button pressed under it. Be very wary of any app that requests this permission and make sure you truly trust the developer.
Check out this link for more details on tapjacking. [XDA Developers - Tapjacking](https://www.xda-developers.com/how-tapjacking-made-a-return-with-android-marshmallow-and-nobody-noticed/)
Yes, they absolutely create security risks. For them to work, you must grant them permission to draw over other screens on your phone, while passing touch events to the underlying screen.

A benevolent, well-behaved app draws a mostly transparent red overlay on top of all other screens.

A malicious app could very well display its own screens completely opaque while secretly passing your touches along to an underlying app. That means you might be doing just about anything, including granting the app more permissions or allowing it to install other apps without having any idea.