FFmpeg produces high system load when downscaling 1920x1080 to 320x240 I'm downscaling an HD live stream to a little "thumbnail" stream, the command looks like this:

ffmpeg -i pipe:0 -vf scale=320:240 -sws_flags fast_bilinear -an -f mpegts pipe:1


Now I would think fast_bilinear is the fastest possible scaler, but it still takes up about 60% of a single core.

Is there another scaler that might be better suited for downscaling? I'm not too interested in the quality.
As @Mulvya said, nearest neighbor is even faster, but will produce bad quality. Nothing resampled, so you can't get faster operation.

But even nearest neighbor and can cause 100% CPU load, it's depends on how much you pump information through it. If you put very large image or 1000fps, you always will get high CPU usage.

Only one thing you may be interested: colorspace. Try to experiment with different color spaces, like yuv420 or rgb32. Maybe some versions of scaler can be faster.