The current GPUs have to deal with a cumbersome, often unfit memory hierarchy. What an FPGA-like model would bring is a much more flexible way to configure the local memories layout. Source - experience of getting over 100x difference in performance wity memory opimisations alone.