Do flaps have a maximum speed limit but spoilers don't? Why? This related question on Aviation SE describes how in most aircraft spoilers (flight spoiler mode not ground spoilers) are deploy-able at any speed But flaps on the other hand mostly have a V_FE above which they cannot be deployed. 

I assume the speed limit imposed on using flaps is due to structural reasons? I am just wondering why spoilers do not have a similar limit? Looking at a spoiler photo in the deployed position it looks like the spoiler mechanism ought to be under a lot more stress than the flaps. Am I wrong here? 

Or are spoilers reinforced to a degree out of need that is not felt necessary for flaps? i.e. There can be a legitimate need to deploy flight-spoilers at a speed higher than for flaps.
Spoilers are used to shed energy when the aircraft is too fast or too high. That means they need to be used when the aircraft is flying fast and are designed accordingly.

On the other hand flaps are used to increase lift and when flying fast, there is plenty of lift, so there is no point in using flaps at high speed and therefore they are not designed for it.

Additionally flaps on large aircraft are more complicated mechanically. Spoilers only rotate around their forward edge, but flaps on large aircraft extend to the rear in addition to rotating downward.