There are 3, 4, 8, and 12-sided nuts, and most exotically, 5-sided nuts all in use for specific applications, but the 6-sided hex nut is most popular because it offers a good trade off between a bunch of factors:
Ease of tightening/loosening, especially in tight spaces: There are six distinct orientations you can grip a hex nut in, each 60 degrees apart. This means that if you can swing your wrench through a 60 degree arc, you can tighten the nut enough to remove the wrench and place it on the next pair of flats. In fact most open end wrenches have the jaws offset 15 degrees form the handle so that by flipping the wrench you can tighten the nut with as little as 30 degrees of access. (Box wrenches often have 12 points, allowing the same principle without the flipping.) A four sided nut would have a significant drawback that even with an offset wrench head, you would need 45 degrees of access to tighten the nut. Additionally, it's much easier to put an open end wrench on a hex nut as the two adjacent faces on the approach side guide the wrench towards the flats. By contrast, if you approach a square nut with the wrench flats not quite parallel to the nut flats, the wrench can very easily jam. Of course when there is space for access with a ratchet these considerations don't make much difference, but as people try to make machines smaller and lighter, designers often leave only the minimum access required to reach fasteners.
Torque transmitted The more points the nut has, the more likely it is that the bearing surface of the point will fail in compression and 'round over.' This problem is exacerbated by the fact that some gap has to be left between the wrench and the nut to allow for manufacturing and alignment tolerances, which significantly decreases the contract area between the wrench and the nut. In practice, it's pretty hard to round over a hex nut unless you are using an adjustable wrench (where slop in the mechanism can lead to a bigger gap between the nut and the wrench and less parallel wrench faces.) A square nut is even harder to round over, and in this aspect is more desirable than a hex nut. 12-point nuts do exist, but if the faces were flat they would round over very easily, so instead they are manufactured with points that increase the size of the torque-transmitting face.
Joint Properties Standard square nuts have a larger bearing area than their standard hex counterparts. This sometimes makes them preferable for a connection to a soft substrate like wood as they are less likely to pull through the material. 441
Because square nuts can accommodate a greater gap between the wrench and the nut, they can be used in captive situations like weld-on cage nuts, or self-aligning server rack nuts. One common use for square nuts is on blind, low strength connections to soft wood. The bolt can be tightened from one side and the nut will embed itself in the wood, with the large faces preventing it from cutting a counter-bore into the wood. By contrast a hex nut would typically cut a circle in the wood and spin. Oversized hex nuts, thick washers, and tension control bolts all allow alternate solutions to these problems using hex nuts, but generally at a higher manufacturing cost.
Material Efficiency In order to maintain the minimum amount of material between the threads and the edge of the nut, square nuts have a fair amount of 'wasted' volume in the corners that increases the amount of metal per nut for a given screw size. As you increase the number of flats, the shape of the nut becomes closer to a circle and more material efficient. (By the same logic, when laying out a connection hex nuts take up less space whereas square nuts have to be placed further apart.) For a few nuts, this difference is negligible, but for a company that buys or produces nuts by the truckload, it would add up very quickly.
Ubiquity Even if these trade-offs changed, the simple fact that most people who work on machinery and structures have tools designed around 6-sided nuts would provide a big dis-incentive to change. Auto mechanics, for example have to buy a whole special set of tools when a car manufacturer decides to use an unusual fastener. In the case of the 5-sided nut, the very reason it is used is to be tamper resistant because very few people own wrenches that fit it, and adjustable tools designed for even-numbered polygons won't work. It is mainly used for fire hydrant fittings and valves. 582