Usually when two objects collide they make contact along a common surface plane. It might be a very small area, almost a point, but the direction of the normal at this point can be identified without ambiguity. The normal contact force between the object lies along the normal to this contact plane at the point of contact.
For ideal edges the direction of the force is mathematically indeterminate. This is because the surface changes direction discontinuously along an infinitesimally thin line. On one side of the line the surface normal points in one direction, on the other side of the line it points in a different direction. Exactly at the corner it is ambiguous which direction the surface normal is pointing. The discontinuity exists regardless of how close you get to the line.
Practically this might not be a problem for a point object, because if you look on a small enough scale you can always decide on which side of the line the point of contact lies, and therefore in which direction the normal contact force acts.
Real edges (also corners) are rounded to some extent. 586
The local surface normal changes smoothly from one direction to another. If you look on a small enough scale you will be able to identify a local surface plane and its normal. Then you can decide how a point object will bounce if it strikes this local plane.
Real objects have finite size. If they are perfectly rigid then a single point of contact can be identified, and a local surface normal, along which the normal force is exerted during a collision. The direction of friction (if any) can also be determined.
However, if the object is finite and also deformable then it will bend around the edge or corner as it collides, making contact over a curved surface which has normals pointing in a range of directions. It is a very difficult problem to work out in what direction the resultant force on the object will act. Small differences in the line along which the ball approaches, or its angle of approach, or its orientation or speed and direction of rotation, or its elasticity, etc - all could make a significant difference to the direction in which the object will bounce off the edge or corner. 544