Question: I am trying to round the corners of a custom shape, but it is just straightening the original curves out.

*Note: This is in Illustrator CS5*

**Here is the shape before the effect:**

[![enter image description here](https://i.stack.imgur.com/luohd.png)](https://i.stack.imgur.com/luohd.png)

The 3 "corners should be rounded".

**Here is what happens when I use the effect:**

[![enter image description here](https://i.stack.imgur.com/3jyfQ.png)](https://i.stack.imgur.com/3jyfQ.png)

**How can I preserve the curves, and just round the actual corners?**

Answer: 
I don't have CS5, but this should work:

Select your path. With the pen tool selected, create points on each side of the anchor points, something like the following:

[![Path with new anchor points added near the original cusp points.](https://i.stack.imgur.com/U3Mk7.png)](https://i.stack.imgur.com/U3Mk7.png)

With the white arrow tool selected, click one of the original cusp points (i.e., corner points) and then `Shift`-click the other two to select all three. Then, click the `Convert selected anchor points to smooth` icon in the toolbar at the top of the screen (alternatively, with the pen tool selected you can `Alt` or `Option`-drag on a corner point to convert it to a smooth point):

[![Convert selected anchor points to smooth icon](https://i.stack.imgur.com/mchHs.png)](https://i.stack.imgur.com/mchHs.png)

You should end up with something similar to this:

[![Path with bulging corners.](https://i.stack.imgur.com/zOo3s.png)](https://i.stack.imgur.com/zOo3s.png)

With the white arrow tool, drag each of the bulging corner points inward until they look appropriate:

[![Bulging corner points fixed.](https://i.stack.imgur.com/glD9N.png)](https://i.stack.imgur.com/glD9N.png)

Here's the new path overlaying your original shape.

[![Path with rounded corners.](https://i.stack.imgur.com/Mz93L.png)](https://i.stack.imgur.com/Mz93L.png)

If you want the corners to be more rounded, make sure the anchor points you added in the first step are farther away from the corner points.
The simple way:
Create additional anchor points around cornes (as it was mentioned at first answer), then apply Round Corner effect as usual.