Question: Let's say I have a business object called Child. It has no persistence logic, and is not designed around the UI. It merely represents a business entity.

Parent will have a reference to the Child objects: List Children

Since Parent has a direct object reference to Child(s), is it bad to also have Child have ParentId? Downside is that it is a duplicated data as there is already reference. Upside, is it will make it easier to populate from the data-layer.

Any opinions on this type of thing in general? If a business object has a reference to another business object is it bad form to also possess the foreign key?

Answer: 
I would avoid bidirectional relationships as much as possible, as they can lead to synchronization problems (`Child` can end up having an obsolete `ParentId`).

That being said, a unidirectional reference to another object *by its ID only* can be very useful.

It's a very natural way of cutting an object graph in slices that can be loaded in memory from the database on demand, as opposed to hydrating the whole graph at once. Much more natural than a complex lazy loading mechanism IMO. References by ID are often used in Domain Driven Design where these slices are autonomous Aggregates.
Keys are used to inform relationships and enforce referential integrity. Whether you need the ParentId key to live in the Child object depends on whether you will find yourself in a situation where you are working with a Child object and you need to know its parent. In general, I would suggest that it is good practice to include the key in the Child object because it allows you to find the identity of the Parent without going through the Parent object. 

You might even run into a situation where you are working with a set of Child objects independent of the Parent objects. In that case, you might decorate the Child object with a minimal set of properties from the Parent object (name, for example) if you need that information, rather than generating the full Parent object. Your solution will depend on the specifics of your use case, the language you are using, whether you expect these objects to be long-lived, the cost of garbage collecting stale objects, etc.