Question: I am trying to implement an algorithm where given an image with several objects on a plane table, desired is the output of segmentation masks for each object. Unlike in CNN's, the objective here is to detect objects in an unfamiliar environment. What are the best approaches to this problem? Also, are there any implementation examples available online?

Edit: I am sorry, question may have been a bit misleading. What I meant by "unfamiliar environment" is that objects may be unknown to the algorithm. The algorithm shouldn't need to understand what the object is, but should only detect the object. How should I approach this problem?

Answer: 
The state-of-the-art (SOTA) for image segmentation would be Facebook's [Mask-RCNN](https://arxiv.org/abs/1703.06870).

While it is usually trained on dataset such like [COCO](http://cocodataset.org/#home) or [Pascal](http://host.robots.ox.ac.uk/pascal/VOC/) which feature real-life objects, you can re-trained it on a dataset of your choice, real or not.

Facebook provides an implementation ([Detectron](https://github.com/facebookresearch/Detectron)) under the Apache2 license. Give it a try!
This might be something that you are looking for. Since you ask for image segmentation and not `semantic / instance` segmentation, I presume you don't require the labelling for each segment in the image.

The method is called `scene-cut` which segments an image into class-agnostic regions in an unsupervised fashion. This works very well in case of indoor cluttered environment.

Paper link: [arxiv](https://arxiv.org/abs/1709.07158)

Code: [code](https://github.com/trungtpham/SceneCut)