Question: I am trying to implement an algorithm where given an image with several objects on a plane table, desired is the output of segmentation masks for each object. Unlike in CNN's, the objective here is to detect objects in an unfamiliar environment. What are the best approaches to this problem? Also, are there any implementation examples available online?

Edit: I am sorry, question may have been a bit misleading. What I meant by "unfamiliar environment" is that objects may be unknown to the algorithm. The algorithm shouldn't need to understand what the object is, but should only detect the object. How should I approach this problem?

Answer: 
Actually, your task is supervised. [`Segnet`](https://arxiv.org/abs/1511.00561) can be good architecture for your purpose which one of its implementations can be accessed [here](https://github.com/alexgkendall/caffe-segnet). *SegNet learns to predict pixel-wise class labels from supervised learning. Therefore we require a dataset of input images with corresponding ground truth labels. Label images must be single channel, with each pixel labelled with its class [...](http://mi.eng.cam.ac.uk/projects/segnet/tutorial.html)*.

Also, take a look at [**Fully Convolutional Networks**](https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf) which are well suited for your task.

---

Based on the edits in the question, I add extra information. There are numerous methods that can be applied for this task. Basically the easiest one is to use a *background* label and classify those classes that you don't know as background by employing the mentioned architectures. By doing so you will have labels which can have overlap for background class which is a probable downside of this approach but its advantage is that in cases where your trained labels are frequently used in the inputs, you can have a relatively light version of architecture which recognizes the unknown classes.
This might be something that you are looking for. Since you ask for image segmentation and not `semantic / instance` segmentation, I presume you don't require the labelling for each segment in the image.

The method is called `scene-cut` which segments an image into class-agnostic regions in an unsupervised fashion. This works very well in case of indoor cluttered environment.

Paper link: [arxiv](https://arxiv.org/abs/1709.07158)

Code: [code](https://github.com/trungtpham/SceneCut)