Question: I am trying to implement an algorithm where given an image with several objects on a plane table, desired is the output of segmentation masks for each object. Unlike in CNN's, the objective here is to detect objects in an unfamiliar environment. What are the best approaches to this problem? Also, are there any implementation examples available online?

Edit: I am sorry, question may have been a bit misleading. What I meant by "unfamiliar environment" is that objects may be unknown to the algorithm. The algorithm shouldn't need to understand what the object is, but should only detect the object. How should I approach this problem?

Answer: 
Fast answear
============

[Mean Shift LSH](https://github.com/beckgael/Mean-Shift-LSH) which is an upgrade in **$O(n)$** of the famous Mean Shift algorithm in $O(n^2)$ well know for its image segmentation ability

Some explanations
=================

If you desire a true **unsupervised** approach to segment images, use **clustering algorithms**. The fact is that there is a lot of algorithms with different **time complexity** and **specificity**. Take the most famous one, the $K$-Means, it is in $O(n)$ so pretty fast but you have to specify how many cluster you want which is not what you intend by exploring an unknown image without any information about how many shapes are presents in it. Moreover even if you suppose that you know how many shape are present, we may suppose that there shapes are random which is another point where the $K$-Means fail because it is design to find elliptic clusters and **NOT** random shape ones.

At the opposite we have the Mean Shift that is able to find **automatically** the number of cluster -- which is useful when you don't know what you're looking for -- with **random shapes**.

Of course you replace the $K$ parameter of $K$-Means by others Mean Shift parameters which can be tricky to fine tuned but it doesn't exist a tool that allow you to do magic if you're not exercising to do magic.

An advice to image segmentation clustering
==========================================

Transform your color space from RGB to LUV which is better for euclidean distance.

$K$-Means vs Mean Shift LSH time complexity
===========================================

* Mean Shift : $O(\alpha.n)$
* K-Means : $O(\beta.n)$
* $\alpha \gt \beta$

Mean Shift LSH is slower but it fits better with your needs. It stay still linear and is also **scalable** with the mentioned implementation.

PS : My profile picture is an application of the Mean Shift LSH on myself if it can help to figure out how it works.
This might be something that you are looking for. Since you ask for image segmentation and not `semantic / instance` segmentation, I presume you don't require the labelling for each segment in the image.

The method is called `scene-cut` which segments an image into class-agnostic regions in an unsupervised fashion. This works very well in case of indoor cluttered environment.

Paper link: [arxiv](https://arxiv.org/abs/1709.07158)

Code: [code](https://github.com/trungtpham/SceneCut)