Crowdsourced Clustering: Querying Edges vs Triangles
We consider the task of clustering items using answers from non-expert crowd workers. In such cases, the workers are often not able to label the items directly, however, it is reasonable to assume that they can compare items and judge whether they are similar or not. An important question is what queries to make, and we compare two types: random edge queries, where a pair of items is revealed, and random triangles, where a triple is. Since it is far too expensive to query all possible edges and/or triangles, we need to work with partial observations subject to a fixed query budget constraint. When a generative model for the data is available (and we consider a few of these) we determine the cost of a query by its entropy; when such models do not exist we use the average response time per query of the workers as a surrogate for the cost. In addition to theoretical justification, through several simulations and experiments on two real data sets on Amazon Mechanical Turk, we empirically demonstrate that, for a fixed budget, triangle queries uniformly outperform edge queries. Even though, in contrast to edge queries, triangle queries reveal dependent edges, they provide more reliable edges and, for a fixed budget, many more of them. We also provide a sufficient condition on the number of observations, edge densities inside and outside the clusters and the minimum cluster size required for the exact recovery of the true adjacency matrix via triangle queries using a convex optimization-based clustering algorithm.
The paper considers the problem of clustering a partially observed graph and compares two types of queries: edge queries vs triangle queries. The problem is motivated by the task of clustering unlabeled items using crowdsourcing platforms. In this setting, an edge query reveals a random pair of items to a crowd worker for comparison whereas a triangle query reveals three random items and asks if any two of them, none of them, or all items are in the same cluster. The authors then try to answer the following question: for a fixed query budget, which query type can better retrieve the true underlying graph and therefore results in a more exact clustering?   They provide a few generative stochastic models for both query types. The models allow them to compute the probability that a random edge in the observed graph is in error. They show that under reasonable assumptions on the parameters of their models, triangle queries find more reliable edges. They provide empirical evidence by presenting results from several simulations and two real data sets. Their experiments show that clustering via triangle queries outperforms clustering via edge queries by finding more reliable edges and finding more edges for a fixed budget.   
The authors propose a novel strategy for partially observing a graph that consists of disjoint cliques(clusters). As they mention in the paper, an important application of this problem is in crowdsourcing platforms where a fixed budget is given and we want to cluster a large set of items as precisely as possible. While I think the problem is interesting and potentially useful across a variety of applications, I have two concerns regarding the novelty and thoroughness experimental analysis.          Regarding novelty, although the authors provide some related work, I feel they could greatly improve on the comparison on two areas in particular: (1) the benefit of triadic comparisons in crowdsourcing and (2) the utility of sampling triangles for graph or adjacency matrix estimation. Regarding (1) the authors do mention related work but there is no experimental comparison with any of them.  Similarly for (2)"Adaptively Learning the Crowd Kernel" provides a method to estimate the adjacency matrix -- how does this compare to the current work?          Along this line the authors must either compare their approach to other approaches and give a clear explanation of why this is not possible. Further, the experiments do not leave me convinced that querying triangles is in general a better strategy. This is because of the way that authors compare costs of an edge query with a triangle query. In order to have a fair comparison between the number of edge queries and triangle queries that one can generate with a fixed budget, each query type must be associated with a cost. The authors suggest that the cost of a query can be measured by computing its entropy which is only available if the generative model of the graph is known. For real world problems, the authors use the average response time per query as the cost of a query. In section 3, they claim that empirical studies shows that the cost of a triangle query is approximately 1.5 times of an edge query, but there is no reference for this. Of course one can empirically estimate this ratio for a new data set by finding the average response time per query for each query type, but what if this ratio is very large and therefore only a few triangle queries are allowed with the given budget? It is not clear to me why querying triangles is a better choice in this scenario. On the other hand, the paper claims that the cost of a triangle query is never more than 3 times the cost of an edge query because each triangle query contains three edges. I am not convinced by this reasoning. It seems that they ignore the fact that identifying dependencies between edges makes the problem harder to solve but at the same time they are using the advantage of solving triangle queries which are more reliable than edge queries exactly because they consider edge dependencies.            I would suggest the authors compare the performance of edge queries vs triangle queries over a wider range of cost ratios to study if there is a turning point such that using edge queries reveals more information about the graph for a fixed budget(i.e., the cost of triangle queries are so high that we can only make a few of them). Followed by an empirical study that shows triangle queries are statistically unlikely to be so expensive.