## Assessing the Variety of a Concept Space Using an Unbiased Estimate of Rao's Quadratic Index

Anubhab Majumder 1* , Ujjwal Pal 1 , Amaresh Chakrabarti 1

1 Department of Design and Manufacturing, Indian Institute of Science, Bengaluru, India

*Corresponding author email: anubhabm@iisc.ac.in

## Abstract

Past research relates design creativity to 'divergent thinking', i.e., how well the concept space is explored during the early phase of design. Researchers have argued that generating several concepts would increase the chances of producing better design solutions. 'Variety' is one of the parameters by which one can quantify the breadth of a concept space explored by the designers. It is useful to assess variety at the conceptual design stage because, at this stage, designers have the freedom to explore different solution principles so as to satisfy a design problem with substantially novel concepts. This article elaborates on and critically examines the existing variety metrics from  the  engineering  design  literature,  discussing  their  limitations. A  new  distance-based  variety  metric  is proposed, along with a prescriptive framework to support the assessment process. This framework uses the SAPPhIRE model of causality  as  a  knowledge  representation  scheme  to  measure  the  real-valued  distance between two design concepts. The proposed framework is implemented in a software tool called 'VariAnT.' Furthermore, the tool's application is demonstrated through an illustrative example.

Keywords: conceptual design, variety, SAPPhIRE model, design space exploration.

## 1 Introduction

Designing is regarded as a means of changing existing situations into preferred ones. The engineering design process has been broadly classified into four stages: task clarification, conceptual design, embodiment design and detail design (Pahl &amp; Beitz, 1996). Conceptual design is an early stage in the design process, which involves generating solution concepts to satisfy the functional requirements of a design problem (Chakrabarti  &amp; Bligh, 1994). The conceptual design stage is crucial because a high percentage of the product cost is committed at this stage (Saravi et al., 2008). Decisions made during this stage will strongly affect all the subsequent stages of the design process. Past research (Baer, 2014) relates creativity to 'divergent thinking', i.e., how well the concept space is explored during the early phase of design. The ability to explore the breadth of the concept space is directly related to the ability to restructure problems and is, therefore, an important measure of creativity in design (Shah et al., 2003). Concept space exploration depends on the capacity to produce a wider 'variety' of ideas with higher 'fluency' (Shah et al., 2003). Researchers (Chakrabarti &amp; Bligh, 1994; Shah et al., 2003) have argued that generating  several  concepts  would  increase  the  chances  of  producing  better  design  solutions.  Past  empirical studies also showed that the variety of the concept space is positively correlated with the novelty of the generated set of ideas (Jagtap et al., 2015; Srinivasan &amp; Chakrabarti, 2010a; Kurtoglu, 2009). Generating many concepts, i.e., with higher 'fluency,' that differ from one another only in minor or superficial ways does not prove effective in concept generation (Shah et al., 2003). Thus, it is imperative to generate and explore a diverse set of alternative solution  concepts  during  the  early  stages  of  the  design  process. This  approach  ensures  that  designers  have  a multitude of options to consider before selecting and pursuing the most promising design concept.

Variety indicates how well one has explored the concept space of a design problem. Variety metrics, often referred to as 'diversity' metrics, are also popular in other domains such as economics and ecology. Mathematically, a variety or diversity measure should tell us the probability that two objects (or species in the case of ecology) selected at random (without replacement) from a sample will belong to different groups (Hurlbert 1971; Ahmed et al. 2021).

This article critically examines the existing variety metrics from the engineering design literature and highlights the limitations of the existing metrics through test cases. To address these limitations, a new variety metric is proposed based on Rao's quadratic diversity index (Rao 1982), along with a prescriptive framework to support the assessment process.  The proposed metric offers advantages over the existing variety metrics as it eliminates the need for a tree-like representation of a design concept space. This framework uses the SAPPhIRE model of causality as a knowledge representation scheme to measure the real-valued distance between two design concepts across seven distinct levels of abstraction. The proposed framework is implemented in a software tool called 'VariAnT.' Furthermore, the tool's practical application is demonstrated through an illustrative example where the 'variety' of a concept space generated using an AI chatbot is evaluated.

## 2 Background and Related Work

This section aims to review some of the existing metrics for measuring the variety of a concept space proposed in the engineering design domain. At the conceptual design stage, a designer moves freely between different levels of  abstraction  and  generates  ideas  corresponding  to  a  particular  abstraction  level  (Srinivasan  &amp;  Chakrabarti, 2010b). Most of the existing literature on variety considers four abstraction levels: physical principle, working principle,  embodiment and detail.  Let  us  consider  an  example  case  illustrated  in Figure 1 ,  where  a  designer explores ideas with an AI chatbot to fulfil the function of 'pumping water.' The chatbot provided two ideas at the physical  principle  level:  'centrifugal  force'  and  'positive  displacement.'  Now,  based  on  these  two  physical principles,  five  concepts  are  generated,  each  representing  a  distinct  idea  at  the  working  principle  level.  The exploration  can  go  further  at  the  embodiment  and  detail  level,  which  may  lead  to  a  concept  space C = {C1 ,  C 2 , … ,  C N} with 𝑁 number  of  concepts  where  each  can  be  represented  in α levels  of  abstraction.  For convenience, we have only represented the concepts at two levels of abstraction: physical principle ( α = 1 ) and working principle ( α = 2 ). At each α , there exists a different set of ideas, 𝐼 α = {𝐼 𝑖 α ∣ 𝑖 ∈ 1,2, … , β α} , where βα denotes a total number of ideas at α . Therefore, a concept 𝐶𝑖 can be generated by combining different ideas taken from the idea space 𝐼 α as shown in Figure 1 . We refer to the concept space shown in Figure 1 as 𝐶 𝐴 where 𝑁 = 5 .

Figure 1. Concept space ( 𝐶 𝐴 ) generated by combining different ideas from different levels of abstraction.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-23ca25de-1fac-4dd1-b519-f64aab6d3814.jpg)

In most of the literature on variety assessment, researchers have used a tree-like structure, called a genealogy tree, to represent a concept space. For example, Figure 2 illustrates the genealogy tree of concept space, 𝐶 𝐴 . Here, 𝑛𝑖 α denotes the total number of concepts that use the 𝑖 𝑡ℎ idea from the idea space 𝐼 α . If βα is the maximum number of  ideas  in 𝐼 α ,  then  we  can  write, ∑ βα 𝑛𝑖 α 𝑖=1 = 𝑁 .  Using  the  concept  space 𝐶 𝐴 and  its  corresponding  tree  as  a common ground; we have elaborated on the existing variety assessment techniques in the following part of this section.

Figure 2. Tree constructed from concept space 𝐶 𝐴 .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-aa2a00ba-1fd4-415c-90e7-874bd914755b.jpg)

## 2.1 Existing Variety Metrics in Engineering Design

In engineering design, there are two commonly used approaches to assess design variety: subjective ratings and objective ratings. As an example of subjective evaluation, Linsey et al. (2011) proposed a method where a coder intuitively categorises design concepts based on their overall differences. Concepts with similar features are sorted into different bins or pools. At the end of the sorting process, an individual's variety score is calculated as the ratio of the number of bins the concepts were sorted into to the total number of bins. This subjective approach relies on the  coder's  mental  model  rather  than  a  numerical  procedure  (Ahmed  et  al.,  2021).  In  contrast,  the  objective approaches replace subjective human raters with a deterministic formula that relies on a few measured attributes of a set of designs. Our work particularly focuses on objective approaches. Here, we have reviewed four existing objective variety metrics related to the engineering design domain: the variety metrics proposed by Shah et al. (2003), Nelson et al. (2009), Verhaegen et al. (2013), and Ahmed et al. (2021). Note that other objective variety metrics also do exist in engineering design literature, but we kept those out of our discussion for the following reasons: (a) for example, the metric proposed by Henderson et al. (2017) uses other design assessment metrics such as quality, effectiveness, novelty, applicability, etc. to calculate the variety of the solution concepts and thus, the  variety  metric  cannot  be  considered  as  an  independent  measure;  (b)  in  case  of  the  metric  proposed  by Srinivasan and Chakrabarti (2009), the measure is dependent on the order in which the ideas are generated, even though this should not have any impact of the overall variety of a concept space as long as it contains the same set of concepts, and hence the metric is not applicable for the concept space 𝐶 𝐴 .

## 2.1.1 Metric proposed by Shah

Shah et al. (2003) proposed a metric for measuring the variety of a concept space as follows. The design problem is first decomposed into its essential functions or characteristics. The conceptual origins (i.e., physical principles, working principles, embodiment, and details) of the concepts are analysed through hierarchical or abstraction levels based on how the concepts fulfil each design function. At the highest level of abstraction ( α = 1 ), concepts are differentiated by the physical principles used by each to satisfy the same function; this is the most significant extent of finding differences between concepts. At the second level ( α = 2 ), concepts are differentiated based on working principles, even though they share the same physical principle. At the third ( α = 3 ) and fourth ( α = 4 ) levels, concepts have different embodiment and detail, respectively. The number of branches in the genealogy tree indicates the variety of concepts. If greater variety is to be valued, branches at upper levels should get a higher rating than the number of branches at lower levels. Shah et al. (2003) have assigned values of 10, 6, 3, and 1 to physical principle, working principle, embodiment, and detail levels, respectively. If there is only one branch at a given level (i.e., βα = 1 ), it shows no variety, and the score assigned is 0; otherwise, the score is the number of branches times the weight assigned to that level. A genealogy tree needs to be constructed for each function of a concept. Not all functions are equally important, so a weight 𝑓 𝑗 is assigned to account for the importance of each. Then, the overall variety measure 𝑉 takes the following form:

$$V = 1 0 \times \sum _ { j = 1 } ^ { m } f _ { j } \sum _ { \alpha = 1 } ^ { 4 } \frac { w _ { \alpha } \beta _ { \alpha } } { v _ { m \alpha x } }$$

In Equation 1, βα is the total no. of branches at level α 𝑤α ; is the weight for level α (suggested weights are: 𝑤1 = 10 𝑤2 , = 6 𝑤3 , = 3 , and 𝑤4 = 1 ); 𝑚 is the total no. of functions; and 𝑉 𝑚𝑎𝑥 is the maximum possible variety score. 𝑉 𝑚𝑎𝑥 would be obtained if all concepts used different physical principles ( α = 1 ). Thus, 𝑉 𝑚𝑎𝑥 = 𝑤1 × 𝑁 where 𝑁 is the total number of concepts. Therefore, Equation 1 reduces to

$$V = \Sigma _ { j = 1 } ^ { m } f _ { j } \Sigma _ { \alpha = 1 } ^ { 4 } \frac { w _ { \alpha } \beta _ { \alpha } } { N }$$

$$\frac { w _ { \alpha } \beta _ { \alpha } } { N }$$

Now, applying Equation 2 to the concept space 𝐶 𝐴 results in variety score 𝑉(𝐶 𝐴 ) = (10×2)+(6×5) 5 = 10 .

## 2.1.2 Refinements proposed by Nelson

Nelson et al. (2009) refined the above metric of Shah et al. to resolve the following two limitations: (a) it produces a lower variety score for greater variety by double counting the ideas at each level in the tree, and (b) the variety score is normalised by the number of concepts, which may not indicate the actual design space exploration. Instead of using the number of branches at each level, Nelson et al. use the number of differentiations of branches at a particular  level.  For  example,  two  branches  at  the  physical  principle  level  correspond  to  only  a  single differentiation between physical principles (i.e. there is only one difference), and three branches at the physical principle level correspond to two differentiations, and so on. Thus, the number of differentiations is always one less than the number of branches at a given hierarchical level. No differentiations occur when a single branch emanates from a node. So, Equation 3 is modified as follows:

$$V = \Sigma _ { j = 1 } ^ { m } f _ { j } \left ( \frac { w _ { 1 } ( \beta _ { 1 } - 1 ) + \Sigma _ { d = 2 } ^ { 4 } w _ { \alpha } \Sigma _ { l = 1 } ^ { \beta _ { d - 1 } } d _ { l } } { N - 1 } \right )$$

Where 𝑤1 (β 1 -1) is  the  score  for  differentiation  at  the  physical  principle level ( α = 1 ), 𝑑𝑙 is  the  number of differentiations at node   (one less than the number of branches emanating from node  ). One is subtracted from 𝑙 𝑙 𝑁 to preserve the normalisation from 0 to 10 since the maximum number of differentiations is one less than the number of concepts. Nelson et al. have also shown that, even when the number of concepts increases, the variety may decrease as it is normalised by the number of concepts, whereas a non-normalised variety score measures the actual design concept space exploration. So, Equation 3 reduces to:

$$V = \sum _ { j = 1 } ^ { m } f _ { j } \left ( w _ { 1 } ( \beta _ { 1 } - 1 ) + \sum _ { \alpha = 2 } ^ { 4 } w _ { \alpha } \sum _ { l = 1 } ^ { \beta _ { \alpha - 1 } } d _ { l } \right )$$

Nelson et al. have suggested the values of 𝑤α to 10, 5, 2, and 1 to assure that at least two concepts at a lower hierarchical level must be added to equal the variety gain by adding a single concept at the next higher hierarchical level. Applying Equation 3 to the concept space 𝐶 𝐴 results in variety score, 𝑉(𝐶 𝐴 ) = 10×(2-1)+5×(1+2) 5-1 = 6.25 .

## 2.1.3 Refinements proposed by Verhaegen

It can be observed from Equation 2 and Equation 3 that, in both cases, the variety measure does not consider the distribution of 𝑁 concepts over βα nodes, i.e., 𝑛𝑖 α . Thus, using both metrics results in a similar variety score for the cases where βα remains constant, and the distribution of 𝑁 concepts in the idea space (i.e., 𝑛𝑖 α ) differs across βα nodes. Verhaegen et al. (2013) addressed the same issue and termed this as 'accounting for the degree of uniformness of distribution'. Following the refinement proposed by Verhaegen et al., Equation 2 can be modified into Equation 5 by replacing βα in Equation 2 with the inverse of the Herfindahl index (Herfindahl 1997).

$$V ^ { \alpha } = w _ { \alpha } \Sigma _ { j = 1 } ^ { m } f _ { j } \left ( \frac { 1 } { N \times H _ { \alpha } } \right ), H _ { \alpha } = \Sigma _ { i = 1 } ^ { \beta _ { \alpha } } p _ { i } ^ { 2 }$$

In Equation 5, 𝑉 α denotes the variety score at abstraction level 𝛼 𝐻α , is the Herfindahl index at level α 𝑝𝑖 ; is the proportion of ideas of variable  , i.e., for the 𝑖 𝑖 𝑡ℎ idea in the idea space 𝐼 α , 𝑝𝑖 = 𝑛 𝑖 α 𝑁 .

Using  Equation  5,  the  variety  score  at α = 1 for  the  concept  space 𝐶 𝐴 can  be  calculated  as 𝑉 1 (𝐶 𝐴 ) = 10 × ( 5 2 2 +3 2 ) = 3.85 . Similarly, for α = 2 , the score 𝑉 2 (𝐶 𝐴 ) = 6 × ( 5 1 2 +1 2 +1 2 +1 2 +1 2 ) = 6 . Unlike Shah et al. and Nelson et al., Verhaegen et al. did not provide an aggregated variety score formula considering all abstraction levels. Nonetheless, one approach to derive an overall variety score 𝑉 is through a weighted average, defined as:

$$V = \frac { \sum _ { V } ^ { \alpha } } { \sum _ { \alpha } }$$

Equation 6 yields an overall variety score for concept space 𝐶 𝐴 in a scale of 0 to 1 , 𝑉(𝐶 𝐴 ) = 3.85+6 10+6 = 0.615 .

## 2.1.4 Metric proposed by Ahmed

Ahmed et al. (2021) described the Sharma-Mittal entropy (SME) as a generalised class of methods for measuring diversity (or variety) in other 'non-engineering' domains. They have shown that the Herfindahl index (also known as Herfindahl-Hirschman index or HHI), adopted by Verhaegen et al. (2013), can also be derived from SME.

However, instead of using the inverse of the Herfindahl index, they have proposed a variant of the HHI named HHID (i.e., Herfindahl-Hirschman Index for Design) as a new metric for measuring variety. According to the proposed method, the variety measure of a concept space at an abstraction level α can be written as follows:

$$V ^ { \alpha } = 1 - \frac { \Sigma _ { \iota = 1 } ^ { \beta _ { \alpha } } ( n _ { f } ^ { \alpha } ) ^ { 2 } } { N ^ { 2 } }, \, f o r \ N, \beta _ { \alpha } \geq 1$$

Ahmed et al. (2021) have compared the above-proposed metric with other existing variety metrics by Shah et al. (2003)  and  Nelson  et  al.  (2009).  They  have  shown  that  the  HHID  has  advantages  over  the  existing  metrics regarding accuracy, sensitivity, optimizability and generalizability. Accuracy was measured with respect to ground truth data sets constructed by experts, and it was found that the HHID-based scores align better with human ratings compared to the other two existing measures. Ahmed et al. (2021) proposed an empirical method of evaluating metric sensitivity by randomly selecting sets of different concepts and comparing the scores obtained by different metrics. The results showed that Shah's and Nelson's metrics gave the same scores to a large percentage of sets and thus proved less sensitive compared to HHID. They have also stated that the HHID closely follows the GiniSimpson Index (GSI), commonly used as a measure of diversity in ecology, given as follows:

$$G S I = 1 - \lambda, \quad \lambda = \sum _ { i = 1 } ^ { Z } \pi _ { i } ^ { 2 }$$

In Equation 8, λ is known as the Simpson Index (Simpson, 1949). π𝑖 (𝑖 = 1 … 𝑍) are the proportion of individuals in the various groups in an infinite population where each individual belongs to one of 𝑍 groupings. λ can be interpreted as the probability that two individuals that are randomly and independently picked from the population belong to the same group. In contrast, the complement of λ in  Equation 8 equals the probability that the two individuals belong to different groups. This is also known as the probability of inter-species encounter (Hurlbert, 1971).

It is important to note that the HHID given in Equation 7 is defined for a sample concept space with 𝑁 concepts, where the estimate of λ , i.e., λ ̂ , is considered as ∑ βα ( 𝑛 𝑖 α 𝑁 ) 2 𝑖=1 . However, using the unbiased estimate of λ , given by Simpson (1949), Equation 7 can be modified as follows:

$$V ^ { \alpha } = 1 - \frac { \sum _ { i = 1 } ^ { \beta \alpha } n _ { i } ^ { \alpha } ( n _ { i } ^ { \alpha } - 1 ) } { N ( N - 1 ) }, \ f o r \ N \geq 2, \beta _ { \alpha } \geq 1$$

We may refer to Equation 8 as the Gini-Simpson Index for Design or GSID. Equation 8 is also known as the GiniSimpson Diversity Index and is widely used by ecologists as a well-known conventional index for measuring diversity in an ecosystem (Chen et al., 2018; Augousti et al., 2021). The unbiased estimate of λ used in Equation 8  is  also  equivalent  to  the  normalised  Herfindahl-Hirschman  Index  (Cracau  &amp;  Lima,  2016).  Now,  applying Equation 7 to the concept space 𝐶 𝐴 results in variety scores 𝑉 1 (𝐶 𝐴 ) = 0.48 and 𝑉 2 (𝐶 𝐴 ) = 0.8 at α =  1 and α = 2 , respectively. Whereas Equation 9 yields variety scores 𝑉 1 (𝐶 𝐴 ) = 0.6 and 𝑉 2 (𝐶 𝐴 ) = 1 at α =  1 and α =  2 , respectively. For large sample sizes ( 𝑁 → ∞ ), GSID asymptotically follows the HHID. The advantage of using GSID instead of HHID as a measure for variety in engineering design is further discussed in Section 3.

## 3 Issues with Existing Variety Metrics

To investigate the issues with the aforementioned variety metrics, these need to be evaluated in terms of accuracy and sensitivity. Ahmed et al. (2021) proposed a procedure for empirically estimating the accuracy of a variety metric  by  comparing  its  alignment  with  a  ground  truth  data  set.  The  ground  truth  data  sets  were  prepared considering expert feedback, domain knowledge, or consensus from many individuals. Here, the term 'accuracy' implies the validity of a metric. A measure can only be validated against an external frame of reference or a universally accepted standard. Using experts' ratings to ensure the validity of a metric is a common practice in creativity research (Hennessey et al. 1999). HHID, as a metric for measuring design variety (Equation 7), was validated  by Ahmed  et  al.  (2021)  against  two  ground  truth  data  sets;  one  was  established  by  using  pairwise comparisons  between  sets  of  polygons,  and  the  other  was  constructed  using  milk  frother  design  sketches. Experiments were conducted to benchmark the HHID with the commonly used variety metrics given by Shah et al. (2003) and Nelson et al. (2009). Results from the experiments showed that HHID outperforms the other two metrics in terms of 'accuracy.'

Apart from 'accuracy,' a metric can also be evaluated in terms of 'sensitivity.' In an ideal case, a variety metric should be able to reflect a change in measurement with varying numbers of concepts in a concept space, i.e., 𝑁 , as well as the distribution of concepts over the βα nodes in the idea space 𝐼 α , denoted by 𝑛𝑖 α . For example, let us consider another example of concept space, 𝐶 𝐵 , illustrated in Figure 3 , where a designer explores ideas with an

AI chatbot to fulfil the function of 'pumping water.' The ideas at the physical principle level are identical to 𝐶 𝐴 (see Figure 1 ): 'centrifugal force' and 'positive displacement.' Five concepts are generated based on these two physical principles, each representing a distinct idea at the working principle level. The genealogy tree of the concept space, 𝐶 𝐵 , is shown in Figure 4 . Unlike 𝐶 𝐴 , in the case of 𝐶 𝐵 , 4 out of 5 concepts share identical ideas ('positive displacement') at the physical principle level (as shown in Figure 3 ). This results in different 𝑛𝑖 α values for 𝐶 𝐴 and 𝐶 𝐵 at  the  physical  principle level, and a 'good' variety metric should reflect this difference while accurately  providing  the  variety  scores  for  these  two  concept  spaces.  In  ecology,  this  property  implies  the 'evenness sensitivity' of a diversity (or variety) measure (Crupi, 2019).

Figure 3. Concept space ( 𝐶 𝐵 ) generated by combining different ideas from different levels of abstraction.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-f05eb163-7243-4fe3-9966-c3040c646dad.jpg)

Figure 4. Tree constructed from concept space 𝐶 𝐵 .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-127768f7-1e0e-4786-a0d0-f2f04690ee58.jpg)

In the following sections, we follow a theoretical approach to evaluate the accuracy and sensitivity of the existing metrics. We argue that GSID (Equation 9) can be used as a refined version of HHID (Equation 7) to provide better accuracy and sensitivity for genealogy tree-based variety assessment compared to other existing metrics. The argument is supported by at least two test cases where the existing metrics are not found to be accurate or sensitive. The variety metrics proposed by Shah, Nelson, Verhaegen and Ahmed are hereafter referred to as 'SVS', 'NM', 'IHI', and 'HHID', respectively.

## 3.1 Test Case I

Consider a concept space with 𝑁 number of concepts. At an abstraction level α (e.g., 'physical principle' in Figure 1 ), the total number of ideas or nodes is two, i.e., βα = 2 (e.g., 'centrifugal force' and 'positive displacement'). Now, there could exist different concept spaces with an identical number of total concepts, say 𝑁 = 20 , but with different distributions of concepts over the two nodes, as shown in Figure 5 . In this case, a compelling variety metric should satisfy the following three properties: (a) It should give the maximum score to a completely even distribution  of  concepts  over  the  two  nodes  in  the  idea  space  (i.e., 𝑛1 α = 𝑛2 α = 𝑁 2 )  as  the  probability  that  two concepts that are randomly and independently selected from the concept space share different ideas from the idea space 𝐼 α is maximum in this case; (b) A lower variety score should be given for a skewed distribution of concepts over the two nodes in the idea space; and (c) The score at α should be strictly 0 when all the concepts share identical ideas (e.g., all the concepts share similar physical principle - 'centrifugal force').

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-85d0c480-6376-4261-8fa4-d332902a30d1.jpg)

Figure 5. Example concept spaces for Test Case I with different distributions of concepts over the two nodes at an abstraction level α.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-db4da120-5c70-4c7e-8f61-6452b2b61352.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-08efeec7-c783-4fe1-9b24-44233f59c299.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-0f7c5f7e-c2f5-4207-a68d-9a74b0e80639.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-50480364-6821-4851-bb92-f4bcff1e8d33.jpg)

Figure 6 presents the scores, scaled to a range of 0 to 1 (with 1 indicating the maximum variety), provided by different variety metrics for different distributions of 20 concepts over two nodes at α. For example, ' 5/15 ' on the x-axis denotes the distribution of 20 pump concepts at the physical principle level, where five concepts work based on 'centrifugal force', and the remaining 15 concepts work based on 'positive displacement.' As shown in Figure 6 , the SVS and NM scores remain constant for both even and skewed distributions except for the ' 0/20 ' distribution, where the score changes from a constant value to 0 , and this is due to the change in βα value from 2 to 1 . From this observation, it can be concluded that both SVS and NM lack 'sensitivity,' leading to inaccurate variety scores for this test case. In the case of IHI, HHID and GSID, the scores gradually decrease when the distribution of concepts becomes more skewed and thus, these three metrics satisfy the property of 'sensitivity.' However, for the ' 0/20 ' distribution, unlike the other metrics, IHI gives a score of 0.05 instead of 0 . Hence, IHI provides an inaccurate score for any finite value of 𝑁 when βα = 1 .  Overall, for Test Case I, both HHID and GSID are found to be more compelling compared to the others in terms of both 'accuracy' and 'sensitivity.'

Figure 6. Illustration of the Test Case I.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-a2cb357f-4142-4c77-ac07-25c1a8785f38.jpg)

## 3.2 Test Case II

In this test case, we consider different concept spaces, as shown in Figure 7 , where for each concept space, the concepts are evenly distributed over two nodes in an idea space 𝐼 α . For example, if there are 𝑁 pump concepts, 𝑁/2 of  them work based on 'centrifugal force', and the rest 𝑁/2 work based on 'positive displacement', i.e., 𝑛1 α = 𝑛2 α = 𝑁/2 . Now, consider there are only two concpets in the concept space, i.e., 𝑁 = 2 and both of them have distinct physical principles. In this case, for the abstraction level 'physical principle', the variety score is expected to be 1 , considering the variety metric provides a score in the range 0 to 1 , and 1 denoting the maximum variety. If 𝑁 is large, the variety score should asymptotically follow the value 0.5 , which is the probability of two concepts randomly and independently selected from the concept space (with 𝑁  → ∞ ) are different at the physical principle level.

Figure 7. Example concept spaces for Test Case II with similar distributions of concepts over the two nodes at an abstraction level α.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-a607f9df-65a2-43e9-a293-78b0bb6eb1e0.jpg)

Considering the case described above, the scores given by different variety metrics, scaled to a range of 0 to 1 , are plotted in Figure 8 , where 𝑁 ranges from 2 to 40 (note that the range is chosen arbitrarily). It can be observed that, unlike other metrics, the scores calculated using HHID remain unchanged for varying 𝑁 . HHID also provides an 'inaccurate' score for 𝑁 = 2 , i.e., 0.5 instead of 1 . It can be noticed that the score HHID gives for any finite value of 𝑁 is nothing but the score obtained from GSID for 𝑁  → ∞ . This has occurred because of using a 'biased' estimator of the Simpson Index ( λ, see Equation 8) while defining HHID (see Equation 7). The bias is significant when 𝑁 is small and thus leads to an 'inaccurate' measure of variety. The other three metrics, i.e., SVS, NM, and IHI, are found to be 'accurate' for 𝑁 = 2 . However, when 𝑁 is large, these three metrics provide variety scores, which  asymptotically  diminish  to  the  value 0 instead  of  following  the  value 0.5 and  hence  are  found  to  be 'inaccurate.' Theoretically, one can also observe from Equation 2 and Equation 3 that the cases when βα is much smaller compared to 𝑁 , both SVS and NM will give a score close to 0 irrespective of the distribution of 𝑁 concepts over the βα nodes in the idea space. The same is true for IHI as well. For a smaller βα and larger 𝑁 , in Equation 5,  the  denominator value becomes much greater than the numerator because of the large square terms in the denominator, which ultimately reduces the variety score close to 0 .

Figure 8. Illustration of the Test Case II.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-4d9af379-5e5b-40c0-b1e9-59cfb18e218a.jpg)

## 4 Rationale for Introducing a New Distance-Based Variety Metric

Test Cases I and II suggest that for genealogy tree-based variety assessment, the 'bias-corrected' GSID is a better metric for assessing the variety of a concept space compared to other existing metrics. Additionally, Test Case II implies  that  in  the  future,  researchers  should  consider  an  'unbiased'  estimator  of  any  existing  entropy-based measure, such as the Sharma-Mittal entropy (Ahmed et al., 2021), while proposing a new metric.

However, a major assumption for all the existing tree-based variety assessment approaches is that all the ideas in the idea space are considered equally distant. Let's consider an example where three different types of clutches are there in a concept space: plate clutch ( 𝐶1 ),  centrifugal clutch ( 𝐶2 ),  and  electromagnetic clutch ( 𝐶3 ).  If  we consider the idea space for 'driving input' (representing a design attribute or an abstraction level), then the three different  ideas  with  respect  to 𝐶1 , 𝐶2 ,  and 𝐶3 are  'spring  force,'  'centrifugal  force,'  and  'electric  current,' respectively. In this case, a domain expert may find 'spring force' and 'centrifugal force' to be more similar to each other compared to 'electric current.' Thus, constructing a tree-like structure or grouping ideas in distinct nodes becomes difficult when the distance variable between two ideas is continuous instead of binary. In such cases, none of the existing metrics are applicable as a measure of variety. The same limitation of existing metrics was also pointed by Ahmed et al. (2021).

From Equations 2, 3 and 5, it can be observed that SVS, NM and IHI metrics use the weight of each function (that a design concept needs to satisfy) to assess variety. The problem with this approach is that the weight of a function they use has to do with the value or usefulness of the concept rather than its variety, which should instead signify how different a concept is from other concepts and not how good these concepts are relative to one another.

Most of the existing variety metrics have used four abstraction or hierarchical levels (physical principle, working principle, embodiment and detail) to represent a concept. These four levels were first introduced by Shah et al. (2003). However, the distinction between a physical principle and a working principle remains unclear, and there is a lack of comprehensive guidelines to break a design concept into these four abstraction levels. For instance, Ramachandran et al. (2018) reported their struggle to define the physical and working principles appropriately while creating genealogy trees for the design concepts. Hence, it is up to the designers to carefully describe the principles of the design concepts to ensure repeatability of their variety scores. The problem becomes more evident for complex design concepts which may use multiple physical principles at different operating states to satisfy an intended function (Majumder et al. 2023). For example, a hair dryer can have two operating states: 𝑂𝑁1 and 𝑂𝑁2 . At 𝑂𝑁1 , it uses two physical principles, 'heating' and 'blowing' of air, whereas, at 𝑂𝑁2 , it only uses 'blowing' of air  to  satisfy  the  intended function. Hence, it is important to choose an appropriate knowledge representation scheme which can describe a complex design concept more comprehensively.

Our work addresses the above issues from three different aspects:

- 1. We eliminate the requirement of genealogy tree-based representation of the concept space and provide a new variety metric, based on Rao's quadratic diversity index (Rao 1982), that is applicable to measure the  variety  of  a  concept  space  irrespective  of  which  abstraction  levels  and  how  many  of  them  are considered while representing the concepts.
- 2. In addition to providing a new metric, we offer a framework for assessing variety where we have adopted the SAPPhIRE model of causality as a knowledge representation scheme that comprehensively captures the function, behaviour and structure of a design concept in seven different abstraction levels: States, Actions,  Parts,  Phenomena,  Inputs,  oRgans  and  Effects  (Chakrabarti  et  al.  2005).  In  our  proposed framework, the distance between two concepts at an abstraction level α is  measured by analysing the textual similarity between their respective SAPPhIRE construct descriptions. All the pairwise distances are stored in a distance matrix.
- 3. A software tool - 'VariAnT' (Variety Assessment Tool) - has also been developed to automate the variety assessment process for a given design concept space.

## 5 A Prescriptive Framework for Assessing Variety

The overall procedural breakdown of the prescriptive variety assessment framework is depicted in Figure 9 . In the  following  part  of  this  section,  we  elaborate on  the framework.  First,  we  present  the  SAPPhIRE model of causality as a knowledge representation scheme for storing the design concepts into a structured data frame. Next, we demonstrate a method for measuring the distance between two design concepts by comparing the constructs of their respective SAPPhIRE models, utilizing a state-of-the-art Natural Language Processing (NLP) technique. Following this, we propose a new variety metric and introduce the 'VariAnT' tool, which embodies the proposed framework. Finally, we provide an example to demonstrate the variety calculation using the new framework.

Figure 9. The overall procedural breakdown of the proposed variety assessment framework.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-4de4183d-5f07-4587-9518-07abdcde6e51.jpg)

## 5.1 Knowledge Representation

## 5.1.1 SAPPhIRE - a Model of Causality

Chakrabarti et al. (2005) introduced SAPPhIRE, a causality model, to comprehensively elucidate the causality of natural and engineered systems. SAPPhIRE comprises seven fundamental constructs or abstraction levels: States, Actions,  Parts,  Phenomena,  Inputs,  oRgans,  and  Effects.  The  interrelationships  among  these  constructs,  as depicted  in Figure  10 a,  can  be  summarised  as  follows:  Parts,  which  encompass  physical  components  and interfaces, play a crucial role in the formation of oRgans, which represent the properties and conditions of a system and its environment necessary for the interaction. oRgans, in conjunction with Input(s) in the form of material, energy, or information, collectively trigger physical Effect(s), which subsequently give rise to physical Phenomena and  induce  a  State  change  in  the  system.  This  State  change  is  further  interpreted  as  Action(s),  an  abstract description of an interaction that can serve as an input or create/activate (new)  Part(s). Figure 10 b shows an example where the SAPPhIRE model explains how a hot body cools down in the presence of a surrounding fluid medium.

Figure 10. (a) The SAPPhIRE model of causality (Chakrabarti et al. 2005), (b) An example SAPPhIRE model explaining how a hot body cools down (Srinivasan &amp; Chakrabarti, 2009).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-8fefd564-b62e-4090-96f5-01e3db72ec42.jpg)

In comparison to the Function-Behaviour-Structure (FBS) model (Gero &amp; Kannengiesser, 2004), in SAPPhIRE, the 'action' encompasses the notion of 'function,' while 'parts' can be understood as 'structure.' The remaining constructs of SAPPhIRE collectively contribute to a comprehensive depiction of 'behaviour.' Many systems can be represented using a single instance of a SAPPhIRE model. However, when dealing with complex systems that necessitate  more  detailed  descriptions,  multiple  SAPPhIRE  models  are  required  for  representation.  By  using multiple instances of the SAPPhIRE, Siddharth et al. (2018) created causal chains to represent the functioning of a complex system, which provides elements of description that are absent in other existing models. In our past work (Majumder et al., 2023), we found that the majority of existing models lack an explanation of how to model the functionalities of a system with multiple operating states (also referred to as multi-state systems), and the models that do consider multiple operating states do not explicitly explain a method to capture the underlying causal  relationships  inherent  within  the  system.  To  address  these  issues,  we  proposed  an  integrated  function modelling approach using SAPPhIRE as the basis (Majumder et al., 2023). Broadly, the proposed approach uses the model of a 'transformation system' (Hubka &amp; Eder, 2012) to identify the 'technical process,' the 'operand(s)' (the entity undergoing transformation), and the 'technical system' (or the 'operator') responsible for driving the 'technical process.' Then, the causal processes associated with the  'transformation system' are explained with multiple instances of the SAPPhIRE model. In the past, researchers also reported the use of SAPPhIRE abstraction levels in synthesising design concepts (Srinivasan &amp; Chakrabarti, 2009; Bhatt et al., 2021; Trollman et al., 2023). For example, Trollman et al. (2023)  employed the SAPPhIRE model to conceptualise different strategies  for countering speculative wheat market fluctuations. Additionally, multiple software tools, such as IDEA-INSPIRE (Chakrabarti et al., 2017) and OPAL (Peters et al., 2021), were developed by researchers that utilise the SAPPhIRE model to create a database of existing design concepts. The SAPPhIRE model has also been used to capture the knowledge obtained during the testing phases of product design, such as the potential failure modes (Siddharth et al., 2020).

Overall, there are at least four significant advantages of using SAPPhIRE as a knowledge representation scheme for assessing the variety of concept space: (a) SAPPhIRE model evolved as an integration of, and therefore has been argued to be richer than, various other models in literature such as those in FBS (Qian and Gero 1996), SBF (Goel et al. 2009), Domain Theory (Abramsky 1991), Theory of Technical Systems (Hubka and Eder 2012) and Metamodel  (Jouault  and  B'ezivin  2006);  (b)  SAPPhIRE  provides  a  comprehensive  causal  description  of  a complex system where three different kinds of causal relationships: 'transitional,' 'action,' and 'transformational' causalities can be explicitly explained (Majumder et al. 2023); (c) Compared to other existing models, SAPPhIRE provides more flexibility for modelling systems with varying levels of complexity (Siddharth et al. 2018); and (d) SAPPhIRE  is  a  generic  model  that  can  represent  the  causality  of  both  natural  and  engineering  systems (Chakrabarti et al. 2005) and thus possesses the potential to enhance the applicability of the proposed framework across various domains in future. However, the current work focuses only on engineering systems.

## 5.1.2 Concept Space Data Frame

Assuming that we already have explored a concept space 𝐶 -  comprised of 𝑁 concepts  - for  a  given  design problem, each concept in the concept space can be represented with a single SAPPhIRE instance or a set of SAPPhIRE instances. For the 𝑖 𝑡ℎ concept ( 𝐶𝑖 ), the associated set of SAPPhIRE instances can be represented as follows:

$$S _ { i } = \{ S _ { i } ^ { a } | \, \alpha \in 1, 2, \dots, 7 \}$$

Where α takes values ranging from 1 to 7 , corresponding to different levels of abstraction, as shown in Figure 11 . If the SAPPhIRE model of the concept 𝐶𝑖 consists of 𝑘𝑖 instances of SAPPhIRE, then a list of SAPPhIRE constructs ( 𝑆 𝑖 α ) at an abstraction level α can be shown as in Equation 11, where 𝑒 𝑖 α𝑘 denotes a construct of the 𝑘 𝑡ℎ instance of SAPPhIRE at abstraction level α .

$$S _ { i } ^ { \alpha } = \{ e _ { i } ^ { \alpha k } | k \in 1, 2, \dots, k _ { i } \}$$

## SAPPhIRE constructs

Instance ID /

Figure 11. Representation of the concept space in a data frame.

| N   |     |     |     |     |     |    |
|-----|-----|-----|-----|-----|-----|----|
|     |     | e21 | ei  |     |     | ei |
|     | 1k1 | 2k1 | 3k1 | 4k1 | 5k1 |    |

## 5.2 Vector Encoding

In this step, the text data stored in the data frame is converted into numerical vectors that can be used to calculate the textual similarity between the SAPPhIRE constructs of two concepts at a particular level of abstraction. For each concept, first, we convert the list of strings, 𝑆 𝑖 α ,  into a single string, 𝐿𝑖 α using a concatenation operation, denoted by ' ⊕ ' in Equation 12.

$$L _ { i } ^ { \alpha } = e _ { i } ^ { \alpha 1 } \oplus e _ { i } ^ { \alpha 2 } \oplus \dots \oplus e _ { i } ^ { \alpha k _ { i } }$$

Next, we consider two such strings: 𝐿𝑖 α and 𝐿 𝑗 α that are generated from the SAPPhIRE constructs of the 𝑖 𝑡ℎ concept and 𝑗 𝑡ℎ concept at abstraction level α . Then, the strings 𝐿𝑖 α and 𝐿 𝑗 α are encoded as vectors  denoted by 𝐴𝑖 α ⃗ ⃗ ⃗ ⃗ and 𝐴𝑗 α ⃗ ⃗ ⃗ ⃗ , respectively. We employ a S-BERT (Reimers &amp; Gurevych, 2019) to generate the text embeddings in our proposed framework. We use an open-source, state-of-the-art pre-trained sentence transformer model, 'all-MiniLM-L6-v2', provided by HuggingFace 1 . This model maps the text to a 384-dimensional dense vector space and is widely used for NLP tasks like clustering, semantic search, semantic similarity, etc. We may also employ other embedding models, such as OpenAI embeddings 2 , universal-sentence-encoder 3 , etc., providing access to high-quality, pretrained language models for various NLP tasks.

## 5.3 Distance Matrix

The distance matrix is a 𝑁  × 𝑁 matrix representing all the pairwise distances (or dissimilarity values) between the design concepts. Once we get the vector embeddings for the 𝑖 𝑡ℎ and 𝑗 𝑡ℎ concept at an abstraction level α , the distance 𝑑𝑖𝑗 α between 𝐴𝑖 α ⃗ ⃗ ⃗ ⃗ and 𝐴𝑗 α ⃗ ⃗ ⃗ ⃗ is calculated as:

$$d _ { i j } ^ { \alpha } = 1 - s i m ( \overline { A _ { i } ^ { \alpha } }, \overline { A _ { j } ^ { \alpha } } ), \ s i m ( \overline { A _ { i } ^ { \alpha } }, \overline { A _ { j } ^ { \alpha } } ) = \frac { \overline { A _ { i } ^ { \alpha } }, \overline { A _ { j } ^ { \alpha } } } { | \overline { A _ { i } ^ { \alpha } } | | \overline { A _ { j } ^ { \alpha } } | }$$

Where, 𝑠𝑖𝑚(𝐴𝑖 α ⃗ ⃗ ⃗ ⃗ , 𝐴 ⃗ ⃗ 𝑗 α ) ⃗ ⃗ measures  the  cosine  similarity  by  comparing  the  orientation  of  two  vectors  in  a  highdimensional abstract space (Nandy et al., 2022). Further, the distance value 𝑑𝑖𝑗 α is calculated by subtracting the cosine similarity value from 1. The distance between any two concepts is associative, i.e., 𝑑𝑖𝑗 α = 𝑑 𝑗𝑖 α , and for 𝑖 = 𝑗 , 𝑑𝑖𝑗 α = 0 .

1 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2

2 https://platform.openai.com/docs/guides/embeddings/embedding-models

3 https://pypi.org/project/spacy-universal-sentence-encoder/

## 5.4 Obtaining Variety Scores

The variety score of the 𝑖 𝑡ℎ concept ( 𝐶𝑖 ) at an abstraction level α becomes the average distance of the 𝑖 𝑡ℎ concept from the other (𝑁 - 1) concepts in that concept space ( 𝐶 ). Therefore,

$$V _ { i } ^ { \alpha } = \frac { \Sigma _ { j = 1 } ^ { N } d _ { j } ^ { \alpha } } { ( N - 1 ) }, \ f o r \ N \geq 2$$

The variety score of a concept space ( 𝐶 ) at an abstraction level α becomes the average variety score of all concepts (i.e., 𝑉 𝑖 α ) in that concept space, as shown in Equation 15.

$$V ^ { \alpha } ( C ) = \frac { \Sigma _ { L j \pi 1 } ^ { N } d _ { L } ^ { \alpha } } { N ( N - 1 ) }, \, f o r \ N \geq 2$$

The expression of 𝑉 α in Equation 15, derived using simple heuristics, is equivalent to an unbiased estimator of Rao's Quadratic Diversity Index (Rao 1982) - one of the widely used measures of ecological diversity (Pavoine et al. 2005, Daly et al. 2018). Henceforth, Equation 15 is referred to as Rao's Quadratic Index for Design or RQID. Note that RQID reduces to the GSID (Equation 9) in the case where all 𝑑𝑖𝑗 α takes the value 0 or 1, i.e., the distance variable between two ideas is binary-valued. Detailed mathematical proof  regarding the unbiased estimate of Rao's Quadratic Diversity Index and its relation with the Gini-Simpson Index can be found in Chen et al. (2018).

Now, considering all abstraction levels of SAPPhIRE, the weighted average variety of an individual concept 𝑉(𝐶𝑖 ) and the concept space 𝑉(𝐶) as a whole can be calculated as follows:

$$V ( C _ { i } ) = \frac { \Sigma _ { \alpha = 1 } ^ { 7 } w _ { \alpha } v _ { i } ^ { \alpha } } { \Sigma _ { \alpha = 1 } ^ { 7 } w _ { \alpha } }$$

$$V ( C ) = \frac { \sum _ { \alpha = 1 } ^ { 7 } w _ { \alpha } v ^ { \alpha } ( C ) } { \sum _ { \alpha = 1 } ^ { 7 } w _ { \alpha } }$$

Similarly,  we  can  also  get  a  weighted  average  distance  matrix 𝐷(𝐶) = [𝐷𝑖𝑗 ] 𝑁×𝑁 for  the  concept  space 𝐶 considering all abstraction levels of SAPPhIRE, where an element 𝐷𝑖𝑗 of the matrix 𝐷(𝐶) is calculated as follows:

∑ 7

𝑤α𝑑 𝑖𝑗

α

𝐷𝑖𝑗

α=1

∑ 7

𝑤α

=

(18)

α=1

The weights ( 𝑤α ) of different abstraction levels are arbitrarily assigned as: 𝑤1 = 7 (Actions), 𝑤6 = 6 (States), 𝑤5 = 5 (Inputs), 𝑤4 = 4 (Phenomena), 𝑤3 = 3 (Effects), 𝑤2 = 2 (oRgans), 𝑤1 = 1 (Parts). The weightage values are adapted from Srinivasan &amp; Chakrabarti (2010a). The rationale is to obtain a higher variety score for a given concept space where the concepts differ at a higher abstraction level. However, finding an optimised set of weights requires further research, which is outside this article's scope.

## 5.5 Computer Implementation

A software tool - called VariAnT (Variety Assessment Tool) - has been developed to automate the assessment process. The tool's Graphical User Interface (GUI) is created with the PySimpleGUI 4 Python library and is shown in Figure 12 . The GUI consists of the following six panels:

- 1. The user can create an Excel file consisting of SAPPhIRE construct information (i.e., 𝑆𝑖 , 𝑖 =  1 …  𝑁 ) of all  the  concepts  (i.e., 𝐶𝑖 , 𝑖 =  1 …  𝑁 )  of  a  concept  space  (i.e., 𝐶 with 𝑁 concepts). This  file  can  be imported into the tool data frame using the Import Data panel.
- 2. The tool also allows users to enter the data directly by typing the SAPPhIRE constructs into the data frame through the Enter data panel.
- 3. Once the data is imported/entered into the tool, the user can use the Concept space panel to display the current  data  frame.  This  panel  displays  the  data  frame  in  a  click-enabled  table  where  each  row corresponds to a particular SAPPhIRE instance description (i.e., 𝑒 𝑖 𝛼𝑘 ) of a concept 𝐶𝑖 .
- 4. As an additional feature, the SAPPhIRE constructs of the 'clicked' row can be exported as an image using the SAPPhIRE instance panel of the tool GUI.
- 5. VariAnT allows the user to choose a vector encoding method, e.g., S-BERT, as discussed in Section 5.2. The user can initiate the assessment process by clicking the Calculate Variety button. The tool enables users to apply uniform weights or customize the weight values for different abstraction levels.

4 https://www.pysimplegui.org/

- 6. Once the calculations are done, the variety score of the concept space, i.e., 𝑉 (𝐶) (see Equation 17), is displayed in the Results panel. In addition to the 𝑉 (𝐶) score, the tool generates four different plots to provide more insights to the user. These plots are discussed in detail in the following section.

Figure 12. VariAnT user interface.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-e364b3ad-2633-44f2-8a4b-b7ffdea688e2.jpg)

## 6 Demonstrating the Proposed Framework with an Example Case

## 6.1 Generating Design Concepts using ChatGPT and SAPPhIRE Model

To demonstrate the proposed variety assessment framework, a sample concept space ( 𝐶 𝑊 , 𝑁  =  4 ) is considered, which consists of 4 concepts generated using the web interface of ChatGPT 5 . We have used the GPT-4o model with default settings. The concepts were generated in response to the following design problem: 'How to boil water?' The model was asked to provide four concepts and explicitly describe them using seven abstraction levels of SAPPhIRE, adhering to the definitions of SAPPhIRE constructs. The prompting strategy used for this purpose is outside this paper's scope and is discussed elsewhere (Majumder et al., 2024). The concepts generated and corresponding  SAPPhIRE constructs  are  given  in Table  1.  Now,  given  the  SAPPhIRE  constructs  of  the  four concepts of the concept space 𝐶 𝑊 , the variety assessment is carried out using the software tool (text embedding model: 'all-MiniLM-L6-v2'; the weights, 𝑤𝛼 ,    assigned  to  different  abstraction  levels  are  consistent  with  the discussion in Section 5.4), and the results obtained from the tool are discussed as follows.

5 https://chatgpt.com/

Table 1. Design concepts and the corresponding SAPPhIRE models generated with the help of ChatGPT (GPT4o).

| SAPPhIRE constructs   | Concept 1: Electric Kettle                                                                                                                                                   | Concept 2: Gas Stove with Kettle                                                                                                               | Concept 3: Solar Water Heater                                                              | Concept 4: Friction Heater                                                                                                                                                               |
|-----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Actions               | Boiling of Water                                                                                                                                                             | Boiling of Water                                                                                                                               | Boiling of Water                                                                           | Water becomes warm                                                                                                                                                                       |
| (change of) States    | Initial State: Cold Water (low temperature), Final State: Hot Water (high temperature)                                                                                       | Initial State: Cold Water (low temperature), Final State: Hot Water (high temperature)                                                         | Initial State: Cold Water (low temperature), Final State: Hot Water (high temperature)     | Initial State: Cool Water (moderate temperature), Final State: Warm Water (higher temperature)                                                                                           |
| Phenomena             | Conversion of electrical energy to heat energy, Transfer of heat energy to water                                                                                             | Combustion of Gas (conversion of chemical energy to heat energy), Transfer of heat from flame to kettle, Transfer of heat from kettle to water | Conversion of solar energy to heat energy, Transfer of heat to water in tank               | Conversion of mechanical energy to heat energy through friction, Transfer of heat from friction rods to water                                                                            |
| Effects               | Joule Heating, Heat Conduction                                                                                                                                               | Chemical reaction between gas and oxygen producing heat, Heat Conduction                                                                       | Photovoltaic Effect, Heat Transfer                                                         | Frictional Heating                                                                                                                                                                       |
| Inputs                | Electrical energy (from power supply)                                                                                                                                        | Natural Gas (fuel)                                                                                                                             | Solar energy (sunlight)                                                                    | Mechanical energy (from manual rotation of the crank handle)                                                                                                                             |
| oRgans                | Conductive material for the heating element, Contact between the heating element and water container, Insulated environment to prevent heat loss, Closed system (lid closed) | Combustible Gas, Oxygen (from air), Metallic Kettle (good conductor of heat), Stable Pot Stand, Controlled Environment (regulated flame)       | Presence of Photovoltaic Cells, Conductive Pipes, Insulated Tank, Availability of Sunlight | High-friction Material for Rods, Smooth Bearings to ensure easy rotation, Water-tight Container, Insulated Environment to prevent heat loss, Constant Manual Input (consistent rotation) |
| Parts                 | Heating Element, Water Container, Power Cord, Lid, Base Plate, Switch, Insulation                                                                                            | Gas Burner, Kettle, Gas Supply Pipe, Knob (for control), Flame Ignition Mechanism, Pot Stand                                                   | Solar Panels, Water Tank, Pipes, Insulation, Heat Exchanger, Temperature Sensor            | Friction Rods, Water Container, Rotating Shaft, Bearings, Crank Handle                                                                                                                   |

## 6.2 Results and Discussion

The variety score of the concept space as a whole is calculated as 𝑉 (𝐶 𝑊 ) =  0.387 . The individual variety scores ( 𝑉 (𝐶𝑖 ), 𝑖 =  1 …  4 ) are exported as a bar chart from the tool, as shown in Figure 13 a. Here, the Friction Heater's individual variety score, 𝑉 (𝐶4 𝑊 ) =  0.46 , is the highest compared to other concepts.

Figure  13 b  shows  the  variety  score  of  the  concept  space  ( 𝐶 𝑊 )  at  an  abstraction  level 𝛼 ,  i.e., 𝑉 𝛼 (𝐶 𝑊 ) (see Equation 15). In this figure, 𝑉 𝛼 (𝐶 𝑊 ) ( = 𝑉 𝑖 𝛼 ̅ ̅ ̅ ̅ ) score, depicted in the box plot, is represented by a horizontal orange line inside the box, providing a visual indication of the central tendency of the 𝑉 𝑖 𝛼 scores (see Equation 14). Here, outliers are shown in red circles. At a particular abstraction level, these outlier values indicate which concept(s) significantly  varied  from  the  other  concepts.  For  example, Figure  13 b  shows  that  at  the  State  change  level, 𝑉 𝑖 𝛼=𝑠𝑡𝑎𝑡𝑒𝑠 score  of  the  Friction  Heater  is  higher  compared  to  the  other  3  concepts.  Srinivasan  &amp;  Chakrabarti (2010a)  conducted  observational  studies  involving  an  individual  designer  in  each  design  session  to  solve  a

conceptual design problem. The objective was to test whether 'an increase in the size and variety of ideas used while designing should enhance the variety of concepts produced, leading to an increase in the novelty of the concept space.' They concluded that a more diverse set of ideas generated at a higher abstraction level leads to a greater  chance  of  producing  a  newer  concept.  Thus,  examining  the  variety  score  across  different  levels  of abstraction, as shown in Figure 13 b, may provide valuable insights into the idea-space exploration. For example, while synthesising concepts, a designer or design team can focus on the abstraction level with a low variety score and try to diversify the ideas in that particular idea space.

Figure 13 c shows a heatmap of the weighted average distances ( 𝐷𝑖𝑗 ) between each pair of concepts ( 𝐶𝑖 𝑊 , 𝐶 𝑗 𝑊 ) where, 𝑖, 𝑗 =  1 …  4 . Using these pairwise distance values, the tool can perform clustering where the user defines the  number  of  clusters.  Once  the  number  of  clusters  is  defined,  the  tool  implements  the  K-means  clustering algorithm  and  prints  the  resulting  cluster  labels. Additionally,  it  generates  a  dendrogram  from  the  clustering results,  as  shown  in Figure  13 d.  This  plot  helps  to  visualize  sets  of  concepts  that  are  similar  to  each  other. Researchers  argued  that  it  is  easier  to  explore  a  large  concept  space  meaningfully  when  designers  browse  a clustered concept space because they only need to go through a few concepts from each cluster to get a fair overview, instead of going through every concept (Langdon &amp; Chakrabarti, 1999).

Figure 13. Results obtained for the concept space 𝐶 𝑊 :  (a)  Individual variety scores; (b) Variety scores of the concept space at different levels of abstraction; (c) The weighted average distances between each pair of concepts; (d) Dendrogram generated from the clustering results.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00684v1-figure-1f61558f-cfae-4b32-90a7-e2d367e412d0.jpg)

## 7 Conclusions and Future Work

This paper reviewed several existing metrics from the engineering design literature that were proposed to quantify the variety of design concept spaces. It was found that most of the existing variety metrics employ a genealogy tree-based approach.  Two test cases were conducted to highlight the limitations of these existing variety metrics. The metrics were evaluated in terms of accuracy and sensitivity. Here, the term 'accuracy' denotes the validity of a metric; 'sensitivity' indicates the ability of a metric to reflect a change in measurement with varying numbers of concepts in a concept space, as well as the distribution of concepts over the nodes of their genealogy tree. Results from two test cases suggested that the 'bias-corrected' GSID (Gini-Simpson Index for Design) was better for assessing the variety of a concept space than the other existing metrics. It was found that a major assumption underlying all these metrics is that the ideas in the idea space are considered equally distant. However, in practice, a real-valued distance can be obtained between two different concepts at a particular abstraction level.

To address this research gap, a new prescriptive framework has been proposed for assessing the variety score of a concept space. One of the distinct features of the proposed framework is the use of the SAPPhIRE model of causality as a knowledge representation scheme, which resolve the problem of representing complex systems. The other significant advantage of the framework is using the RQID (Rao's Quadratic Index for Design) as a variety metric, which enables us to use real-valued pairwise distances instead of creating a genealogy tree. It was found that RQID reduces to the GSID if the distance variable between two ideas is binary-valued, i.e., 0 or 1. Hence, the proposed metric, by default, qualifies for a genealogy tree-based variety assessment. We have also discussed a step-by-step  approach  to  measure  the  distance  between  two  design  concepts  by  comparing  their  respective SAPPhIRE constructs. The proposed variety assessment framework was embodied into a new software tool called 'VariAnT.' The tool provides a GUI and automates the proposed variety assessment process. Finally, the tool was tested with an example concept space, and the results obtained from the tool were discussed. It is important to note that the accuracy of the proposed variety assessment method also depends on the accuracy and consistency of  the  SAPPhIRE  models  created  for  the  respective  concepts  in  a  concept  space.  In  this  research  work,  the SAPPhIRE models used to demonstrate the proposed framework were assumed to be accurate and consistent.

There are three potential research directions in the future: (a) Besides S-BERT, it is also possible to incorporate several alternative vector encoding methods into the framework, such as OpenAI embeddings, universal-sentenceencoder, etc. Thus, finding the most appropriate method or combination of methods that correlate better with expert  variety  score  ratings  requires  an  empirical  investigation;  (b)  In  this  work,  the  weights  of  different abstraction levels have been assigned arbitrarily, where the highest importance is given to the function followed by behaviour and structure. Therefore, additional optimisation algorithms can be incorporated to optimise the set of weights that can maximise the sensitivity or discriminating power of the proposed framework; (c) Lastly, apart from assessing the variety, the proposed framework could also be integrated with existing 'design-by-analogy' or 'analogy retrieval' tools, such as IDEA-INSPIRE (Chakrabarti et al., 2017), DANE (Vattam et al., 2011), etc., so as to provide insights into the variety of the analogy space retrieved by the tool. The proposed method could also be utilised to control analogical distance while performing a search.

Competing interests. The  authors  declare  that  they  have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data  availability  statement. The  authors  confirm  that  all  data  generated  or  analysed  during  this  study  are included in the paper.

## References

Abramsky, Samson (1991). 'Domain theory in logical form'. In: Annals of pure and applied logic 51.1-2, pp. 177.

Ahmed, Faez et al. (2021). 'Design variety measurement using sharma-mittal entropy'. In: Journal of Mechanical Design 143.6.

Augousti, AT et al. (2021). 'A new diversity index'. In: Physical Biology 18.6, p. 066004.

Baer, J. (2014). Creativity and divergent thinking: A task-specific approach. Psychology Press.

Bhatt, A. N., Majumder, A., &amp; Chakrabarti, A. (2021). Analyzing the modes of reasoning in design using the SAPPhIRE model of causality and the Extended Integrated Model of Designing. Ai Edam, 35(4), 384-403.

Chakrabarti, A and Thomas P Bligh (1994). 'An approach to functional synthesis of solutions in mechanical conceptual design. Part I: Introduction and knowledge representation'. In: Research in Engineering Design 6, pp. 127-141.

Chakrabarti, A, P Sarkar, et al. (2005). 'A functional representation for aiding biomimetic and artificial inspiration of new ideas'. In: Ai Edam 19.2, pp. 113-132.

Chakrabarti, A., Siddharth, L., Dinakar, M., Panda, M., Palegar, N., &amp; Keshwani, S. (2017). Idea Inspire 3.0-A tool for analogical design. In Research into Design for Communities, Volume 2: Proceedings of ICoRD 2017 (pp. 475-485). Springer Singapore.

Chen, Youhua, Yongbin Wu, and Tsung-Jen Shen (2018). 'Evaluation of the estimate bias magnitude of the Rao's quadratic diversity index'. In: PeerJ 6, e5211.

Cracau, Daniel and Jos´e E Dur´an Lima (2016). 'On the normalised herfindahl-hirschman index: a technical note'. In: International Journal on Food System Dynamics 7.4, pp. 382-386.

Crupi,  Vincenzo  (2019).  'Measures  of  Biological  Diversity:  Overview  and  Unified  Framework'.  In:  From Assessing  to  Conserving  Biodiversity:  Conceptual  and  Practical  Challenges.  Ed.  by  Elena  Casetta,  Jorge Marques da Silva, and Davide Vecchi. Cham: Springer International Publishing, pp. 123-136. isbn: 978-3030-10991-2. doi: 10. 1007/978-3-030-10991-2\_6. url: https://doi.org/10.1007/978-3-030-10991- 2\_6.

Daly,  Aisling J, Jan M  Baetens,  and  Bernard  De  Baets  (2018).  'Ecological  diversity:  measuring  the unmeasurable'. In: Mathematics 6.7, p. 119.

Gero, John S and Udo Kannengiesser (2004). 'The situated function-behaviour-structure framework'. In: Design studies 25.4, pp. 373-391.

Goel, Ashok K, Spencer Rugaber, and Swaroop Vattam (2009). 'Structure, behavior, and function of complex systems: The structure, behavior, and function modeling language'. In: Ai Edam 23.1, pp. 23-35.

Henderson, D., Helm, K., Jablokow, K., McKilligan, S., Daly, S., &amp; Silk, E. (2017, August). A comparison of variety  metrics  in  engineering  design.  In  International  Design  Engineering  Technical  Conferences  and Computers and Information in Engineering Conference (Vol. 58219, p. V007T06A004). American Society of Mechanical Engineers.

Hennessey,  Beth  A,  Teresa  M  Amabile,  and  Jennifer  S  Mueller  (1999).  'Consensual  assessment'.  In: Encyclopedia of creativity 1, pp. 347-359.

Herfindahl, Orris C (1997). Concentration in the steel industry. Columbia University.

Hubka, Vladimir and W Ernst Eder (2012). Theory of technical systems: a total concept theory for engineering design. Springer Science &amp; Business Media.

Hurlbert,  Stuart  H  (1971).  'The  nonconcept  of  species  diversity:  a  critique  and  alternative  parameters'.  In: Ecology 52.4, pp. 577-586.

Jouault, Fr´ed´eric and Jean B´ezivin (2006). 'KM3: a DSL for Metamodel Specification'. In: Formal Methods for  Open  Object-Based  Distributed  Systems:  8th  IFIP WG  6.1  International  Conference,  FMOODS 2006, Bologna, Italy, June 14-16, 2006. Proceedings 8. Springer, pp. 171-185.

Kurtoglu, T., Campbell, M. I., &amp; Linsey, J. S. (2009). An experimental study on the effects of a computational design tool on concept generation. Design Studies, 30(6), 676-703.

Langdon,  P.,  &amp;  Chakrabarti,  A.  (1999,  August).  Browsing  a  large  solution  space  in  breadth  and  depth.  In Proceedings of the International Conference on Engineering Design (ICED99), Munich (Vol. 3, pp. 18651868).

Linsey, Julie S et al. (2011). 'An experimental study of group idea generation techniques: understanding the roles of idea representation and viewing methods'.

Majumder, Anubhab, Apoorv Naresh Bhatt, and Amaresh Chakrabarti (2023). 'Using SAPPhIRE for Functional Modelling  of  Multi-state  Systems'.  In:  Design  in  the  Era  of  Industry  4.0,  V olume  3.  Ed.  by  Amaresh

Chakrabarti and Vishal Singh. Singapore: Springer Singapore. isbn: 978-981-99-0428-0.

Majumder,  A.,  Bhattacharya,  K.,  &amp;  Chakrabarti,  A.  (2024).  Development  and  Evaluation  of  a  RetrievalAugmented  Generation  Tool  for  Creating  SAPPhIRE  Models  of  Artificial  Systems.  arXiv  preprint arXiv:2406.19493.

Nandy,  Ananya,  Andy  Dong,  and  Kosa  Goucher-Lambert  (2022).  'Evaluating  Quantitative  Measures  for Assessing Functional Similarity in Engineering Design'. In: Journal of Mechanical Design 144.3.

Nelson, B. A., Wilson, J. O., Rosen, D., &amp; Yen, J. (2009). Refined metrics for measuring ideation effectiveness. Design Studies, 30(6), 737-743.

Pahl, Gerhard and Wolfgang Beitz (1996). Engineering Design - A Systematic Approach. Springer London. Pavoine, Sandrine, S´ebastien Ollier, and Dominique Pontier (2005). 'Measuring diversity from dissimilarities with Rao's quadratic entropy: are any dissimilarities suitable?' In: Theoretical population biology 67.4, pp.

231-239.

Peters, S., Fortin, C., &amp; McSorley, G. (2021, July). Investigating the Effectiveness of a Causal MBSE Software Tool in Modelling CubeSat Conceptual Design Data. In IFIP International Conference on Product Lifecycle Management (pp. 128-142). Cham: Springer International Publishing.

Qian, Lena and John S Gero (1996). 'Function-behavior-structure paths and their role in analogy-based design'. In: AI EDAM 10.4, pp. 289-312.

Ramachandran, S. K., Ahed, F., Fuge, M., Hunter, S., &amp; Miller, S. (2019, January). Is there too much variety in the execution of the variety metric. In ASME 2019 International Design Engineering Technical Conferences &amp; Computers and Information in Engineering Conference IDETC/CIE 2018.

Rao,  C  Radhakrishna  (1982).  'Diversity:  Its  measurement,  decomposition,  apportionment  and  analysis'.  In: Sankhy¯a: The Indian Journal of Statistics, Series A, pp. 1-22.

Reimers, Nils and Iryna Gurevych (Nov. 2019). 'Sentence-BERT: Sentence Embeddings using Siamese BERTNetworks'. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. url: http://arxiv.org/abs/1908.10084.

Jagtap, S., Larsson, A., Hiort, V ., Olander, E., &amp; Warell, A. (2015). Interdependency between average novelty, individual average novelty, and variety. International Journal of Design Creativity and Innovation, 3(1), 4360.

Saravi, Mohammad et al. (2008). 'Estimating cost at the conceptual design stage to optimise design in terms of performance  and  cost'.  In:  Collaborative  Product  and  Service  Life  Cycle  Management  for  a  Sustainable World:  Proceedings  of  the  15th  ISPE  International  Conference  on  Concurrent  Engineering  (CE2008). Springer, pp. 123- 130.

Shah, Jami J, Steve M Smith, and Noe Vargas-Hernandez (2003). 'Metrics for measuring ideation effectiveness'. In: Design studies 24.2, pp. 111-134.

Siddharth,  L, Amaresh  Chakrabarti,  and  Srinivasan  Venkataraman  (2018).  'Representing  complex  analogues using  a  function  model  to  support  conceptual  design'.  In:  International  Design  Engineering  Technical Conferences and Computers and Information in Engineering Conference. Vol. 51739. American Society of Mechanical Engineers, V01BT02A039.

Siddharth,  L.,  Chakrabarti, A.,  &amp;  Ranganath,  R.  (2020).  Modeling  and  structuring  design  rationale  to  enable knowledge reuse. Systems Engineering, 23(3), 294-311.

Simpson, Edward H (1949). 'Measurement of diversity'. In: nature 163.4148, pp. 688- 688.

Srinivasan, V., &amp; Chakrabarti, A. (2010a). 'Investigating novelty-outcome relationships in engineering design'. In: AI EDAM, 24(2), pp. 161-178.

Srinivasan, V., and Chakrabarti, A. (2010b). 'An Integrated Model of Designing.' ASME. J. Comput. Inf. Sci. Eng. September 2010; 10(3): 031013. https://doi.org/10.1115/1.3467011

Srinivasan,  V.,  and  Chakrabarti,  A.  (2009).  Sapphire-an  approach  to  analysis  and  synthesis.  In  DS  58-2: Proceedings of ICED 09, the 17th International Conference on Engineering Design, Vol. 2, Design Theory and Research Methodology, Palo Alto, CA, USA, 24.-27.08. 2009 (pp. 417-428).

Vattam, S., Wiltgen, B., Helms, M., Goel, A. K., &amp; Yen, J. (2011). DANE: fostering creativity in and through biologically inspired design. In Design creativity 2010 (pp. 115-122). Springer London.

Verhaegen, Paul-Armand et al. (2013). 'Refinements to the variety metric for idea evaluation'. In: Design Studies 34.2, pp. 243-263.
Anubhab Majumder Ujjwal Pal Amaresh Chakrabarti
Assessing the Variety of a Concept Space Using an Unbiased Estimate of Rao's Quadratic Index
Past research relates design creativity to 'divergent thinking,' i.e., how
well the concept space is explored during the early phase of design.
Researchers have argued that generating several concepts would increase the
chances of producing better design solutions. 'Variety' is one of the
parameters by which one can quantify the breadth of a concept space explored by
the designers. It is useful to assess variety at the conceptual design stage
because, at this stage, designers have the freedom to explore different
solution principles so as to satisfy a design problem with substantially novel
concepts. This article elaborates on and critically examines the existing
variety metrics from the engineering design literature, discussing their
limitations. A new distance-based variety metric is proposed, along with a
prescriptive framework to support the assessment process. This framework uses
the SAPPhIRE model of causality as a knowledge representation scheme to measure
the real-valued distance between two design concepts. The proposed framework is
implemented in a software tool called 'VariAnT.' Furthermore, the tool's
application is demonstrated through an illustrative example.