While hyper-parameters (HPs) are important for knowledge graph (KG) learning, existing methods fail to search them efficiently. To solve this problem, we first analyze the properties of different HPs and measure the transfer ability from small subgraph to the full graph. Based on the analysis, we propose an efficient two-stage search algorithm KGTuner, which efficiently explores HP configurations on small subgraph at the first stage and transfers the top-performed configurations for fine-tuning on the large full graph at the second stage. Experiments show that our method can consistently find better HPs than the baseline algorithms within the same time budget, which achieves 9.1% average relative improvement for four embedding models on the large-scale KGs in open graph benchmark. Our code is released in https://github. com/AutoML-Research/KGTuner . 1 1 Introduction Knowledge graph (KG) is a special kind of graph structured data to represent knowledge through entities and relations between the entities (Wang et al., 2017; Ji et al., 2021). Learning from KG aims to discover the latent properties from KGs to infer the existence of interactions among entities or the types of entities (Wang et al., 2017; Zhang and Yao, 2022). KG embedding, which encodes entities and relations as low dimensional vectors, is an important technique to learn from KGs (Wang et al., 2017; Ji et al., 2021). The existing models range from translational distance models (Bordes et al., 2013), tensor factorization models (Nickel et al., 2011; Trouillon et al., 2017; Balaevic et al., 2019), neural network models (Dettmers et al., 2017; Guo et al., 2019), to graph neural networks (Schlichtkrull et al., 2018; Vashishth et al., 2020). Hyper-parameter (HP) search (Claesen and De Moor, 2015) is very essential for KG learning. In this work, we take KG embedding methods (Wang et al., 2017), as a good example to study the impact of HPs to KG learning. As studied, the HP configurations greatly influence the model performance (Ruffinelli et al., 2019; Ali et al., 2020). An improper HP configuration will impede the model from stable convergence, while an appropriate one can make considerable promotion to the model performance. Indeed, studying the HP configurations can help us make a more scientific understanding of the contributions made by existing works (Rossi et al., 2021; Sun et al., 2020). In addition, it is also important to search for an optimal HP configuration when adopting KG embedding methods to the real-world applications (Bordes et al., 2014; Zhang et al., 2016; Saxena et al., 2020). Algorithms for HP search on general machine learning problems have been well-developed (Clae-sen and De Moor, 2015). As shown in Figure 1(a), the search algorithm selects a HP configuration from the search space in each iteration, then the evaluation feedback obtained by full model training is used to update the search algorithm. The optimal HP is the one achieving the best performance on validation data in the search process. Representative HP search algorithms are within sample-based methods like grid search, random search (Bergstra and Bengio, 2012), and sequential model-based Bayesian optimization (SMBO) methods like Hyperopt (Bergstra et al., 2013), SMAC (Hutter et al., 2011), Spearmint (Snoek et al., 2012) as well as BORE (Tiao et al., 2021), etc. Recently, some subgraph-based methods (Tu et al., 2019; Wang et al., 2021) are proposed to learn a predictor with configurations efficiently evaluated on small subgraphs The predictor is then transferred to guide HP search on the full graph. However, these methods fail to efficiently search a good configuration of HPs for KG embedding models since the training 2715 ! cost of individual model is high and the correlation of HPs in the huge search space is very complex. To address the limitations of existing HP search algorithms, we carry a comprehensive understanding study on the influence and correlation of HPs as well as their transfer ability from small subgraph to full graph in KG learning. From the aspect of performance, we classify the HPs into four different groups including reduced options , shrunken range , monotonously related and no obvious patterns based on their influence on the performance. By analyzing the validation curvature of these HPs, we find that the space is rather complex such that only tree-based models can approximate it well. In addition, we observe that the consistency between evaluation on small subgraph and that on the full graph is high, while the evaluation cost is significantly smaller on the small subgraph. Above understanding motivates us to reduce the size of search space and design a two-stage search algorithm named as KGTuner. As shown in Figure 1(b), KGTuner explores HP configurations in the shrunken and decoupled space with the search algorithm RF+BORE (Tiao et al., 2021) on a subgraph in the first stage, where the evaluation cost of HPs are small. Then in the second stage, the configurations achieving the top10 performance at the first stage are equipped with large batch size and dimension size for fine-tuning on the full graph. Within the same time budget, KGTuner can consistently search better configurations than the baseline search algorithms for seven KG embedding models on WN18RR (Dettmers et al., 2017) and FB15k-237 (Toutanova and Chen, 2015). By applying KGTuner to the large-scale benchmarks ogbl-biokg and ogbl-wikikg2 (Hu et al., 2020), the performances of embedding models are improved compared with the reported results on OGB link prediction leaderboard. Besides, we justify the improvement of efficiency via analyzing the design components in KGTuner. We firstly revisit the important and common HPs in KG embedding. Following the general framework (Ruffinelli et al., 2019; Ali et al., 2020), the learning problem can be written as P  =arg min PL ( F (  , P ) , D + , D  )+ r ( P ) , (1) where F is the form of an embedding model with learnable parameters P , D + is the set of positive samples from the training data, D  represents negative samples, and r (  ) is a regularization function. There are four groups of hyper-parameters (Table 1), i.e., the size of negative sampling for D  , the choice of loss function L , the form of regularization r (  ) , and the optimization arg min P . Embedding model. While there are many existing embedding models, we follow (Ruffinelli et al., 2019) to focus on some representative models. They are translational distance models TransE (Bordes et al., 2013) and RotatE (Sun et al., 2019), tensor factorization models RESCAL (Nickel et al., 2011), DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2017) and TuckER (Balaevic et al., 2019), and neural network models ConvE (Dettmers et al., 2017). Graph neural networks for KG embedding (Schlichtkrull et al., 2018; Vashishth et al., 2020; Zhang and Yao, 2022) are not studied here for their scalability issues on large-scale KGs (Ji et al., 2021). Negative sampling. Sampling negative triplets is important as only positive triplets are contained in the KGs (Wang et al., 2017). We can pick up m triplets by replacing the head or tail entity with uniform sampling (Bordes et al., 2013) or use a full set of negative triplets. Using the full set can be defined as the 1VsAll (Lacroix et al., 2018) or kVsAll (Dettmers et al., 2017) according to the positive triplets used. The methods (Cai and Wang, 2018; Zhang et al., 2021) requiring additional models for negative sampling are not considered here. Loss function. There are three types of loss functions. One can use margin ranking (MR) loss (Bor-des et al., 2013) to rank the positive triplets higher over the negative ones, or use binary cross entropy (BCE) loss, with variants BCE_mean, BCE_adv (Sun et al., 2019) and BCE_sum (Trouillon et al., 2017), to classify the positive and negative triplets as binary classes, or use cross entropy (CE) loss (Lacroix et al., 2018) to classify the positive triplet as the true label over the negative triplets. Regularization. To balance the expressiveness and complexity, and to avoid unbounded embeddings, the regularization techniques can be considered, such as regularizers like Frobenius norm (FRO) (Yang et al., 2015; Trouillon et al., 2017), Nuclear norm (NUC) (Lacroix et al., 2018) as well as DURA (Zhang et al., 2020b), and dropout on the embeddings (Dettmers et al., 2017). Optimization. To optimize the embeddings, important optimization choices include the optimizer, such as SGD, Adam (Kingma and Ba, 2014) and Adagrad (Duchi et al., 2011), learning rate, initializers, batch size, embedding dimension size, and add inverse relation (Lacroix et al., 2018) or not. Denote an instance x = ( x 1 , x 2 . . . , x n ) , which is called an HP configuration, in the search space X . Let F ( P , x ) be an embedding model with model parameters P and HPs x , we define M ( F ( P , x ) , D val ) as the performance measurement (the larger the better) on validation data D val and L ( F ( P , x ) , D tra ) as the loss function (the smaller the better) on training data D tra . We define the problem of HP search for KG embedding models in Definition 1. The objective is to search an optimal configuration x   X such that the embedding model F can achieve the best performance on the validation data D val . Definition 1 (Hyper-parameter search for KG embedding) . The problem of HP search for KG embedding model is formulated as x  = arg max x X M (cid:0) F ( P  , x ) , D val (cid:1) , (2) P  = arg min PL (cid:0) F ( P , x ) , D tra (cid:1) . Definition 1 is a bilevel optimization problem (Colson et al., 2007), which can be solved by many conventional HP search algorithms. The most common and widely used approaches are sample-based methods like grid search and random search (Bergstra and Bengio, 2012), where the HP configurations are independently sampled. To guide the sampling of HP configurations by historical experience, SMBO-based methods (Bergstra et al., 2011; Hutter et al., 2011) learn a surrogate model to select configurations based on the results that have been evaluated. Then, the model parameters P are optimized by minimizing the loss function L on D tra in Eq. (3). The evaluation feedback M of x on the validation data D val is used to update the surrogate. There are three major aspects determining the efficiency of Definition 1: (i) the size of search space X , (ii) the validation curvature of M (  ,  ) 2717 Figure 2: Ranking distribution of selected HPs. in Eq. (2), and (iii) the evaluation cost in solving arg min PL in Eq. (3). However, the existing methods (Ruffinelli et al., 2019; Ali et al., 2020) directly search on a huge space with commonly used surrogate models and slow evaluation feedback from the full KG due to the lack of understanding on the search problem, leading to low efficiency. To address the mentioned limitations, we measure the significance and correlation of each HP to determine the feasibility of the search space X in Section 4.1. In Section 4.2, we visualize the HPs that determine the curvature of Eq. (2). To reduce the evaluation cost in Eq. (3), we analyze the approximation methods in Section 4.3. Following (Ruffinelli et al., 2019), the experiments run on the seven embedding models in Section 2 and two widely used datasets WN18RR (Dettmers et al., 2017) and FB15k-237 (Toutanova and Chen, 2015). The experiments are implemented with PyTorch framework (Paszke et al., 2017), on a machine with two Intel Xeon 6230R CPUs and eight RTX 3090 GPUs with 24 GB memories each. We provide the implementation details in the Appendix D.1. Considering such large amount of HP configurations in X , we take the simple and efficient approach where HPs are evaluated under control variate (Hutter et al., 2014; You et al., 2020), which varies the i -th HP while fixing the other HPs. First, we discretize the continuous HPs according to their ranges. Then the feasibility of the search space X is analyzed by checking the ranking distribution and consistency of individual HPs. These can help us shrink and decouple the search space. The detailed setting for this part is in the Appendix B.1. Ranking distribution. To shrink the search space, we use the ranking distribution to indicate what HP values perform consistently. Given an anchor configuration x , we obtain the ranking of different values   X i by fixing the other HPs, where X i is the range of the i -th HP. The ranking distribution is then collected over the different anchor configurations in X i , different models and datasets. According to the violin plots of ranking distribution shown in Figure 2, the HPs can be classified into four groups: (a) reduced options , e.g., Adam is the best optimizer and inverse relation should not be introduced; (b) shrunken range , e.g., learning rate, reg. weight and dropout rate are better in certain ranges; (c) monotonously related : e.g., larger batch size and dimension size tend to be better; (d) no obvious patterns : e.g., the remaining HPs. Consistency. To decouple the search space, we measure the consistency of configurations' rankings when only a specific HP changes. For the i -th HP, if the ranking of configurations' performance is consistent with different values of   X i , we can decouple the search procedure of the i -th HP with the others. We measure such consistency with the spearman's ranking correlation coefficient ( SRCC ) (Schober et al., 2018). Given a value   X i , we obtain the ranking r ( x ,  ) of the anchor configurations x  X i by fixing the i -th HP as  . Then, the SRCC between the two HP values  1 ,  2  X i is computed as 1  (cid:80) x X i | r ( x ,  1 )  r ( x ,  2 ) | 2 |X i |  ( |X i | 2  1) , (4) where |X i | means the number of anchor configurations in X i . SRCC indicates the matching rate of rankings for the anchor configurations in X i with respect to x i =  1 and x i =  2 . Then the consistency of the i -th HP is evaluated by averaging the SRCC over the different pairs of (  1 ,  2 ) for X i , the different models and different datasets. The larger consistency (in the range [  1 , 1] ) indicates that changing the value of the i -th HP does not influence much on the other configurations' ranking. As in Figure 4, the batch size and dimension size show higher consistency than the other HPs. Hence, the evaluation of the configurations can be consistent with different choices of the two HPs. This indicates that we can decouple the search of batch size and dimension size with the other HPs. We analyze the curvature of the validation performance M (  ,  ) w.r.t x  X . Specifically, we follow (Li et al., 2017) to visualize the validation loss' landscape by uniformly varying the numerical HPs in two directions (20 configurations in each direction) on the model ComplEx and dataset WN18RR. From Figure 3(a), we observe that the curvature is quite complex with many local maximum areas. To gain insights from evaluating these configurations and guide the next configuration sampling, we learn a surrogate model as the predictor to approximate the validation curvature. The curvatures of three common surrogates, i.e., Gaussian process (GP) (Williams and Rasmussen, 1995), multi-layer perceptron (MLP) (Gardner and Dorling, 1998) and random forest (RF) (Breiman, 2001), are in Figures 3(b)-3(d). The surrogate models are trained with the evaluations of 100 random configurations in the search space. As shown, both GP and MLP fail to capture the complex local surface in Figure 3(a) as they tend to learn a flat and smooth distribution in the search space. In comparison, RF is better in capturing the local distributions. Hence, we regard RF as a better choice in the search space. A more detailed comparison on the approximation ability of different surrogates is in the Appendix B.3. The evaluation cost of the HP configuration on an embedding model is the majority computation cost in HP search. Thus, we firstly evaluate the HPs that have influence on the evaluation cost, including batch size, dimension size, number of negative samples loss function and regularizer. Then, we analyze the evaluation transfer ability from small subgraph to the full graph. Cost of different HPs. The cost of each HP value   X i is averaged over the different anchor configurations in X i , different models and datasets. For fair comparsion, the time cost is counted per thousand iterations. We find that the evaluation cost increases significantly with larger batch size and dimension size, while the number of negative samples and the choice of loss function or regularizer do not have much influence on the cost. We provide two exemplar curves in Figure 5 and put the remaining results in the Appendix B.4. Transfer ability of subgraphs. Subgraphs can efficiently approximate the properties of the full graph (Hamilton et al., 2017; Teru et al., 2020). We evaluate the impact of subgraph sampling on HP search by checking the consistency between evaluations results on small subgraph and those on the full graph. First, we study how to sample subgraphs. There are several approaches to sample small subgraphs from a large graph (Leskovec and Faloutsos, 2006). We compare four representative approaches in Figure 6, i.e., Pagerank node sampling (Pagerank), random edge sampling (Random Edge), single-start random walk (Single-RW) and multi-start random 2719 walk (Multi-RW). For a fair comparison, we constrain the subgraphs with about 20% of the full graph. The consistency between the sampled subgraph with the full graph is evaluated by the SRCC in (4). We observe that multi-start random walk is the best among the different sampling methods. Apart from directly transferring the evaluation from subgraph to full graph, we can alternatively train a predictor with observations on subgraphs and then transfers the model to predict the configuration performance on the full graph. From Figure 6, we find that directly transferring evaluations from subgraphs to the full graph is much better than transferring the predictor model. In addition, we show the consistency and cost in terms of different subgraph sizes (percentage of entities compared to the full graph) in Figure 7. As shown, evaluation on subgraphs can significantly improve the efficiency. When the scale increases, the consistency increases but the cost also increases. To balance the consistency and cost, the subgraphs with 20% entities are the better choices. By analyzing the ranking distribution and consistency of HPs in Section 4.1, we observe that not all the HP values are equivalently good, and some HPs can be decoupled. These observations motivate us to revise the search space in Section 5.1. Based on the analysis in Section 4.2 and 4.3, we then propose an efficient two-stage search algorithm in Section 5.2. To shrink the search space, we mainly consider groups (a) and (b) of HPs in Section 4.1. From the full results in the Appendix B.2, we observe that Adam can consistently perform better than the other two optimizers, the learning rate is better in the range of [10  4 , 10  1 ] , the regularization weight is better in [10  8 , 10  2 ] , dropout rate is better in [0 , 0 . 3] , and add inverse relation is not a good choice. To decouple the search space, we consider batch size and dimension size that have larger consistency values than the other HPs, and are monotonously related to the performance as in group (c). However, the computation costs of batch size and dimension size increase prominently as shown in Figure 5. Hence, we can set batch size as 128 and dimension size as 100 to search the other HPs with low evaluation cost and increase their values in a fine-tuning stage. Given the full search space X , we denote the shrunken space as XS and the further decoupled space as X S|D . We achieve hundreds of times size reduction from XS to X S|D and we show the details of changes in the Appendix C. 5.2 Two-stage search algorithm (KGTuner) As discussed in Section 4.3, the evaluation cost can be significantly reduced with small batch size, dimension size and subgraph. This motivates us to design a two-stage search algorithm, named KGTuner, as in Figure 1(b) and Algorithm 1. Algorithm 1 KGTuner: two-stage search algorithm Require: KG embedding model F , dataset D , and budget B ; 1: shrink the search space X to XS and decouple XS to X S|D ; # state one : efficient evaluation on subgraph 2: sample a subgraph (with 20% entities) G from D tra by multi-start random walk; 3: repeat 4: sample a configuration  x from XS | D by RF+BORE; 5: evaluate  x on the subgraph G to get the performance; 6: update the RF with record (cid:0)  x , M ( F ( P  ,  x ) , G val ) (cid:1) ; 7: until B / 2 budget exhausted; 8: save the top10 configurations in X  S|D ; # state two : fine-tune the top configurations 9: increase the batch/dimension size in X  S|D to get  X  ; 10: set y  = 0 and re-initialize the RF surrogate; 11: repeat 12: select a configuration  x  from  X  by RF+BORE; 13: evaluate on full graph G to get the performance; 14: update the RF with record (cid:0)  x  , M ( F ( P  ,  x  ) ,D val ) (cid:1) ; 15: if M ( F ( P  ,  x  ) , D val ) > y  then y   M ( F ( P  ,  x  ) , D val ) and x    x  ; end if 16: until the remaining B / 2 budget exhausted; 17: return x  . In the first stage, we sample a subgraph G with 20% entities from the full graph D tra by multi-start random walk. Based on the understanding of curvature in Section 4.2, we use the surrogate model random forest (RF) under the state-of-the art framework BORE (Tiao et al., 2021), denoted as RF+BORE, to explore HPs in X S|D on the subgraph G in steps 3-7. The top10 configurations evaluated in this stage are saved in a set X  S|D . In the second stage, we increase batch size and dimension size for configurations in X  S|D to generate a new set  X  . Then, the configurations in  X  are searched by the RF+BORE again in steps 11-16 until the remaining B / 2 budget is exhausted. Finally, the configuration x  achieving the best performance on the full validation data D val is returned for testing. e now summarize the main differences of KGTuner with the existing HP search algorithms, i.e. Random (random search) (Bergstra and Bengio, 2012), Hyperopt (Bergstra et al., 2013), SMAC (Hutter et al., 2011), RF+BORE (Tiao et al., 2021), and AutoNE (Tu et al., 2019). The comparison is based on three aspects, i.e., search space, surrogate model and fast evaluation, in Table 2. KGTuner shrinks and decouples the search space based on the understanding of HPs' properties, and uses the surrogate RF based on the understanding on validation curvature. The fast evaluation on subgraph in KGTuner selects the top10 configurations to directly transfer for fine-tuning, while AutoNE (Tu et al., 2019) just uses fast evaluation on subgraphs to train the surrogate model, and transfers the surrogate model for HP search on the full graph. In Figure 6, the transfer ability of the surrogate model is shown to be much worse than direct transferring. In this part, we compare the proposed algorithm KGTuner with six HP search algorithms in Table 2. For AutoNE, we allocate half budget for it to search on the subgraph and another half budget on the full graph with the transferred surrogate model. The baselines search in the full search space (in Table 1) with the same amount of budget as KGTuner. We use the mean reciprocal ranking (MRR, the larger the better) (Bordes et al., 2013) to indicate the performance. Efficiency. We compare the different search algorithms in Figure 8 on an in-sample dataset WN18RR and an out-of-sample dataset ogbl-biokg. The time budget we set for WN18RR is one day's clock time, while that for ogbl-biokg is two days' clock time. For each dataset we show two kinds of figures. First, the best performance achieved along the clock time in one experiment on a specific model ComplEx. Second, we plot the the ranking of each algorithm averaged over all the models and datasets. Since AutoNE and KGTuner run on the subgraphs in the first stage, the starting points of them locate after 12 hours. The starting point of KGTuner is a bit later than AutoNE since it constrains to use large batch size and dimension size in the second stage, which is more expensive. As shown, random search is the worst. SMAC and RF+BORE achieve better performance than Hyperopt and Ax since RF can fit the space better than TPE and GP as in Section 4.2. Due to the weak transfer ability of the predictor (see Figure 6) and the weak approximation ability of GP (see Fig-2721 ure 3), AutoNE also performs bad. KGTuner is much better than all the baselines. We show the full search process of the two-stage algorithms AutoNE and KGTuner on WN18RR in Figure 9(a). By exploring sufficient number of configurations in the first stage, the configurations fine-tuned in the second stage can consistently achieve the best performance. Effectiveness. For WN18RR and FB15k-237, we provide the reproduced results on TransE, ComplEx and ConvE with the original HPs, HPs searched by LibKGE and HPs searched by KGTuner in Table 3. The full results on the remaining four embedding models RotatE, RESCAL, DistMult and TuckER are in the Appendix D.2. Overall, KGTuner achieves better performance compared with both the original reported results and the reproduced results in (Ruffinelli et al., 2019). We observe that the tensor factorization models such as RESCAL, ComplEx and TuckER have better performance than the translational distance models TransE, RotatE and neural network model ConvE. This conforms with the theoretical analysis that tensor factorization models are more expressive (Wang et al., 2018). To further demonstrate the advantage of KGTuner, we apply it to the Open Graph Benchmark (OGB) (Hu et al., 2020), which is a collection of realistic and large-scale benchmark datasets for machine learning on graphs. Many embedding models have been tested there by two large-scale KGs for link prediction, i.e., ogbl-biokg and ogbl-wikikg2. Due to their scale, the evaluation cost of a HP configuration is very expensive. We use KGTuner to search HPs for embedding models, i.e., TransE, RotatE, DistMult, ComplEx and AutoSF (Zhang et al., 2020a), on OGB. Since the computation costs of the two datasets are much higher, we set the time budget as 2 days for ogbl-biokg and 5 days for ogbl-wikikg2. All the embedding models evaluated here are constrained to have the same (or lower) number of model parameters 2 . More details on model parameters, standard derivation, and validation performance are in the Appendix D.3. As shown in Table 4, KGTuner consistently improves the performance of the four embedding models with the same or fewer parameters compared with the results on the OGB board. In this subsection, we probe into how important and sensitive the various components of KGTuner are. Space comparison. To demonstrate the effectiveness gained by shrinking and decoupling the search space, we compare the following variants: (i) RF+BORE on the full space X ; (ii) RF+BORE on the shrunken space XS ; (iii) RF+BORE on the decoupled space X S|D , which differs from KGTuner by searching on the full graph in the first stage; and (iv) KGTuner in Algorithm 1. All the variants, i.e., RF+BORE, have one day's time budget. As in Figure 9(b), the size of search space matters for the search efficiency. The three components, i.e., space shrinkage, space decoupling, and fast evaluation on subgraph, are all important to the success of KGTuner. Size of subgraphs. We show the influence of subgraph sizes with different ratios of entities (10%, 20%, 30%, 40%, 50%) from the full graph in Fig-2 We run all models on ogbl-wikikg2 with 100 dimension size to avoid out-of-memory, instead of 500 on OGB board. Model ComplEx and dataset WN18RR are used in these experiments. ure 9(c). Using subgraphs with too large or too small size is not guaranteed to find good configurations. Based on the understanding in Figure 7, the subgraph with small size have poor transfer ability and those with large size are expensive to evaluate. Hence, we should balance the transfer ability and evaluation cost by sampling subgraphs with 20%  30% entities. Budget allocation. In Algorithm 1, we allocate B / 2 budget for both the first and second stage. Here, we show the performance of different allocation ratios, i.e., B / 4 , B / 2 , and 3 B / 4 in the first stage and the remaining budget in the second stage. As in Figure 9(d), allocating too many or too few budgets to the first stage is not good. It either fails to explore sufficient configurations in the first stage or only fine-tunes a few configurations in the second stage. Allocating the same budget to the two stages is in a better trade-off. In analyzing the performance of KG embedding models, Ruffinelli et al. (2019) pointed out that the earlier works in KG embedding only search HPs in small grids. By searching hundreds of HPs in a unified framework, the reproduced performance can be significantly improved. Similarly, Ali et al. (2020) proposed another unified framework to evaluate different models. Rossi et al. (2021) evaluated 16 different models and analyzed their properties on different datasets. All of these works emphasize the importance of HP search, but none of them provide efficient algorithms to search HPs for KG learning. AutoSF (Zhang et al., 2020a) evaluates the bilinear scoring functions and set up a search problem to design bilinear scoring functions, which can be complementary to KGTuner. Understanding the HPs in a large search space is non-trivial since many HPs only have moderate impact on the model performance (Ruffinelli et al., 2019) and jointly evaluating them requires a large number of experiments (Fawcett and Hoos, 2016; Probst et al., 2019). Considering the huge amount of HP configurations (with 10 5 categorical choices and 5 continuous values), it is extremely expensive to exhaustively evaluate most of them. Hence, we adopt control variate experiments to efficiently evaluate HPs' properties instead of the quasi-random search in (Ruffinelli et al., 2019; Ali et al., 2020). Technically, we are similar to AutoNE (Tu et al., 2019) and e-AutoGR (Wang et al., 2021) by leveraging subgraphs to improve search efficiency on graph learning. Since they do not target at KG embedding methods, directly adopt them is not a good choice. Besides, based on the understanding in this paper, we demonstrate that transferring the surrogate model from subgraph evaluation to the full graph is inferior to directly transferring the top configurations for KG embedding models. In this paper, we analyze the HPs' properties in KG embedding models with search space size, validation curvature and evaluation cost. Based on the observations, we propose an efficient search algorithm KGTuner that efficiently explores configurations in a reduced space on small subgraph and then fine-tunes the top configurations with increased batch size and dimension size on the full graph. Empirical evaluations show that KGTuner is robuster and more efficient than the existing HP search algorithms and achieves competing performance on large-scale KGs in open graph benchmarks. In the future work, we will understand the HPs in graph neural network based models and apply KGTuner on them to solve the scaling limitations in HP search. This work was supported in part by The National Key Research and Development Program of China under grant 2020AAA0106000.
abstain objective objective result other abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result result objective abstain abstain abstain abstain abstain method method other other other method other other other other method other other other other other other other other other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other other other other other other other other method abstain other objective method objective abstain objective other