A novel pseudo‑random number generator for IoT based on a coupled map lattice system using the generalised symmetric map

Pseudo-random number generators (PRNGs) are one of the building blocks of cryptographic methods and therefore, new and improved PRNGs are continuously developed. In this study, a novel method to generate pseudo-random sequences using coupled map lattices is presented. Chaotic maps only show their chaotic behaviour for a specified range of control parameters, what can restrict their application in cryptography. In this work, generalised symmetric maps with adaptive control parameter are presented. This novel idea allows the user to choose any symmetric chaotic map, while ensuring that the output is a stream of independent and random sequences. Furthermore, to increase the complexity of the generated sequences, a lattice-based structure where every local map is linked to its neighbouring node via coupling factor has been used. The dynamic behaviour and randomness of the proposed system has been studied using Kolmogorov–Sinai entropy, bifurcation diagrams and the NIST statistical suite for randomness. Experimental results show that the proposed PRNG provides a large key space, generates pseudo-random sequences and is computationally suitable for IoT devices.

Introduction
Computers have evolved from large calculating machines to smart handheld devices that have revolutionized human lives [1]. The concept of Internet of Things (IoT) has dramatically changed the way things used to work, since smart computers are replacing humans in the loop. In an IoT framework, large scale decisions are based on the data generated by the IoT sensors; for instance we could cite temperature control in a factory, intruder alarm systems in a smart home or emergency doctor call for patients in smart care between others [2]. Keeping in mind the importance of the IoT data, it is a crucial requirement to ensure data integrity and security [3]. The most common practice for securing data is to deploy cryptographic systems such as private key encryption, public key encryption and digital signatures etc. [4]. One of the building blocks on which cryptographic primitives depend are the seed values or in simple words the random number sequences that would make the keys difficult to predict [5].
Random numbers can be broadly categorized into truerandom numbers (TRNs) [6] and pseudo-random numbers (PRNs) [7]. TRNs are usually considered to be the result of physical processes that produce unpredictable patterns, for instance noise (atmospheric or thermal), and other phenomena like (electromagnetic and quantum) are the sources of true random sequences. Whereas, PRNs are generated via mathematical algorithms that depend on some initial conditions and output a sequence that exhibits similar properties to TRNs [8]. There are several benchmarks available to test and validate the random nature of a number sequence. In literature, several Pseudorandom number generators (PRNGs) have been proposed, and still researchers are coming up with new ideas for lightweight PRNGs that would be a better fit for IoT based applications [9]. The limitations of using PRNGs for IoT data are: (1) the IoT sensor devices are resource constrained and (2) the sensor devices generate large amount of data constantly. Keeping in mind the nature of IoT data, there is a need for PRNGs that could offer wider range of initial seed values to choose from.
Chaos theory is a branch of mathematics that has been extensively studied due to its unique properties of apparent disorder and randomness. The chaotic systems can generate diverging and disorderly sequences, which appear to be random, being in fact deterministic but highly sensitive to initial conditions [10]. These particular characteristics make chaotic systems a good candidate for PRNGs. Chaos based PRNGs have been an active area of research but unfortunately, relatively few chaotic maps have been explored for this purpose [11]. In this paper, a family of symmetric chaotic maps have been considered as the local maps for coupled map lattices (CMLs) and an adaptive approach is proposed for control parameter selection. The proposed model introduces additional control parameters which contribute to the generation of highly chaotic sequences with large key space for cryptographic applications.
The remaining paper is organised as follows: The recent literature and related advances will be discussed in Sect. 2. The proposed model and its analysis are presented in Sect. 3. Testing and validation of proposed model for IoT applications is shown in Sect. 4. Section 5 concludes the study and discusses future research aims.

Related work
There is substantial amount of published research on chaos based PRNGs. In this paper, the reviewed literature explores chaos based PRNGs and mainly focuses on CML based PRNGs.
There is a variety of chaotic maps available for pseudo-random number generation. A famous choice of many researchers is to use the logistic map, which is one of the most widely studied chaotic maps [12]. Various versions of modified logistic maps have been used for PRNGs including pseudo-randomly enhanced logistic map (PELM) [13], floating point based modified logistic chaotic system [14], hyper-chaotic modified robust logistic map (HC-MRLM) [15], modified logistic map for increased key-space range [16], nonlinear digitalized modified logistic map [17], and optimised logistic map using perturbation operation [18]. The logistic map exhibits chaotic behaviour for a very small range of control parameter values between [3.57, 4]. However, there have been studies in attempt to increase the parameter range for the chaotic behaviour in logistic map. In a recent study on pseudo-random bit generator, logistic maps were used to generate multimodal maps and the parameter interval showed an increase in the chaotic range [19]. In a few other studies, researchers have used different chaotic maps such as the Lorenz map [20], the Tinkerbell map [21] and the Chirikov map [22].
Several electronic and hardware based implementations for chaos based PRNG have been presented. Matheus et al. proposed a gate level hardware implementation for a PRNG based on an exponential chaotic map [23]. The hardware implementation was designed using Field Programmable Gate Array (FPGA) device for the proposed PRNG system. There is a significant number of hardware implementations for chaotic based PRNGs using FPGAs including FPGA based PRNG using four-wing memristive hyperchaotic system and bernoulli map [24], reconfigurable chaotic PRNG based on FPGA [25], chaos based bitwise dynamical PRNG on FPGA [26], and FPGA implementation of chaos based PRNG for secure communication [27]. FPGAs are preferred by algorithm designers for prototyping as they can execute the code on gate level using hardware descriptive languages. Since, in real world applications microcontrollers are used more commonly, thus several proposals on chaos based PRNGs have been made using microcontrollers. For instance, an analog circuit and microcontroller based PRNG application of a new easy realizable 4D chaotic system has been recently proposed [28], in another study a chaotic random bit generator has been realized with a microcontroller [29], and hardware implementation of initials-boosted coexisting chaos in a two dimensional sine map [30]. The IoT technology that is rapidly gaining popularity in everyday applications depend on resource constrained devices, and there are not many solutions available for IoT sensors. In a recently published work, the authors claimed to design encryption solution for IoT devices based on novel chaotic map but it had several shortcomings [31].
They decided to use non-chaotic algorithms like PRESENT and LED to encrypt initial conditions, this would add overhead to the remaining chaos based encryption process. The time analysis performed for the proposed algorithm was conducted using Intel(R) Core(TM) i5-4210U 2.40 GHz CPU, 4GB RAM computer which is equipped with strong computing power and still takes minimum 6.7 s for 1000 iterations and 72.5 s for 14,373 iterations. The suggested encryption algorithm would fail completely if it was being deployed on a real IoT temperature sensor, which has a single core processor and sends temperature feed multiple times during one second time. Considering, the importance of IoT data and the challenges in terms of limited power and computing resources, there is a dire need for chaos based cryptosystem that is lightweight in terms of computation and also provide robust security against attacks.
Chaos based PRNGs are used in several security applications, for instance stream ciphers [32], block ciphers [33], secure communications [34], image encryption [35], and video encryption [36] amongst others. Despite their popularity and successful applications, there are several shortcomings that chaos based PRNGs possess. Ramazan Yeniçeri et al. published a study in which they showed how a random number generator based on a time delay differential equation could be attacked [37]. To generate an attack, they predicted the values before hand and coupled them with future states of a time delay based chaotic system. Once the system was synchronized with it, the generated signals were similar to the original ones. In another study, the cryptanalysis of chaos based PRNGs was conducted and several improvements were suggested to rectify system weaknesses [38]. Youling et al. conducted cryptanalysis on chaos based cryptosystems with the aspect of hardware based attacks [39]. They first implemented chaos based cryptosystem onto a microcontroller device and conducted side channel attack analysis and correlation power analysis attack, which can be performed by studying the execution time and power consumption of the cryptographic algorithm. Furthermore, in various other studies, vulnerability and security analysis has been performed for chaos based PRNGs in the context of cryptographic applications. In a recent study, researchers conducted security analysis of the efficient chaos based PRNG for video encryption application [40], and in another study a thorough vulnerability analysis has been performed for chaos based PRNG [41].
A remedy to the weaknesses discussed in chaos based PRNGs could be to use complex dynamical systems to generate choas rather than using simple one (1D) or two dimensional (2D) chaotic maps , this would make the prediction of generated patterns quite strenuous to crack. A multidimensional chaotic system with discrete time and discrete spatial extension was introduced by Kaneko in the 1980's [42]. These type of spatio-temporal chaotic systems are known as CMLs, that produce a wide array of behaviours. CMLs are capable of producing complex chaotic sequences as they form lattice structure which comprises of local chaotic maps. Despite the complex chaotic behaviour, there are some inherent shortcomings in CML systems which limit their application as PRNG. One major drawback in the CML systems is the limited range of parameter space for the spatio-temporal chaos and another limitation is the variation in local chaotic behaviors i.e some lattices might not reach chaotic zone [43]. There have been several improvements to the Kaneko's model of CML system for PRNG applications. Ping et al. recommended a pseudo-random bit generator (PRBG) based on mixing the state variables of a CML to generate more complex pseudo-random sequences [44]. In another study, a variable time delay has been suggested for the CML based PRNG. The results show that the proposed system is not hefty with computations and also passed all randomness tests provided by NIST [45]. Recently, a CML system with asymmetric coupling has been used as PRBG with a sawtooth map as the local map [46]. The authors also suggested the use of specific systemic parameters, that could convert complex floating calculations to bit operations. In [47], the authors proposed a CML based on discrete chaotic iteration (CMLDCI) that combine the CML and chaotic iteration to produce random sequences for PRNG. By chaotic iteration, they refer to a convergence theorem for chaos coined by Devany et al. in early literature [48] .The idea of a Multi Bit Random Number Generator (MBRNG) published recently, proposed a CML based random number generator based on logistic map [49]. The proposed system does not present any novelty, as it uses logistic map as the local map similar to the conventional CML system. In another article, the authors used skew tent map as local map [50]. The idea is further extended to analyse the behaviour of system by cross coupling skew tent maps.
In the light of reviewed literature, majority of the latest research proposals use a very few well known maps like the logistic map [12] or tent map [51] as local maps for the CML systems. Logistic and tent maps belong to the family of symmetric chaotic maps which contain other higher order maps as well. To our best knowledge from the literature review, there has not been any attempt to explore the family of symmetric maps in general for the CMLs. In this paper, we introduce the concept of Generalised Symmetric Map (GSM) as local map for the CML system and also propose the concept of adaptive control parameter values that results in highly chaotic and complex PRNG.

Proposed model
In this paper, a unique PRNG model has been proposed for IoT devices, that generates random sequences with large key space and could be used in various cryptographic applications. The proposed model comprises of a spatiotemporal CML system using GSM as local map. The concept of adaptive values, based on accumulation points of local GSM map has also been introduced. This ensures that the local maps stays in the chaotic range for majority of the time and they outcome pseudo-random sequences for any initial seed values. Detailed working of the proposed system is explained in the subsequent subsections.

Coupled map lattice (CML)
This idea of generating spatiotemporal chaos via CMLs was published in a series of studies during the 1980's ranging from the exploration of spatio-temporal intermittency of coupled map lattices [52] to spatio-temporal coupled nonlinear oscillators [53], and space-time dynamics in video feedback [54], with Kaneko being the most active contributor in this domain [55]. The simplest way to visualise a CML system is to consider the diffusive CML model as shown in Eq. (1) [56]. This is a diffusive two-way CML model, where the parameter i gives the ith lattice point, which in this case is coupled with two of its neighbouring nodes i.e. right node (i + 1) and left node (i − 1) . i varies between [1, L] where L is the total number of lattice points. The coupling factor is represented by , which is a real number in [0, 1]. Finally, n is the discrete time step for the CML system. Here, f(x) denotes the local map, which is usually a logistic or tent map but in the proposed model, f(x) could be any chosen chaotic map from symmetric maps family.

Generalised symmetric map (GSM)
Symmetric maps belong to family of chaotic maps, which when plotted exhibit mirror image of the plot itself (maintain symmetry of its shape). Figure 1a and b show the mapping and psuedo phase trajectory to plot a 2D and 3D image of the tent map. The tent map is known to be a first order symmetric map as it is a straight line map and is symmetric across the x-axis. Figure 2a  where alpha ( ) and beta ( ) are the control parameters for the GSM. ∈ [1,4] represents the map order and is the control parameter that decides chaotic or non-chaotic zone of the chosen map being a real number [0, 1]. The core working behind GSM is to utilise any map from the symmetric chaotic maps using a single mathematical function i.e Eq. (2). This can be observed that by increasing the value of parameter in the range [1,4], transitions the Using GSM as local maps widens the range of maps to choose from, by providing the liberty to choose local map as any symmetric chaotic map. The selection of and the specific choice of corresponding control parameters would become extremely difficult for an adversary to guess the exact order and type of map used.

Adaptive ˇ values
Chaotic maps present chaotic behaviour for specific values of control parameter and initial conditions. The control parameter value at which a 1D map enters in the chaotic zone is known as accumulation point of the map. The accumulation point can be computed via the Lyapunov exponent analysis [57]. The point where the Lyapunov exponent switches from negative to positive values, can be considered as the accumulation point. This can be easily explained by visualizing the Lyapunov spectrum of the Logistic map, as shown in Fig. 5. For the logistic map, is set to 2 in GSM Eq. (2), and the accumulation point (a) can be observed between 0.89 and 0.90 (0.89995 ignoring the high precision). Therefore, the concept of accumulation point allows us to formulate a system, where we built an adaptive scheme  that calculates values for any value of such that they always remain above the accumulation point maintaining a chaotic output for the system. Furthermore, to investigate the trend of accumulation points and control parameter , we calculate Lyapunov spectrum for GSMs by varying the values of in the range [− 4, 4]. Table 1 summarizes the range of values and their respective accumulation points. This can be observed from Table 1 data, that the GSM function is non-chaotic for the interval when is in the range [ − 0.5, 0.5 ]. To visualise the trend of accumulation points for GSM, Table 1 data is interpolated as shown in Fig. 6. As in this study, we are more focused onto exploring symmetric maps and having consistant range of parameter for cryptographic applications, we ignore the negative range of and consider  range between [1,4] which corresponds to symmetric maps of order one to four.
From the above discussion, it can be concluded that there is a higher chance for the system to exhibit chaotic behaviour, if the control parameters are chosen such that which would result in system values to remain above the accumulation points. Therefore, to assure that the system remains in chaotic zone majority of the time, we introduce a concept of adaptive ( ad ) as shown in Eq. (3).
where ad is the adaptive value, calculated for a specific accumulation point. The present lattice number is presented as i. A new parameter Gamma ( ) is introduced to control the spread of ad values across all the lattice points.
is defined as a positive real number i.e. ( > 0 ). The main functionality of Eq. (3) is to keep the ad values always in range above the accumulation point a ≤ ad ≤ 1 . The ad function when supplied with accumulation points for GSM, returns ad values such that, the CML system stays above the accumulation points and therefore exhibits chaotic output depending on the type of local map chosen.

KS entropy analysis
The Lyapunov spectra of a system [58] is an important tool to classify its chaotic behaviour. A CML system with N -dimensional phase space gives rise to N Lyapunov exponents. To calculate the Lyapunov spectra in a CML system, one needs the product of the Jacobian matrices ( J p ) of the map as shown in Eq. (4).
where J n is the individual Jacobian calculated at time step n. J p is the final product of all Jacobian matrices divided by the time step n. Lyapunov exponents i of the CML system can be calculated via Eq. (5). (4) where the logarithms of eigenvalues of J p are retrieved that are finally divided by the time steps n. Once all the Lyapunov exponents are found, the Lyapunov Spectra is defined as the ordered set { 1 ≤ 2 ≤ ⋯ ≤ N } . The sum of all positive Lyapunov exponents leads to the Kolmogorov-Sinai (KS) entropy [59]. KS entropy density h can be represented using Eq. (6).
where i are the Lyapunov exponents calculated in the previous steps. N is the number of lattice points, or dimension of the CML lattice. To further analyse the chaotic behaviour of the CML system based on GSM with ad , KS entropy density analysis has been visualised by varying control parameters. Figure 7 shows the KS entropy density analysis of a system with 10 lattice points, 1000 calculated time steps and 100 discarded iterative values to account for transient behaviour, ranges between [1,4], e spread between [0,1], set to 0.25 and is adaptive. Since is a tuning parameter for ad values, it should be tuned with respect to the system. The value for can be chosen by observing the ad value calculated for each lattice point. In our case we observed that gave a good spread of ad values for ( = 0.25 for N = 10 ) and ( = 0.025 for N = 100). Figure 7 shows the KS entropy density analysis of the CML system with GSM mapping and ad values for 10 lattice points, whereas Fig. 8 shows the result for the system with a 100 lattice points. It can be observed from Fig. 7 that there were a number of regions of ( , ) parameter, space where the entropy density can be assumed to be zero, whereas, in Fig. 8 the system shows a uniformly populated KS-entropy plot with very few regions of low density. Therefore from Figs. 7 and 8, it can be speculated that increasing the number of lattices improves the overall chaotic behaviour of the CML system with GSM and ad .

Bifurcation analysis
Bifurcation diagrams aid in visualising overall perioddoubling progression of a system with increase in control parameters. The KS entropy analysis performed in the previous subsection shows that the coupling factor ( ) had great influence on the overall behaviour of the system. Therefore, bifurcation diagrams of the proposed system for first, middle and last lattice point of the proposed CML system have been visualised, for varying values of (i.e. 0.2, 0.5, and 0.8). The bifurcation diagrams were plotted for increasing values on x-axis in range [1,4], across the values x values generated by CML system on y-axis ranging between [0,1]. Figure 9 shows the bifurcation analysis of the proposed system with 10 lattice points, 1000 initial discarded iterations, 2000 calculated iterations, and set to 0.25. To investigate the lattice behaviour in detail, in Fig. 9 columns present lattice number and rows denote the specific values. The results have been realised for first, middle and last lattice of the proposed system to interpret overall behaviour of the system. The results can be visually interpreted as several periodic regions appearing for at 0.2 and 0.8 whereas, for coupling factor of 0.5, the chaotic pattern is evenly spread. Similar experiments were performed for the lattice number increased to 100 lattice points, 1000 initial discarded iterations, 2000 calculated iterations and value used as 0.025. Figure 10 shows the detailed trend of bifurcation plots for first, fiftieth, hundredth lattice point presented in columns, whereas the rows of Fig. 10 show the increasing values of coupling factor ( = 0.2, 0.5 and 0.8). Overall, the system with 100 lattice points appears more consistently chaotic for all coupling factors as compared to the system with 10 lattice points. Considering KS entropy experiments and bifurcation diagram analysis, this can be deduced that increasing the lattice number of the proposed system results generation of highly chaotic sequences.

Random number generator using proposed system
In this section, we define the steps to create a PRNG based on a CML system with GSM mapping and adaptive . We also tested the proposed method for randomness by benchmarking it to the available standards and applying it to a real world example of IoT sensors. A pseudo random number must hold some properties close to true random numbers. Another requirement for PRNGs used with IoT sensors devices is high throughput and low latency, thus we assumed 64-bit precision representation using fixed point arithmetic. The detailed steps for generating pseudo-random sequence (64-bit) are explained in Algorithm 1. Step 1: Select the sequence length required (Calculated as nxN ).
Set the number of time steps = n Select the number of lattice points = N Step 2: Calculate adaptive β for every lattice point i.
Step 3: Discard t number of iterations to avoid transient response.
Step 4: Iterate the CML system for n number of iterations.

64-bit random number sequence
if sequence length from Step 4 not appropriate then Repeat from Step 1 to Step 5 with new n and N value. else Quit end if

Key space analysis
Random number generators play basic role in cryptographic applications, generating strong encryption and decryption keys. By the term strong, we mean that the keys with higher key space are harder to guess, thus the higher the key space is, the stronger is the cryptographic key. More specifically, a cryptographic key smaller than 128-bit is not considered strong enough to be used for data encryption purposes [60]. The proposed method depends on several control parameters and initial conditions which collectively contribute towards the key space. The experiments were performed using Python 3.7 as programming language with floating point as data type with 53 bits representation (16 digits precision i.e 10 −16 ). Therefore, the key space can be calculated from parameters: The key space for the proposed algorithm is ≃ 2 372 , which is significantly large to survive brute force attacks.

Key randomness analysis
To validate the randomness of the system, a thorough testing has been performed using the Statistical Randomness Test Suite (SP 800-22) made available by National Institute of Standards and Technology (NIST) [63]. The NIST randomness test suite comprises of 15 randomness tests as shown in Table 3. Table 3 also shows the parameters that were used during the randomness testing experiments. To best analyse the behaviour of the system, we performed all 15 randomness tests for varying values of and the . The heat map chart is plotted such that, for every chosen and , the system was tested for the 15 NIST randomness tests, and the number of passed tests is shown in the respective block. The legend on the right side of heat map can also be used to categorize the maximum and minimum number of NIST randomness tests passed using color codes. Considering that could be any number in range [1,4], in between [0,1] and variable can make up to numerous possibilities to generate pseudo-random sequences, but for fair analysis to test randomness of the system, NIST randomness tests have been conducted on increasing values of in range(0.5 ≤ ≤ 4 ) at an interval of 0.25, and selected coupling values ( = 0, 0.1, 0.3, 0.5, 0.7, 0.9, 1). Figure 11 shows the heat map chart for the CML system with 10 lattice points, the overall number of pass tests were more than 10 except for some chunks between 1.75 ≤ ≥ 3.75 . The color legend on right of Fig. 11 shows the number of NIST tests passed out of a total fifteen randomness tests. Whereas, Fig. 12 shows the heat map chart for the proposed system with 100 lattice points. As expected, the overall behaviour of the system with 100 lattice points appears to be better in randomness as compared to the system with 10 lattice points. This can be observed that for majority of and coupling values, the number of NIST tests 'passed' is greater than 10 randomness tests. This can be concluded that increasing the number of lattices, increases the randomness of output generated from CML based PRNG. The use-case considered for the proposed method is for IoT applications, where IoT sensors generate a huge amount of data at high frequency, which needs to be encrypted on the journey to its destination. Encrypting constant streams of data having critical timelines is a very challenging task, as fresh supply of encryption keys are required consistently. The proposed pseudo-random number generator serves the purpose, as the user can generate a wide range of pseudorandom sequences that are independant of any previous sequences. Figures 11 and 12 depict that for majority of the randomness tests, the pass rate is above 10 tests which means the sequences generated are fairly pseudorandom in nature. Thus, unlimited number of keys can be confidently generated for different devices and sessions, slightly varying the control parameters for the proposed system generates new pseudo-random sequences that are different and independant from the previously generated sequences.

Performance analysis on IoT devices
The ubiquity of IoT devices is increasing and thus it is crucial to secure the data generated by such hardware. The limitation that hinders the deployment of security primitives on IoT devices is that they are scarce in terms of computation, memory and power resources. Thus, when designing the security algorithms, one should keep the resource limitations of IoT devices under consideration. Even though, IoT devices have evolved and sensors currently available on the market are quite efficient and reliable. To test the proposed algorithm, we chose highly resource constrained systems on chip, Raspberry pi zero (single-core processor) and Raspberry pi 3B+ (quad-core processor). Table 4 summarises the execution time of the proposed PRNG on both sensor devices with increasing number of iterations and lattice points. It should be noted that increasing lattice number and number of iterations together has double impact on the computation time but still the proposed PRNG is capable of generating pseudorandom sequences in less than a second.

Comparison with existing techniques
One important stage in proposal of new algorithm is to perform a comparative analysis of proposed method with existing state of the art techniques. This type of comparison allows the researchers to deduce a fair opinion on what evaluation matrices have been considered during the proposal of new algorithm. Table 5 shows the summary of comparison performed between proposed pseudo-random number generator with the existing PRNG techniques based on CML systems. This can be observed from Table 5, that in majority of existing techniques, the evaluation of proposed ideas has not been performed using all standard measures. In fact, only some of the evaluation techniques have been considered for validation of proposed method. In this study, for the proposed CML based PRNG, we realised detailed proposal of GSM using CML and visualised their pseudo phase trajectories and Lyapunov spectrum. To analyse the entropy analysis of the system, KS entropy plots were created. Furthermore, for detailed examination of lattice behaviour, bifurcation diagrams were plotted for first, middle and last lattice point of the proposed system. In other existing techniques, only few studies like [44] have used KS entropy or bifurcation analysis to study their proposed CML systems. Whilst designing a PRNG for cryposystem, it is crucial to consider the key space and execution speed of the proposed algorithm. Again a very few researchers consider validating their proposed algorithm against the key space size. Majority of the discussed PRNG schemes in Table 5 in have not used keyspace analysis to measure the strength of their proposed algorithm in terms of security robustness. Most important test for a new PRNG algorithm is to test for the degree of randomness using standard benchmarks. There are variety of randomness tests available but the standard test recognised by research community is NIST   [63]. In Table 5, this can be observed that almost all the discussed existing PRNG techniques have been tested using NIST randomness test suite. Few studies like [45] have tested their algorithm against other randomness testing benchmarks such as TESTU01 [64]. This can be deduced from the comparative analysis that the proposed method has been evaluated on the basis of standard benchmarking and testing standards available.

Conclusion
In this study, generalised symmetric chaotic maps have been explored. A method to generalise the symmetric maps has been introduced, which allows different chaotic maps to be chosen by changing single a parameter . Chaotic maps do not always generate chaotic behaviour, rather they show such behaviour only in a specific range of control parameters and initial conditions. We proposed the concept of adaptive control parameter ad , which based on the accumulation points, strongly increases the likelihood of chaotic behaviour of the map, irrespective of what value is chosen. A tuning parameter has also been proposed, which helps in manually adjusting the spread of values. The additional control parameters contribute towards large key space for PRNG. The KS entropy analysis, bifurcation diagrams and randomness tests show that the proposed system tends to show random behaviour for majority of values with exception to few periodic chunks.
The proposed PRNG has also been tested on real time IoT sensors to validate the computational efficiency. It is concluded that the proposed PRNG system can generate cryptographically strong keys, and is computationally suitable for lightweight devices such as IoT sensors. In future, this work would be extended to develop novel cryptosystems, such as image and video encryption schems.

Author Contributions All authors contributed equally.
Funding This research is supported by the BTIIC (British Telecom Ireland Innovation Centre) project, funded by British Telecom and Invest Northern Ireland.

Availability of data and materials
The data supporting the findings of this study is included in this article.

Code availability
The softwares and tools used in this study are open source and available online.

Conflict of interest
The authors have no conflicts of interest to declare. All co-authors have seen and agreed with the contents of the manuscript. We certify that the submission is original work and is not under review at any other publication. Ref. [44] Ref. [45] Ref. [46] Ref. [47] Ref. [49] Ref. [50] Chaos Ethics approval This article does not contain any studies with human participants or animals per-formed by any of the authors.

Consent for publication Not applicable.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.


Towards One Shot Search Space Poisoning in Neural Architecture Search

We evaluate the robustness of a Neural Architecture Search (NAS) algorithm known as Efficient NAS (ENAS) against data agnostic poisoning attacks on the original search space with carefully designed ineffective operations. We empirically demonstrate how our one shot search space poisoning approach exploits design flaws in the ENAS controller to degrade predictive performance on classification tasks. With just two poisoning operations injected into the search space, we inflate prediction error rates for child networks upto 90% on the CIFAR-10 dataset.

Introduction
The problem of finding optimal deep learning architectures has recently been automated by neural architecture search (NAS) algorithms. These algorithms continually sample operations from a predefined search space to construct neural networks to optimize a performance metric over time, eventually converging to better child architectures. This intuitive idea greatly reduces human intervention by restricting human bias in architecture engineering to just the selection of the predefined search space (Elsken et al. 2019). Although NAS has the potential to revolutionize architecture search across many applications, human selection of the search space remains a security risk that needs to be evaluated before NAS can be deployed in security-critical domains. While NAS has been studied to further develop more adversarially robust networks through addition of dense connections (Guo et al. 2020), little work has been done in the past to assess the adversarial robustness of NAS itself. Search phase analysis has shown that computationally efficient algorithms such as ENAS are worse at truly ranking child networks due to their reliance on weight sharing (Yu et al. 2019). Finally, most traditional poisoning attacks involve injecting mislabeled examples in the training data and have been executed against classical machine learning approaches (Schwarzschild et al. 2021). We validate these concerns by evaluating the robustness of one such NAS algorithm known as Efficient NAS (ENAS) (Pham et al. 2018) against data-agnostic search space poisoning (SSP) attacks on the CIFAR-10 dataset. Throughout this paper, we focus on the pre-optimized ENAS search spaceŜ = {Identity, 3x3 Separable Convolution, 5x5 Separable Convolution, Max Pooling (3x3), Average Pooling (3x3)} (Pham et al. 2018).

Search Space Poisoning (SSP)
The idea behind SSP, as shown in Figure 1, is to inject precisely designed multiset P of ineffective operations into the ENAS search space, making the search space S :=Ŝ ∪ P. Our approach exploits the core functionality of the ENAS controller to sample child networks from a large computational graph of operations by introducing highly ineffective local operations into the search space. On the attacker's behalf, this requires no a priori knowledge of the problem domain or dataset being used, making this new approach more favourable than traditional data poisoning attacks.

Multiple-Instance Poisoning
As a naïve strategy, we first propose multiple-instance poisoning which increases the likelihood of sampling bad operations by including duplicates of these bad operations in the search spaces. Through experimental results we discovered that biasing the search space this way resulted in final networks that are mostly comprised of these poor operations with error rates exceeding 80%. However, as shown in Figure 2, to perform well this approach requires overwhelming the original search space with up to 300 bad operations (50:1 ratio of bad operations per each good operation) which is unreasonable. The motivation then is to reduce the ratio of bad to good operations down to 1:1, or even lower, to make search space poisoning more feasible and effective.

Towards One Shot Poisoning
In an attempt to improve the attack, we further attempted to reduce the number of poisoning points to just 2 points by adding: (i) Dropout(p = 1) (ii) Stretched Conv(k = 3, padding, dilation = 50) to the original search space. Our rationale is that dropout operations with p = 1 would erase all information and produce catastrophic values such as 0 or not-a-number (NaN). The results were promising, with error rates shooting up to 90% very quickly during training as seen in Figure 3 and Table 1. An example final child network producing these high errors can be observed in Figure 4.   Figure 4: Network produced by ENAS after one shot poisoning with good operations highlighted in green and poisoning operations highlighted in red. Search space utilized is the same as shown in Table 1 with two poisoning points.

Conclusion
In this paper, we focused on examining the robustness of ENAS under our newly proposed SSP paradigm. Our carefully designed poisoning sets demonstrated the potential to make it incredibly easy for an attacker with no prior knowledge or access to the training data to still drastically impact the quality of child networks. Finally, our one-shot poisoning results reveal an opportunity for future work in neural architecture design, as well as challenges to surmount in using NAS for more adversarially robust architecture.

Supplementary Material
Search Space Poisoning (SSP)

Multiple-Instance Poisoning Attacks
Over the course of training, the LSTM controller paired with the RL search strategy in ENAS develops the ability to sample architectures with operations that most optimally reduce the validation error. We propose multiple-instance poisoning which essentially increases the likelihood of a poisonous operation o P being sampled from the poisoned search space S. This is achieved by increasing the frequency of sampling o P from S through the inclusion of multiple-instances of each o P from the poisoning multiset, so-called to allow for duplicate elements. An instance factor q ∈ N ≥1 represents instance multiplication of o P in the multiset q times. Henceforth, the probabilities of sampling oŜ ∈Ŝ and o P ∈ P, respectively, are, P r[oŜ] := 1 |S|+q|P| and P r[o P ] := q |S|+q|P| . From which, it is evident that under a multiple-instance poisoning framework, the probability of sampling poisoned operations is strictly greater than sampling operations inŜ; that is, P r[oŜ] < P r[o P ]. It is also important to note that our technique is not the typical image-agnostic perturbation/universal adversarial perturbation δ ∈ R d intended to fool the target neural network f on almost all the input images from the target distribution X. Finally, another challenge to overcome within our search space poisoning framework is to craft each o P ∈ P such that it counteracts the efficacy of the original operations oŜ ∈Ŝ, which we tackle in the next section.

Crafting Poisoning Sets with Operations
Identity Operation The simplest way to attack the functionality of ENAS is to inject non-operations within the original search space which keep the input and outputs intact. As a result, the controller will sample child models with layers representing computations which preserve the inputs, making the operation highly ineffective within a network architecture. This goal is achieved by the inexpensive identity operation which has no numerical effect on the inputs. It should also be noted that, the identity operation is not a skip connection. Therefore, we define our first poisoning set: P 1 := {Identity}.
Gaussian Noise Layer Typically used in signal processing, electronics and mitigating over-fitting as some form of random data augmentation. Gaussian noise is a type of statistical noise which is in the form of a Normal distribution (X ∼ N (µ, σ 2 )). In PyTorch/Keras, the Gaussian noise layer additively applies zero-centered Gaussian noise passing in the argument of relative standard deviation used to generate the noise. We hypothesize that including such layers with increasingly varied relative standard deviations such as σ = 10, can significantly impact the accuracy of the generated child models making our poisoning set P 2 := {Gaussian (σ = 10)}.
Dropout Layer While dropout layers have historically been shown to be useful in preventing neural networks from over-fitting (Srivastava et al. 2014), a high dropout rate can result in severe information loss leading to poor performance of the overall network. This is because given a dropout probability p ∈ [0, 1], dropout randomly zeroes out some values from the input to decorrelate neurons during training. We hypothesize that including such layers with high dropout probability, such as p = 1, has the potential to contaminate the search space with irreversible effects on the training dynamics of ENAS. Here, we define our poisoning set as P 3 := {Dropout (p = 1)}.
Transposed Convolutions As described earlier, amongst other useful operations the original ENAS search spaceŜ also contains 3x3 and 5x5 convolutional layers (separable). Intuitively, transposed convolutions upsample the input feature map. It is important to note that transposed convolutions do not perform like deconvolutional layers; they actually swap the forward and backward passes of a convolution. Transposed convolutions, also known as fractionally strided convolutions, stride over the output which is equivalent to a fractional stride over the input. We define our poisoning set: P 4 = {3x3 transposed convolution, 5x5 transposed convo-lution}.

Experiments
Experimental Setup To test the effectiveness of our proposed approach, we designed experiments based on previously described methods outlined in Table 1. Each experiment involved training ENAS on the CIFAR-10 dataset for 300 epochs. The results presented in this paper are the average of three runs per experiment.
For a baseline, we created an experiment with the original search spaceŜ (which was run three times). Because we wanted to evaluate the poisoning effectiveness of each operation independently, we created a group of four experiments for each poisoning set P i . We henceforth refer to these experiments as "Space ia" through "Space id". Following multiple-instance poisoning attacks, each experiment includes a poisoning multiset q(P i ) where |q(P i )| = 6, 36, 120, 300 for Space ia, Space ib, Space ic, Space id, respectively. Of note is that the instance-factor q is dependent on the experiment index and |P i |. For example: Space 1c/2c/3c has |P 1 | = |P 2 | = |P 3 | = 1, and q|P 1 | = q|P 2 | = q|P 3 | = 120, so q = 120; Space 4c has |P 4 | = 2 and q|P 4 | = 120, so q = 60; Space 5c/6c have |P 5 | = |P 6 | = 6 and q|P 5 | = q|P 6 | = 120, so q = 20.

Our Files
• The files in which these extended search spaces can be found are in enas poisoning/. • Initial experimental results can be found in ENAS-Experiments/results.


extfieldsofstudy:
Computer Science
Medicine

text:
Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019

This paper reports the results and post-challenge analyses of ChaLearn's AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been introduced in a variety of settings, but lacked fair comparisons. All input data modalities (time series, images, videos, text, tabular) were formatted as tensors and all tasks were multi-label classification problems. Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly. In this setting, DL methods dominated, though popular Neural Architecture Search (NAS) was impractical. Solutions relied on fine-tuned pre-trained networks, with architectures matching data modality. Post-challenge tests did not reveal improvements beyond the imposed time limit. While no component is particularly original or novel, a high level modular organization emerged featuring a"meta-learner","data ingestor","model selector","model/learner", and"evaluator". This modularity enabled ablation studies, which revealed the importance of (off-platform) meta-learning, ensembling, and efficient data management. Experiments on heterogeneous module combinations further confirm the (local) optimality of the winning solutions. Our challenge legacy includes an ever-lasting benchmark (http://autodl.chalearn.org), the open-sourced code of the winners, and a free"AutoDL self-service".

INTRODUCTION
The AutoML problem asks whether one could have a single algorithm (an AutoML algorithm) that can perform learning on a large spectrum of tasks with consistently good performance, removing the need for human expertise (in defiance of "No Free Lunch" theorems [1], [2], [3]). Our goal is to evaluate and foster the improvement of methods that solve the AutoML problem, emphasizing Deep Learning approaches. To that end, we organized in 2019 the Automated Deep Learning (AutoDL) challenge series [4], which provides a reusable benchmark suite. Such challenges encompass a variety of domains in which Deep Learning has been successful: computer vision, natural language processing, speech recognition, as well as classic tabular data (feature-vector representation).
AutoML is crucial to accelerate data science and reduce the need for data scientists and machine learning experts. For this reason, much effort has been drawn towards achieving true AutoML, both in academia and the private sector. In academia, AutoML challenges [5] have been organized and collocated with top machine learning conferences such as ICML and NeurIPS to motivate AutoML research in the machine learning community. The winning approaches from such prior challenges (e.g. auto-sklearn [6]) are now widely used both in research and in industry.
More recently, interest in Neural Architecture Search (NAS) has exploded [7], [8], [9], [10], [11]. On the industry side, many companies such as Microsoft [12] and Google are developing AutoML solutions. Google has also launched various AutoML [13], NAS [14], [15], [16], [17], and meta-learning [18], [19] research efforts. Most of the above approaches, especially those relying on Hyper-Parameter Optimization (HPO) or NAS, require significant computational resources and engineering time to find good models. Additionally, reproducibility is impaired by undocumented heuristics [20]. Drawn by the aforementioned great potential of AutoML in both academia and industry, a collaboration led by ChaLearn, Google and 4Paradigm was launched in 2018 and a competition in AutoML applied to Deep Learning was conceived, which was the inception of the AutoDL challenge series. To our knowledge, this was the first machine learning competition (series) ever soliciting AutoDL solutions. In the course of the design and implementation we had to overcome many difficulties. We made extensive explorations and revised our initial plans, leading us to organize a series of challenges rather than a single one. In this process, we formatted 66 datasets constituting a reusable benchmark resource. Our data repository is still growing, as we continue organizing challenges on other aspects of AutoML, such as the recent AutoGraph competition. In terms of competition protocol, our design provides a valuable example of a system that evaluate AutoML solutions, with features such as (1) multiple tasks execution aggregated with average rank metric; (2) emphasis on any-time learning that urges trade-off between accuracy and learning speed; (3) separation of feedback phase and final blind test phase that prevents leaderboard overfitting. Our long-lasting effort in preparing and running challenges for 2 years is harvested in this paper, which analyses more particularly the last challenge in the series (simply called AutoDL), which featured datasets from heterogeneous domains, as opposed to previous challenges that were domain specific.
The AutoDL challenge analysed in this paper is the culmination of the AutoDL challenge series, whose major motivation is two-fold. First, we desire to continue promoting the community's research interests on AutoML to build universal AutoML solutions that can be applied to any task (as long as the data is collected and formatted in the same manner). By choosing tasks in which Deep Learning methods excel, we put gentle pressure on the community to improve on Automated Deep Learning. Secondly, we create a reusable benchmark for fairly evaluating AutoML approaches, on a wide range of domains. Since computational resources and time cost can be a non-negligible factor, we introduce an any-time learning metric called Area under Learning Curve (ALC) (see Section 2.3) for the evaluation of participants' approaches, taking into consideration both the final performance (e.g. accuracy) and the speed to achieve this performance (using wall-time).
As far as we know, the AutoDL challenges are the only competitions that adopt a similar any-time learning metric.
Acknowledging the difficulty of engineering uni- For domain-specific challenges such as AutoCV, AutoCV2, AutoNLP and AutoSpeech, the challenge results and analysis are presented in [4] and some basic information can be found in Table 1. During the analysis of these previous challenges, we already had several findings that were consistent with what we present in this paper. These include the winning approaches' generalization ability on unseen datasets. As further evidence that we hit the right level of challenge duration and difficulty, 90% of teams found the challenge duration sufficient and 50% of teams  and justify these choices.
The rest of this work is organized as follows. In Section 2, we give a brief overview of the challenge design (see [21] for detailed introduction).  [22].
For the datasets of AutoDL challenge, we are not releasing their identities at this stage to allow us reusing them in future challenges. Some potential uses are discussed in Section 6. However, we summarize their name, domain and other meta-features in Table 2.

Blind testing
Since this was a complete blind evaluation during BOTH phases, we provided additional "public" datasets for practice purposes and to encourage metalearning.
We ran the challenge on the CodaLab platform  This is because all teams adopted a domain-dependent approach and some teams simply used the code of Baseline 3 for certain domains (text in this case).

Metric
The AutoDL challenge encouraged learning in a short time period both by imposing a small time budget of 20 minutes per task and by using an "any-time learning" metric. Specifically, within the time budget, the algorithm could make several predictions (as many as they wanted), along the whole execution. This allowed us to use as performance score the Area under the Learning Curve (ALC): where s(t) is the performance score (we used the NAUC score introduced below) at timestamp t and t is the transformed timẽ Here T denotes the time budget in seconds (e.g.   [24] suggests that average rank is satisfying, compared to other ranking methods, in terms of rank correlation with unseen tasks. We are running similar experiments on AutoDL data and those results hold.

Baseline 3 of AutoDL challenge
As in previous challenges (e.g. AutoCV, AutoCV2, AutoNLP and AutoSpeech), we provided 3 baselines (Baseline 0, 1 and 2) for different levels of use: Baseline 0 is just constant predictions for debug purposes, Baseline 1 a linear model, and Baseline 2 a CNN (see [21] for details  [28], and heavy but more accurate models like LSTM [29] and BERT [30]. They first use light models (such as linear SVC), but the meta-controller switches eventually to other models such as neural networks, with iterative training. If the AUC drops below a threshold or drops twice in a row, the model is switched, or the process is terminated and the best model ever trained is chosen, when the pool is exhausted. with smaller numbers of filters/channels/kernels) and VggVox [33], finally (bidirectional) LSTM [29], with attention mechanism. This strategy allows to make fast early predictions and progressively improves models performance over time to optimize the anytime performance metric.

AUTODL CHALLENGE RESULTS
The The final ranking is computed from the performances on the 10 unseen datasets in the final phase.
To reduce the variance from diverse factors such as randomness in the submission code and randomness of the execution environment (which makes the exact ALC scores very hard to reproduce since the walltime is hard to control exactly), we re-run every submission several times and average the ALC scores.
The average ALC scores obtained by each team are shown in Figure 4 (the teams are ordered by their final ranking according to their average rank). From this figure, we see that some entries failed constantly on some datasets such as frozenmad on Yolo, Kon on Marge and PASA NJU on Viktor, due to issues in their code (e.g. bad prediction shape or out of memory error).
In addition, some entries crashed only sometimes on certain datasets, such as Inspur AutoDL on Tal, whose cause is related to some preprocessing procedure on text datasets concerning stop words. Otherwise, the error bars show that the performances of most runs are stable.

WINNING APPROACHES
A summary of the winning approaches on each domain can be found in Table 3. Another summary using a categorization by machine learning techniques can be found in Table 4. We see in Table 3  workflow shared by almost all teams (see Figure 5).
We note that the module "data" includes not only traditional data of example-label pairs but also metadata, metric, budgets and past performances. These are all potential useful information for meta-learning. based on the meta-data of the current task and poten-tially a meta-dataset of prior tasks (e.g. those provided as public datasets). We believe that this AutoML workflow concisely summarizes the increasingly sophisticated AutoML systems found nowadays and provides the direction for a universal AutoML API design in the future (which is work in progress). This workflow will also be useful for the analysis in Section 5.
The more detailed descriptions for the approaches of the top-3 winning teams and automl freiburg can be found in the Appendix.

POST-CHALLENGE ANALYSES
We carry out post-challenge analyses from different aspects to understand the results in depth and gain useful insights. One central question we ask ourselves is how the components (such as meta-learning, data loading and ensemble), in each approach, affect the speech/times series, text, tabular (and then another cycle in this order). More information on the task can be found in Table 2.

Ablation study
To analyze the contribution of different components in each winning team's solution, we asked 3 teams (DeepWisdom, DeepBlueAI and automl freiburg) to carry out an ablation study, by removing or disabling certain component (e.g. meta-learning, data augmentation) of their approach. We will introduce in the following sections more details on these ablation studies by team and synthesize thereafter.

DeepWisdom
According to the team DeepWisdom, three of the most important components leading to the success of their approach are: meta-learning, data loading and data augmentation. For the ablation study, these components are removed or disabled in the following manner: We carried out experiments on the 10 final phase datasets with the above components removed. The obtained ALC scores are presented in Figure 6. As it can be seen in Figure 6, Meta-Learning can be con-

Combination study
In this section, instead of removing certain components for each winning method, we combine components from different teams. We start from a "base" We focused on the following components, which demonstrated their effectiveness in our previous analyses, including the ablation study:  between the ALC performance and the considered components.
In summary, this limited set of combination experiments did not reveal a significant advantage of mixing and matching modules. The solution of the overall winner DeepWisdom stands out.

Varying the time budget
Up till now, all our experiments are carried out within a time budget of 20 minutes, which may seem rela-

AutoML generalization ability of winning methods
One crucial question for all AutoML methods is whether the method can have good performances on unseen datasets. If yes, we will say the method has AutoML generalization ability. To quantitatively measure this ability, we propose to compare the average rank of all top-8 methods in both the feedback phase and the final phase, then compute the Pearson correlation (Pearson's ρ) of the 2 rank vectors (thus similar to Spearman's rank correlation [43]).
Concretely, let r X be the average rank vector of top teams in the feedback phase and r Y be that in the final phase, then the Pearson correlation is computed The average ranks of top methods are shown in  Table 2, this is a feat from the AutoML community.
Thus it's highly plausible that we are moving one step closer to a universal AutoML solution.

Impact of t 0 in the ALC metric
We recall that the Area under Learning Curve (ALC) is defined by wheret (t) = log(1 + t/t 0 ) log(1 + T /t 0 )     When t0 is large, similar weight is applied on the whole learning curve, performances are uniformly averaged, so being a little bit slow at the beginning is not that bad, and it is more important to have good final performance when the time budget is exhausted (fixed-time learning). The tabular dataset Carla is taken as example. The fact that two learning curves cross each other is a necessary condition for the impact of t 0 on their ranking on this task. Learning curves of top teams on this dataset are shown in 12a. The impact of t 0 on the ALC scores of these curves is shown in 12b. We see that when t 0 changes, the ranking among participants can indeed change, typically the ALC of frozenmad is larger than that of Kon but this is not true for large t 0 . In 12c, the fact that the average rank (over all 10 final phase datasets) varies with t 0 also implies that t 0 can indeed affect the ranking of ALC on individual tasks. However, we see that the final ranking (i.e. that of average rank) is quite robust against changes of t 0 . Very few exceptions exist such as PASA NJU and Inspur AutoDL. Overall, t 0 proved to have little impact, particularly on the ranking of the winners, which is another evidence that top ranking participants addressed well the any-time learning problem.
"meta-learning" and "ensembling": Regarding metalearning, at this stage, it is fair to say that strategies employed are effective, but not very sophisticated.
They rely on pre-selecting off-platform, using pro-

APPENDIX A CHALLENGE PROTOCOL DETAILS
As introduced in Section 2.2, the competition consists of two phases: a feedback phase and a final phase.
Participants should submit an AutoML model code instead of predictions.
Code submitted is trained and tested automatically, without any human intervention. Code submit-  Figure 3 in Figure 14.

APPENDIX C DESCRIPTION OF WINNING METHODS
We present in detail the winning solutions from top-3 winning teams (DeepWisdom, DeepBlueAI and PASA NJU) and the team automl freiburg which made a late submission in the feedback phase but ranked 5th in the final phase. We considered interesting to introduce automl freiburg's approach due to their contributions and for scientific purpose.

C.1 Approach of DeepWisdom (1st prize)
The team DeepWisdom proposed a unified learning framework following a meta-learning paradigm. The framework consists of two parts: meta-train and metainference. The meta-train module takes as input the "public" datasets, which are augmented by the internal data augmentation engine, and the objective function (the ALC metric in the case of the challenge). The meta-trainer generates solution agents, whose objective is to search for best models, using search operators.
In the meta-inference step, a new task is processed taking in one dataset of the challenge. Initial meta-  On the contrary, if training is fast and prediction is slow, the frequency is reduced. This strategy can improve to higher early scores.
In order to improve generalization, multiple light-GBM models are used to make an ensemble with a bagging method.

C.3 Approach of PASA NJU (3rd prize)
The PASA NJU team modeled the problem as three which is an open-source package that uses Bayesian optimization to guide the search of hyperparameters.

C.4 Approach of automl freiburg
In contrast to other teams, automl freiburg adopts a domain-independent approach but focused only the computer vision tasks (i.e. image and video datasets) of this challenge. While for all other tasks automl freiburg simply submitted the baseline to obtain the baseline results, they achieved significant improvement on the computer vision tasks w.r.t. the baseline method. To improve both efficiency and flexibility of the approach, they first exposed relevant hyperparameters of the previous AutoCV/AutoCV2 winner code [54] and identified well-performing hyperparameter configurations on various datasets through hyperparameter optimization with BOHB [36]. They then trained a cost-sensitive meta-model [55] with Auto-Folio [35] -performing hyperparameter optimization for the meta-learner -that allows to automatically and efficiently select a hyperparameter configuration for a given task based on dataset meta-features. The proposed approach on the CV task is detailed next.
First, they exposed important hyperparameters of the AutoCV/AutoCV2 winner's code [54] such as the learning rate, weight decay or batch sizes. Additionally, they exposed hyperparameters for the online execution (which were hard-coded in previous winner solution) that control, for example, when to evaluate during the submission and the number of samples used. To further increase the potential of the existing solution, they extended the configuration space to also include: • An EfficientNet [56] (in addition to kakaobrain's [54] ResNet-18) pre-trained on ImageNet [39]; • The proportion of weights frozen when finetuning; • Additional stochastic optimizers (Adam [57], AdamW [58], Nesterov accelerated gradient [59]) and learning rate schedules (plateau, cosine [60]); • A simple classifier (either a SVM, random forest or logistic regression) that can be trained and used within the first 90 seconds of the submission.
After the extension of the configuration space, they optimized the hyperparameters with BOHB [36] across 300 evaluation runs with a time budget of 300 seconds on eight different datasets (Chucky [61], Hammer [62], Munster [63], caltech birds2010 [64], cifar100 [61], cifar10 [61], colorectal histology [65] and eurosat [66]). These eight datasets were chosen from meta-training data to lead to a portfolio of complementary configurations [67], [68]. Additionally, they added a robust configuration to the portfolio of configurations that performed best on average across the ing choices for training, input pipeline, and architecture) for every task (dataset) in our meta-training set using BOHB [36]. Afterwards, for each dataset i, the best found configuration λ * i is evaluated on the other datasets j ∈ {1, 2, ..., N }, j = i to build the performance matrix (configurations × datasets). For training and configuring the meta-selection model based on performance matrix and the meta-features of the corresponding tasks, the approach uses AutoFolio [35]. At meta-test time, the model fitted by AutoFolio uses the meta-features of the test tasks in order to select a well-performing configuration. eight datasets. Then, they evaluated each configuration of the portfolio for 600 seconds on all 21 image datasets they had collected. In addition, they searched for a tenth configuration (again with BOHB), called the generalist, that they optimized for the average improvement across all datasets relative to the already observed ALC scores. In the end, the meta-train-data consisted of the ALC performance matrix (portfolio configurations × datasets) and the meta-features from the 21 datasets. These meta-features consisted of the image resolution, number of classes, number of training and test samples and the sequence length (number of video frames, i.e. 1 for image datasets). In addition, they studied the importance of the meta features for the meta-learner, and selected an appropriate subset. To optimize the portfolio further, they applied a greedy submodular optimization [68], [69] to minimize the chance of wrong predictions in the online phase. Based on this data, they trained a cost-sensitive meta-model [55] with AutoFolio [35], which applies algorithm configuration based on SMAC [37], [70] to efficiently optimize the hyperparameters of the metalearner. Since the meta-learning dataset was rather small, HPO for the meta-learner could be done within a few seconds. Lastly, they deployed the learned Auto-Folio model and the identified configurations into the initialization function of the winner's solution code.
The workflow of this approach is shown in Figure 15.