December is a time for traditions. For many people, those include giving generously to charitable causes. Unfortunately, some charities seek to manipulate these instincts by making the world seem a grimmer place than it really is.
UNICEF, the United Nations charity with responsibility for children, maintained such a tradition this month with the release of its report The Children Left Behind, which examines the status of poor children in 24 wealthy nations. True to form, UNICEF delivered generally bad news for Canada.
"Rich countries, including Canada, letting poorest children fall behind," is how UNICEF sold its report to the Canadian press. Apparently, we place a dismal 17th in terms of the material well-being of our less-fortunate children. According to Marv Bernstein, chief advisor of advocacy for UNICEF Canada, the only solution is greater taxation and more income redistribution.
It all sounds rather serious, until you read the report. Then the problem seems to go away.
As with all statistical arguments, understanding the UNICEF report requires a careful eye and a healthy dose of skepticism. The study examines the status of poor children within three categories: material well-being, health and education. It also focuses entirely on relative differences; that is, the gap between children in the middle of the income distribution and the average below that line. Exclusively relying on relative indicators is a highly contentious, and generally misleading, way to examine poverty. The UNICEF report is no exception.
While Canada did well in education and health, we scored very low in material well-being. Our performance in housing for the poor was particularly abysmal. A closer look is necessary.
UNICEF claims the number of rooms per person in a house is the best way to measure the adequacy of housing. OK. But it is only interested in how this affects the gap between average and low-income children.
Average families have an estimated 1.5 rooms per person in Canada, according to UNICEF. For poor families, this figure is 1.1 rooms per person. A gap of 0.4 rooms per person is the largest recorded among all nations. As a result we score 20th out of 24 in housing for the poor.
But do poor families really care about the number of rooms in other houses in other parts of town? Or do they simply care about how comfortable they are in their own place?
This is significant because Canada's 1.1 rooms per person for poor children is the highest figure among any of the 24 nations studied in the UNICEF report. Iceland, the top scoring nation in this category, boasts only 0.91 rooms per person in poor households. Greece has just 0.69.
In fact Greece rates fourth highest on the list because it has such a small gap between average and poor housing. But average households in that country have only 0.8 rooms per person. In other words, poor families in Canada have better housing than average families in Greece. Still, Greece ranks much higher than we do. But where would you rather live if you were poor?
Putting Canada at the bottom of this list is clearly an absurd conclusion to draw from the data. If Canada's poor are better housed than the poor in any other country in the world (and better housed than average families in some countries), we should be at the top of the heap and drawing unqualified praise from UNICEF.
More to the point, how would UNICEF propose that we solve Canada's alleged housing crisis? If poor children in Canada already have the best housing anywhere, the only conceivable solution is to remove rooms from the houses of Canada's wealthier families in order to shrink the gap. But this will do nothing to improve anyone's material well-being.
This is not the first time UNICEF has twisted child poverty in Canada for seasonal effect. In November 2007 UNICEF Canada published a report with a chart showing child poverty in this country growing from 14.4% in 1989 to 17.7% in 2007 and declaring the situation a "persistent problem." And yet every number presented was wrong: Some appeared to be drawn from thin air. All used before-tax calculations of income, which ignore the crucial impact of government transfers on poverty indicators. The real story was that after-tax low-income among Canadian children fell from 11.9% to 9.6% over this time. It's still falling.
Child poverty remains a real issue for Canada, but tackling it requires coherent data and a commitment to taking the facts as they come. Whatever the season.


Israeli food san diego Neighbors Silence Pepco's Chain Saws Washington Post, Sept. 30th 2004 Posted Trees, Press coverage on Thursday, September 30th, 2004. Neighbors Silence Pepco's Chain. Carol Barnaby, LCSWC is a mental health therapist in private practice in the Washington,DC metro area. She specializes in the treatment of mental health issues that include, but.
Kraft foods inc stock split William Devry pictures wallpapers gallery biography movies. Biography: William was born and raised in Montreal, Canada. He moved to Florida with his mother when he was 19 and.


In the traditional theatre the concept of space is tightly linked to the model of a container: The black box as a receptacle that is marked out by walls and can always be filled again anew. The stage, the peep box of a closed room, is screened from the outside world of urban space. The exterior stands for reality and the interior is contrasted to reality, the unreal, fictional, imaginary. But theatrical space can be thought of more expansively. It comprises audience as well as performers. The way the space is structured determines how stories can be told and viewed: Who is looking and who is performing? Helgard Haug, founding member of Rimini Protokoll, will focus on different theatrical settings in urban space and on different ways of how an audience can be placed in(to) this setting in their work.


Provision Living at West County is located in Des Peres, Missouri, just west of I-270 and north of Manchester Road. Our location is in the heart of St. Louis County and offers proximity to premier shopping, dining, and many of the area's most established neighborhoods, and is just 20 minutes from downtown St. Louis.
While most senior living communities focus on placing elders either in memory care or assisted living, Provision Living at West County offers a solution for families who may feel uncertain about the right type of care needed for their loved ones. Our approach does not force a choice between assisted living and memory care. Instead, we support memory loss, Alzheimer's and dementia needs for all residents, because aging usually involves some form of memory impairment.
Provision Living at West County is a 60-apartment senior living community that was intentionally designed to meet the unique needs of older adults. With both assisted living and memory care services, residents have their choice of purposefully designed studio and one-bedroom private apartments, which include full bathroom, kitchenette, emergency call response, fire safety system, and individualized heating and cooling. The building also features a beautiful dining room, ground level courtyard, full-service salon, computer room, fitness center, on-site rehabilitation center, and much more.


Technology is increasingly enabling new forms of work to be done at a distance. In large-scale endeavors such as peer production and crowdsourcing, people collaborate with and build off the work of others, many of whom they have never met in person. The combination of social media features with online work environments has the potential to change the nature of work in many fields by increasing visibility of people's actions and allowing people to learn from and assess others' expertise. However, little is known about the psychological and behavioral impact of exposure to increased detail about unknown others in a peer production setting.
In this talk, I will describe a series of studies I have conducted looking at the impact of increased social transparency and the ability to view others' activities on attitudes and behaviors towards strangers in an online peer production context. I will discuss findings from interviews with open source software project owners on GitHub to understand how they were using information to inform their impressions of unknown contributors. These initial insights then informed a series of experiments evaluating how activity trace design and visualization influence perceptions of others and reactions to their work.
I will also discuss recent work looking at another facet of remote collaboration in which we have been exploring how geographically distributed teams are using technology to share and create digital artifacts.
Jennifer Marlow is a Research Scientist at FX Palo Alto Laboratory, Inc. She recently received her PhD from Carnegie Mellon's Human-Computer Interaction Institute. Her research focuses on social computing and computer-supported cooperative work. Specific interests include understanding how technology and social media facilitate online collaboration, learning, and professional development, and enable awareness and interaction across geographic and temporal distance.


text:
Streaming Radio for Philadelphia
More Music. Fewer Interruptions. Just what you want to hear.
FOLLOW: FREE MOBILE APPS:
Alvin Clay	Philadelphia, PA
iradioal
THE POST | News | Sports | Culture | Business | Free Forum
<< prev 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 next >>
Their Conversations
Hot Coffee Thrown on Dunkin' Worker; No Philly Schools Closing; Fake PWD Workers Hit South Philly
by: iradioal - Philadelphia, PA
started: 01/10/14 6:44 pm | updated: 01/10/14 6:44 pm
Police are asking for the public's assistance in identifying a suspect who threw hot coffee on a Dunkin' Donuts employee. The incident happened on Wednesday afternoon at 1pm on the 4700 block of North Broad Street. A customer became upset when a female Dunkin' employee would not accept an expired coupon he had on his iPhone. The suspect argued with the employee and tossed his hot cup of coffee her before leaving. The worker was not seriously injured. Police are investigating the matter. If you... Read More
Chris Christie Apologizes, Fires Top Aide; Penn State Hires Franklin; Kevin Hart Ice Cube in Philly
New Jersey Governor Chris Christie held a lengthly press conference today in response to the growing scandal surrounding the discovery that traffic closures on ramps to the George Washington Bridge were done as political payback for Fort Lee NJ's mayor not endorsing Christie's re-election. Christie maintained that he did not have any knowledge of his administration's involvement but would take responsibility. "I come out here to apologize to the people of New Jersey," he said. "I am... Read More
Ice Jams on Delaware River / Flash Flood Warnings; Suspect Arrested in Fatal Upper Darby Hit-and-Run
Ice jams on the Delaware River near Trenton have prompted a Flood Warning for Bucks and Mercer counties. The warnings are in effect until 4:30 Thursday morning. Officials will be monitoring the water levels on both sides of the river. Route 32 (River Road) between Black Rock Road and Trenton Avenue in Lower Makefield Township has been closed. In New Jersey, flooding has been seen at Route 29 and Market Street and in the lower State House parking. Rising temperatures should help the jam... Read More
Phillies' Broadcasters Chris Wheeler and Gary Matthews are out for 2014
Longtime Philadelphia Phillies' broadcaster Chris Wheeler will not be returning to the broadcaster booth in 2014 after nearly four decades as a play-by-play and color analyst on both TV and radio. Gary Matthews will also not be rejoining the broadcast team after seven years doing color for television and radio.
The decision comes after the Phillies recently signed a new broadcast contract worth $2.5 billion over 25 years. It is reported that the network made the decision to... Read More
Schuylkill Named River of the Year; Christie Signs Immigrant Tuition Bill; Cold Causes Car Troubles
The Schuylkill River has been named the 2014 Pennsylvania River of the Year. The Schuylkill received 43% of the popular vote among the 5 nominated rivers. They include the Schuylkill; Kiskiminetas-Conemaugh rivers in southwestern Pennsylvania; Ohio in the west; Brodhead Creek Watershed in the northeast; and West Branch of the Susquehanna in the north central section of the state. This is the second time the Schuylkill has been named River of the Year, the last time being in 1999. The... Read More
Jihad Jane Sentenced to 10 years; 6-year-old Boy Killed by Bus; Delco Market Worker Stung Scorpion
A Montgomery County woman who called herself 'Jihad Jane' was sentenced today for plotting to kill a Swedish political cartoonist for publishing a cartoon perceived as offensive to Muslims. Colleen LaRose, 50, of Pennsburg, PA, had travelled to Ireland after being recruited on jihadist websites and helped other suspected terrorists in planning the attack, which was never actually carried out. LaRose returned to Philadelphia in 2009 and surrendered. She was sentenced today to 10 years in... Read More
Lincoln Financial Field Snow Removal; Philly Streets Dept. Hard at Work; Msgr. Lynn Released
Probably the biggest snow removal effort short of PennDOT is happening down in South Philly at Lincoln Financial Field. 400 workers from American Winter Services and local unions descended upon the Linc to clear out all the snow in time for tomorrow night's Eagles' playoff game. Bracing the bitter cold and winds the crews headed into the stands to shovel the snow to chutes that send the snow down to field level. It is then pick up in front end loaders and out to dump trucks. The trucks... Read More
Msgr Lynn Out on Bail; Wilmington Woman Found Unresponsive; NYE Drug Deal Gone Bad; Winter Storm
Monsignor William Lynn left state prison today on bail after his conviction on child endangerment was overturned in December by a PA superior Court. Lynn spent the past 18 months at a prison in Waymart in northeastern Pennsylvania. Philadelphia Sherriff's department will take him to Philadelphia to fit him with his electronic monitoring device. Lynn will will go before Common Pleas Judge Sarmina on Monday to go over the terms of his release.
Wilmington Police are... Read More
New Year's Eve Gun Celebrations; Late Week Storm and Cold; New Year Countdown on iradiophilly
Philadelphia's top law enforcements officials including Police Commissioner Charles Ramsey and District Attorney Seth Williams were joined by gun shot victim, Joe Jaskolka, at a press conference today discussing the dangers of using guns to celebrate on New Years Eve. Ramsey reminded us that shooting a gun in the air is dangerous at best and potentially deadly. The laws of gravity still apply to bullets; they go up and eventually and unpredictably come down. Jaskolka recalled walking in... Read More
Msgr. Lynn Granted Bail; Gov. Corbett Drops Lottery Bid; Teen Shot outside party in DE on Saturday
Monsignor William Lynn will be granted bail while his conviction on child endangerment goes through the appeals process. Judge Teresa Sarmina set bail at $250,000, requiring 10 percent to post. Lynn will also have to wear an electronic monitoring anklet while out on bail. Sarmina said that she wants Lynn to appear before her in person so that she can explain the full conditions of his release. Lynn is currently in state prison in Northeast Pennsylvania where he has served 18 months of a... Read More
Welcome to iradiophilly! Philadelphia's Internet Radio Community
Welcome! Start listening now for FREE! The music is always on, so choose a radio station and click to listen. You can find fun Events around Philly and you can list yours on our Events page FREE. We will also promote cool Philly shows, events and fundraisers on the air FREE. There are also great opportunities to become a sponsor or advertise. iRadioPhilly supports the Philadelphia music and arts scene. If you are a local musician or performer, send us your stuff. We will play it on our BYO radio station, Philadelphia's Local Stage. We're designing the radio stations and iradiophilly to reflect Philadelphia's music tastes, so we'll want your input. We're glad you're here. Invite your friends.
HOME | ABOUT | ADVERTISE | CONTACT | EVENTS | PHOTOS | PODCASTS | STATIONS | CONTESTS | SPONSORS | BUSINESSES/ORGs | SCHEDULE | SITEMAP | THE POST © 2010-19 iradiophilly 501(c)(3)
Site Design by Bananaland | Built by quadpain media

redpajama_set_name: RedPajamaCommonCrawl


text:
\section{Introduction}

Curriculum Learning (CL) \cite{narvekar2020curriculum,bengio2009curriculum, TiddBrendan2020GCLf} is a powerful reinforcement learning technique that engenders powerful behaviours by gradually increasing the difficulty of the task to be solved.  This incremental approach has numerous benefits including performance and generalisation ability. To maximise the benefit of a curriculum, it is important to have a training environment that increases in complexity proportional to the learning ability of the agent, and contains relevant features to encourage learning.  Diverse training examples, presented to the learner in an appropriate order \cite{heess2017emergence}, have resulted in the generation of capable, complex behaviours for simulated robots \cite{TiddBrendan2020GCLf, xie2020allsteps}, and aided sim2real transfer of those policies to real robot deployments \cite{lee_learning_2020}.

A typical setup for learning locomotion skills (a popular focus area) involves the creation of a set of terrains of varying difficulty through some terrain generator, typically a parameterised noise function.  The terrains are then presented to the agent, generally in order of difficulty, and the agent learns by solving simpler examples to eventually reach a given target behaviour.  Noise functions are popular candidate terrain generators, as increasing terrain difficulty can be simply realised by increasing values of the noise parameters.  Different generators produce terrains with different geometric features, which has an affect on learning.

To date, the impact of the chosen terrain generator has not been explored, and this is the focus of our paper.  We address the effects of terrain generator on the learning process in two steps.  In step 1, we select a varied range of popular terrain generators from the literature, which includes both direct and indirect encodings. Because the difficulty of an indirectly-coded terrain cannot be ascertained {\em a priori}, we categorise the resulting terrain mesh according to a set of representation-agnostic features selected from related literature.  The selected features become feature descriptors in MAP-Elites, providing a diverse set of high-quality terrains.  Terrains are evolved to fill the archive and generate one curriculum per generator type.   In step two, each curriculum is used to train a bipedal walker, and results compare reachable terrain difficulty and learning speed.  Our approach is illustrated in Figure \ref{fig:fig1}.

We present two main original contributions; (i) the selection of appropriate representation-agnostic feature descriptors, including coverage analysis of curricula evolved using those features, and (ii) analysis of the effects of generator type on learning performance and rate.  Results show the identification of suitable feature descriptors for representation-agnostic terrain-based curriculum learning, and suggest key differences between the  generators, particularly between direct and indirect representations in terms of map coverage.  We provide evidence that a multi-generator approach may be beneficial to the generation of curricula that promote rapid learning.

The remainder of the paper is organised as follows; Section 2 presents pertinent background research.  Section 3 describes our methodology.  Section 4 presents experiments and results, and Section 5 provides a discussion.

\section{Background}

Reinforcement Learning (RL) \cite{kober2013reinforcement} is one of the most commonly used methods to train agent behaviours, and is similar conceptually to an evolutionary algorithm which learns by continually interacting with (and being rewarded by) an environment.  In this study we use Proximal Policy Optimisation (PPO) \cite{ppo} due to its ubiquity as a benchmark method to train agents in complex environments. PPO is a policy gradient method that alternates between sampling data through the agent's interaction with the environment and optimising an objective function using stochastic gradient descent. PPOs main advantage over other policy gradient techniques is the use of multiple epochs of mini-batch updates rather than performing one gradient update per data sample.

Curriculum learning \cite{bengio2009curriculum} is a form of incremental learning \cite{howard2014evolving}, designed to address some common challenges with RL \cite{narvekar2020curriculum}.  Firstly, it allows for agents to solve very hard problems by progressively building competency in easier versions of the problem  \cite{bongard_utility_2010}.  Secondly, it provides a wealth of learning experiences to the agent which can greatly assist with generalisation \cite{KadianAbhishek2020SPDE, HoferSebastian2021SiRa,nichol2018gotta}.   Lee et al. \cite{lee_learning_2020} demonstrate both of these benefits using a particle filter to update parameters of a noise function-based terrain generator, allowing complex terrains to be navigated by a real quadruped after simulated learning.

A guided curriculum approach \cite{TiddBrendan2020GCLf} incrementally removed supporting forces from a bipedal walker's body before increasing terrain complexity and diversity.  Accelerated learning of complex environments is also evidenced with hexapod robots~\cite{9043822}. Miras demonstrated that balancing behaviour could be achieved by training on a tilted plain rather than a flat plain with objects on it, even though balance was not incorporated into the fitness function of the robot~\cite{miras_effects_2019}. Huizinga \cite{huizinga2019evolving} found that ordering of sub-tasks towards learning a difficult task is challenging, and instead proposed the Combinatorial Multi-Objective Evolutionary Algorithm (CMOEA) to simultaneously explore all orderings. This is an effective CL method when subtasks can be clearly defined (e.g. jumping, walking). Xie \cite{xie2020allsteps} demonstrated the critical role of a curriculum to train three bipedal agents to walk on stepping-stone scenarios, with final terrain complexity and learning rate being superior for curriculum approaches compared to non-curriculum RL.  Akkaya \cite{akkaya2019solving} presents a CL approach using automatic domain randomization (ADR), demonstrating vastly improved sim2real transfer compared to a non-curriculum baseline. ADR automatically expands the randomisation range parameterising a distribution over environments. Florensa \cite{florensa2018automatic} showed that using a curriculum generating network it is possible to train an agent to `perform a wide set of tasks without requiring any prior knowledge of its environment'.   Open-ended curricula \cite{zhou2021openended} can learn on terrains that are continually generated during agent learning \cite{poet}.

Procedural Content Generation (PCG) originated in computer graphics and video game design. PCG can create large volumes of high-quality content with controllable randomness, including digital objects, landscapes, levels, textures and 3D models.  PCG has been readily adopted by the machine learning community ~\cite{risi2020increasing} to create a wealth of training data.  Rich and diverse environments encourage agent learning and improve robustness \cite{heess2017emergence}, with works showing that curriculum learning can train several quadrupedal robot policies in parallel to walk over uneven terrains in simulation \cite{rudin2021learning}. Terrain generators have been previously evolved \cite{evoTG,asasas}, however these works focus only on a single representation and do not evaluate terrains in an agent-based learning context.  Also in an evolutionary context, quality-diversity algorithms \cite{pugh2016quality} present an effective method for storing and traversing a curriculum, shown in both robotics and game-playing contexts \cite{Cully_2015} \cite{9300206}.  In particular,the use of aligned feature dimensions within MAP-Elites \cite{mouret2015illuminating} can provide direction to a (stochastic) traversal algorithm.

Overall, we see that curriculum learning is a powerful technique relevant to both evolutionary algorithms and reinforcement learning.  The literature abounds with a smorgasbord of both direct and indirectly-represented generators, including various noise functions as well as CPPNs and GANs.  Additionally, we see many different feature dimensions being used to store the curriculum and permit traversal. However, we identify a significant literature gap in that these different selections of generators and features have not yet been compared.  In this study we compare popular generators and a set of selected feature descriptors for an evolved MAP-Elites based curriculum.  We attempt to inform the selection of appropriate terrain generators and feature dimensions for fellow researchers in the field.  We also demonstrate a curriculum learning approach that simultaneously supports both direct and indirect representations, opening up future work in mixed-generator curricula.

\section{Methodology}

We follow a three-stage process: (i) decide on features, (ii) evolve curricula (iii) learn on curricula.  To allow for the isolated study of generators, we omit other tasks that typically comprise a curriculum (steps, jumps, etc.) and focus purely on locomotion in rough terrains.

\subsection{Generators}

We select 5 popular and diverse generators from the literature: 3 direct (Perlin noise, Diamond Square Noise, Worley Noise) and two indirect (a GAN, and a CPPN). Each generator mapped to a 256x256 resolution pixel map which is converted to a heightmap and then to a solid terrain mesh for simulation.

\subsubsection{Perlin Noise}

Perlin noise was applied to fractal Brownian Motion (fBM). The genome was parameterised into scale [1-100], octaves [1-9], persistence [0.1-0.9], lacunarity [1-3] and a random seed [0-100].

\subsubsection{Diamond Square Noise}

We use a version of the diamond square noise algorithm \cite{diamondsq}.  The four corner points of the heightmap grid are initialised with a random value between -1 and 1 making a square. The following two steps alternate until all the values of the heightmap are assigned:

Diamond step: for each square in the grid, assign the value of the midpoint of that square to be a random percentage \emph{P} of the average of the four corner points, plus a random value \emph{R} between -1 and 1.

Square step:  for each diamond in the grid, assign the value of the midpoint of the diamond to be a random percentage \emph{P} of the average of the four corner points, plus a random value \emph{R} between -1 and 1.

After each iteration of these steps, the range of the random value \emph{R} is reduced using the formula:

\[
\frac{-1}{(steps*D+1)} \leq R \leq \frac{1}{(steps*D+1)}
\]

With $\emph{D} \in [1,10]$, and \emph{steps} corresponding to the number of iterations of the above two steps performed.  \emph{P} is computed as:

\[
Z\leq P \leq 100-Z, Z \in [0,50]
\]

The genome is represented by a random seed, the four initial corner values, percentile (\emph{Z}), and level (\emph{D}).

\subsubsection{Worley Noise}

Worley Noise is typically used to generate procedural textures, and is implemented following \cite{worley}. The genome for the Worley terrain generator consists of a seed for randomisation, the number of feature points $\emph{N} \in [2,400]$ and the index of the point sorted by ascending distance to the current point $\emph{D} \in [0, \frac{N}{4}]$.

\subsubsection{GAN}

The Deep Convolutional GAN (DCGAN) model was trained on real terrain heightmap data with identical resolution to the humanoid in simulation. The original data was in pointcloud format obtained from the OpenTopography website. The ground plane was semantically segmented, and the resulting points interpolated into multiple heightmap patches at the desired resolution. This provided a large amount of high resolution data (30000 256x256 patches) which was used to train the model. The latent vector fed into the generator contained 50 values which formed the genome of the GAN generator.

\subsubsection{CPPN}

The CPPN terrain generator has 2 inputs corresponding to row and column of the heightmap, and one output for the height value. All remaining CPPN settings were taken from the PicBreeder paper \cite{picbreeder}.

\subsection{Features}

We explore a range of generic terrain descriptors for use as MAP-Elites feature descriptors that allow both direct and indirect generators to be compared. Features were aligned to terrain traversability to provide continuity to the final generated curricula, with enough variation to create diversity in the population.  Features originate from a range of non-RL, non-evolutionary fields, mainly from surveying where they are used to categorise real terrains. We use:

\begin{enumerate}
    \item Terrain Ruggedness Index (TRI) \cite{tri} - measures the average difference between a pixel and its 8 neighbouring pixels.
    \item Topographic Position Index (TPI) \cite{tpi} - measures the difference between each pixel and the mean of its 8 neighbouring pixels
    \item Roughness \cite{roughness} - measures the maximum difference between a pixel and its 8 neighbouring pixels
    \item Traversability Estimation Model implemented in \cite{traversability} - A trained model that predicts the traversability of terrains by outputting a traversability map. This model had been trained on different types of terrains to the terrain generators used in this paper. The orientation input to the model was set to 0 as this was the direction the humanoid traversed.
\end{enumerate}

Once generated, a terrain mesh is tagged with values for each feature descriptor.  Settings for kernel size are determined in Section 4.  Roughness, TPI and TRI use a stride length of 2.  The average of the output from each feature was taken as the overall measure.  Initial experimentation found that an archive discretisation of 50 bins per descriptor presented meaningful but achievable differences between terrains in neighbouring cells.

\subsection{Evolutionary Algorithm}

We evolve curriculums for each generator, and compare different pairs of feature dimensions.  Per treatment, we run MAP-Elites for 5000 generations, with 100 random initial genomes.  Per generation, 20 new terrains were generated by randomly selecting from the current members of the archive, mutating, generating feature values, and adding back into the archive as in Figure \ref{fig:fig2}.

\begin{figure}[h!]
\centering
\includegraphics[width=1\columnwidth]{images/MAP_ELITES.png}
\caption{Overview of library generation.  Features must be calculate in simulation before the terrain can be added to the library.}
\label{fig:fig2}
\end{figure}

For noise emitters, each gene in the genome had a mutation probability of 0.35, and mutation altered the allele by $\pm$ a value taken from a normal distribution with covariance set to 10\% of its range.  CPPN mutation followed NEAT \cite{picbreeder}.  The GAN's latent space was stored in 50 variables, which with P=0.07 were mutated by a value taken from a normal distribution with covariance set to 10\% of its range (heuristically determined).

Fitness is set to 1.0 - the difficulty of the generated terrain.  Difficulty is assessed using the base policy of the Bipedal Walker that is trained on flat ground\cite{coumans2017pybullet} and simulated in PyBullet. Difficulty of a terrain is calculated as the average distance travelled out of the best 5 of 20 attempts with random joint initialisations, normalised between 0 and 1. The 5 best were taken as some initialisations were too extreme and caused a large amount of noise in the fitness estimation.  When replacing individuals in a MAP-Elites cell, we kept the lowest difficulty terrain and deleted the higher difficulty one.  This generated a smoother curriculum in terms of fitness, whilst also removing many impossible terrains.  A second step removed the remaining impossible terrains, following Algorithm 1.  Representative terrains for two generators (CPPN and Perlin noise) that achieve difficulties of $\approx$0.1, $\approx$ 0.5, and $\approx$ 0.9 are shown in Figure \ref{fig:fig3}.

\begin{algorithm}[h!]
\caption{Check terrain traversability.}
\begin{algorithmic}
\Procedure{Check Terrain}{$hm$}\Comment{Terrain heightmap}
\State $threshold\gets \frac{robot Height}{3}$\Comment{max incline in metres}
\State $k\gets 26$\Comment{Approx. step length of robot}
\State Initialise  differences result [hm.rows-k,hm.cols-k]
\For{r in [0, hm.rows - kSize)}
\For{c in [0, hm.cols - kSize)}
\State differences[r,c]$\gets$

maximum\_difference(hm[r:r+k,c:c+k])
\EndFor
\EndFor
\State \textbf{return} $max(differences) \leq threshold$
\EndProcedure
\end{algorithmic}
\end{algorithm}

\begin{figure}[ht!]
\begin{center}
\centering
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/cppn_1.png} }
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/cppn_5.png} }
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/cppn_9.png} }\\
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/noise_1.png} }
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/noise_5.png} }
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/noise_9.png} }\\
\end{center}
\caption[]{Representative pixel maps with approximate difficulties of 0.1, 0.5, 0.9 respectively (L-R) for (a)-(c) an indirect representation (CPPN) and (d)-(f) a direct representation (Perlin noise).}
\label{fig:fig3}
\end{figure}

\section{Experimentation}
\subsection{Experiment 1: Features and Curriculum}

Initial experimentation compared feature descriptors with different kernel sizes to measure their alignment to terrain difficulty (fitness). This alignment was ideal for constraining the solution search space to produce a strong traversability gradient in the MAP-Elites grid. Without a strong gradient, creating an effective curriculum becomes difficult. Roughness, TPI and TRI had similar results with TPI being most correlated. Results are shown in Figure \ref{fig:fig4}. We used TPI in learning experiments as it was most aligned, and use Roughness since TPI and TRI (k = 30) were too similar (r = 0.94/0.95) to produce sufficient map diversity. A kernel size of 30 was chosen as it had consistently high correlations to terrain difficulty.  These features could then be used for arbitrary meshes with confidence that they were meaningful metrics.  The traversability model ("Traversability") had the weakest correlation to difficulty so this was not used further.

\begin{figure}[ht!]
\centering
\includegraphics[width=0.95\columnwidth]{images/correlation_heatmap_cppn.png}
\includegraphics[width=0.95\columnwidth]{images/correlation_heatmap_noise.png}
\caption[]{Heatmaps showing Pearson correlation of all considered terrain characterisation features including ground truth: terrain difficulty, for (a) CPPN and (b) Perlin noise. }
\label{fig:fig4}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=0.9\columnwidth]{images/upload.png}
\caption[]{Feature coverage of each generator after each experiment.}
\label{fig:fig5}
\end{figure}

\begin{figure*}[ht!]
\centering
\includegraphics[width=2\columnwidth]{images/new_images/allcoveragefigs_landscape.png}
\caption[]{MAP-Elites archive coverage for three combinations of features: TPI/TRI (row 1), roughness/TRI (row 2), roughness/TPI (row 3) for Diamond Square noise (a), Perlin noise (b), Worley noise (c), GAN (d) and CPPN (e) terrain generators after evolution. Colour corresponds to fitness, measured as the average performance of the base policy humanoid traversing each terrain.  Dark colours represent fitter behaviours.}
\label{fig:fig6}
\end{figure*}

The coverage of each terrain generator is shown in Figure \ref{fig:fig5}.  Pairwise feature comparisons are shown in Figure \ref{fig:fig6}.  All coverage maps show a strong gradient in terrain traversability, with it declining as the map feature values increase. This suggests the chosen features are very representative of traversability. Some variation exists between the different features, but the most significant basis of variation is in the euclidean distance between feature values and (0,0). Each terrain generator presents a characteristic shape and amount of coverage on the map. Table 1 shows the comparative coverage between generators, with CPPN having significantly higher coverage, and able to create more difficult and more diverse terrains than the other generators.    Interestingly the other indirect emitter, GAN, presents the lowest coverage of all. The GAN was trained on real terrains and therefore can only output similar meshes.  The CPPN is unrestricted in this manner.  Perlin noise shows the strongest diagonal mapping to the feature dimensions and presents the best direct encoding, followed by Worley noise.  Diamond-square noise can be seen to struggle to generate reasonable values for TPI.  Feature ranges were identical between generator coverage maps for fair comparison. While the total terrains in each map vary significantly based on coverage, adjusting feature ranges to account for this does not help learning as terrains with comparable features are similarly traversable and are skipped.

\begin{table}[h!]
\centering
\caption{Coverage percent of total map for feature combinations with each terrain generator; maps from Figure \ref{fig:fig5}.}
\begin{tabular}{|p{2.3cm}| c c c|}
      \hline
 Terrain Generator & Roughness/TPI & Roughness/TRI & TPI/TRI \\ [0.5ex]
 \hline
 CPPN & 23.16\% & 15.52\% & 14.92\% \\
 \hline
 GAN & 2.48\% & 2.48\% & 1.48\% \\
 \hline
 Perlin Noise & 3.92\% & 3.92\% & 2.80\% \\
 \hline
 Worley Noise & 2.88\% & 1.84\% & 2.12\% \\
 \hline
 Diamond Square & 3.12\% & 1.36\% & 1.88\% \\
 \hline \hline
 Combined & 23.64\% & 15.56\% & 15.56\% \\ [1ex]
 \hline
\end{tabular}

\label{table:1}
\end{table}

\subsection{Experiment 2: Bipedal Walker Learning}

\begin{figure}[ht!]
\centering
\subfloat[]{\includegraphics[width=1.0\columnwidth]{images/11.png}}\\
\subfloat[]{\includegraphics[width=1.0\columnwidth]{images/22.png}}
\caption[]{Illustrative performance comparison between (a) the different terrain generators, and (b) CPPN terrains between the proposed map-based curriculum algorithm and incrementally increasing roughness with standard CL, showing the most difficult terrain completed at the current training iteration. Plots finish when 30k epochs is reached or IT\&E has excluded all terrains. Averaged over 10 trials, with confidence shown for $\pm$1 standard error.}
\label{fig:fig7}
\end{figure}

In the second experiment we carry out a typical learning experiment on the evolved curricula.  We trained the walker from the base policy using the PyTorch PPO implementation from SpinningUp \cite{SpinningUp2018} with default parameters with a maximum of 30,000 epochs.  The base policy provides the prior for IT\&E.  Traversal of the map is the same Gaussian Process (GP) technique as described by Cully et al. \cite{gaussianprocess}, using the same recommended parameters: $\rho$ = 0.4, $\alpha$ = 0.9, $\kappa$ = 0.05, ${\sigma^2}_{noise}$ = 0.001, and the same covariance function (Matern kernel). PPO trains for 40 epochs at a time until the humanoid reaches the 90\% traversability threshold (90\% of maximum fitness achieved) or when there had been 100 traversability evaluations on the terrain. Once this condition is met, the GP model finds the highest fitness (easiest) terrain from evaluations with the updated PPO model. However, unlike the original IT\&E, our algorithm removes terrains that are estimated by the model to be easier than the terrain just trained on at each update. This ensures that the algorithm incrementally increases difficulty. This also avoids the problem of the curriculum visiting previously easy terrains that may appear hard after learning on complex environments which require a significant change in policy. The learning process ends when all terrains are used. To assess the performance of each terrain generator's curriculum map, the hardest terrain successfully traversed at each training epoch was recorded. This is shown in Figure 7, and includes training using a combined map with the highest fitness terrain chosen when grid squares overlap between generators. Hardest terrains are measured in terms of how large their feature values are, which is the euclidean feature distance from (0,0). CPPN and Combined significantly outperform the other four generators with very similar large coverage profiles. CPPN and combined are similar in performance.  Diamond Square Noise and the GAN perform similarly, and have similar coverage maps. Worley and Perlin Noise also obtain a similar final difficulty, and have similarly shaped coverage maps although Perlin covers more of the archive. Learning speed is another key determinant of generator performance --- Fig.\ref{fig:fig7}. We measure this by designating terrains at a certain Euclidean distance from the top-right corner of the map, and recording the iteration at which the walker learns one of those terrains.  Table \ref{table:2} shows some interesting results.  First, the Perlin curriculum learns much faster than the Worley curriculum, despite covering comparable archive area.  The high regularity of Perlin Noise may account for this with policies that are more generalisable between Perlin Noise terrains.  Second, although CPPN and combined maps achieve comparable difficulty, a combined map that starts on higher-fitness noise terrains before transitioning to CPPN terrains achieves difficulty milestones much faster than the single CPPN generator.  The result suggests that combining different generators in a single curriculum may be beneficial, however investigating is out of scope for the current study.  Finally, the effect of the map creation and map-traversal algorithm on performance against classic curriculum learning with a single feature (Figure \ref{fig:fig7}(b)). CPPN terrains from the map were sorted by their roughness, and every fifth terrain was used, similar to standard CL by incrementally increasing a difficulty parameter. This trained much slower and did not succeed on terrains as difficult as in the map-based algorithm.

\begin{table*}[ht!]
\centering
\caption{Showing learning speed: the number of training epochs required to learn a designated terrain a given \% away from the maximum possible map distance (70.71).}
\begin{tabular}{|p{2.3cm}| c c c c c c c c c c|}
      \hline
 Terrain Generator & 5\% & 10\% & 15\% & 20\% & 25\% & 30\% & 35\% & 40\% & 45\% & 50\% \\ [5.0ex]
 \hline
 CPPN & \textbf{40} & 80 & \textbf{120} & \textbf{240} & 960 & 4720 & \textbf{7000} &  \textbf{9880} & \textbf{13280} & -  \\
 \hline
 GAN & \textbf{40} & 80 & 1000 & - & - & - & - & - & - & - \\
 \hline
 Perlin Noise & \textbf{40} & \textbf{40} & 160 & 320 & 960 & 8680 & - & - & - & -  \\
 \hline
 Worley Noise & \textbf{40} & 80 & 480 & 2640 & 4160 & 14080 & - & - & - & - \\
 \hline
 Diamond Square & 80 & 160 & 1280 & - & - & - & - & - & - & - \\
 \hline \hline
 Combined & 80 & 120 & 160 & 320 & \textbf{440} & \textbf{3720} & 7280 & 10280 & 15040 & \textbf{29000} \\ [1ex]
 \hline

\end{tabular}
\label{table:2}
\end{table*}

\section{Discussion}

In this paper we investigated the utility of different feature descriptors for map-based curricula.  Using a ground truth of how hard a terrain was for a pre-trained bipedal walker, we showed how several feature descriptors taken from the literature allow for the assessment of meaningful difficulty purely from a terrain mesh.  Using pairwise combinations of features as MAP-Elites feature dimensions, we then showed the effect of feature dimension selection on archive coverage and fill pattern.

In a second step, we used our selected feature descriptors to build a MAP-Elites-based curriculum for different terrain generation algorithms, including direct and indirect representation.  Using our representation-agnostic feature descriptors, we could directly compare archive properties.  We show that CPPNs present the best coverage, and make the most difficult terrains learnable by the agent. GANs were most limited in terms of coverage, with noise generators taking up the middle ground.  Perlin and Worley noise in particular presented useful coverage patterns. Interestingly, Perlin noise allowed for much faster learning (achievement of terrain milestones) than Worley.  Even more interestingly, we see that a combined map presents significantly faster learning than a pure CPPN map, as it learns first on fitter (easier) terrains (where cells are shared with multiple generators) before transitioning to pure CPPN terrains that the other generators cannot reach.

Future work will investigate this, as well as exploring the effects of hyperparameter tuning (all parameters were set to default), as well as the generalisability of these findings to other high-impact problems.  However, purely in terms of learning about how agent-based curriculum RL works, this paper presents several important contributions for the research community.  Future work will also apply the methods presented to the training of real robots, where the effect of generator types and the map-based curriculum may be able to contribute in narrowing the sim2real gap in addition to increasing learning speed. Moreover, game-playing agents could benefit from this curriculum technique. Further exploration of feature descriptors should be conducted in different curriculum contexts to measure their influence on policy performance. We hope this will lead to more terrain generator methods used in curriculum learning, as well as more principled selection of generator.

\bibliographystyle{ACM-Reference-Format}

\section{Introduction}

Curriculum Learning (CL) \cite{narvekar2020curriculum,bengio2009curriculum, TiddBrendan2020GCLf} is a powerful reinforcement learning technique that engenders powerful behaviours by gradually increasing the difficulty of the task to be solved.  This incremental approach has numerous benefits including performance and generalisation ability. To maximise the benefit of a curriculum, it is important to have a training environment that increases in complexity proportional to the learning ability of the agent, and contains relevant features to encourage learning.  Diverse training examples, presented to the learner in an appropriate order \cite{heess2017emergence}, have resulted in the generation of capable, complex behaviours for simulated robots \cite{TiddBrendan2020GCLf, xie2020allsteps}, and aided sim2real transfer of those policies to real robot deployments \cite{lee_learning_2020}.

A typical setup for learning locomotion skills (a popular focus area) involves the creation of a set of terrains of varying difficulty through some terrain generator, typically a parameterised noise function.  The terrains are then presented to the agent, generally in order of difficulty, and the agent learns by solving simpler examples to eventually reach a given target behaviour.  Noise functions are popular candidate terrain generators, as increasing terrain difficulty can be simply realised by increasing values of the noise parameters.  Different generators produce terrains with different geometric features, which has an affect on learning.

To date, the impact of the chosen terrain generator has not been explored, and this is the focus of our paper.  We address the effects of terrain generator on the learning process in two steps.  In step 1, we select a varied range of popular terrain generators from the literature, which includes both direct and indirect encodings. Because the difficulty of an indirectly-coded terrain cannot be ascertained {\em a priori}, we categorise the resulting terrain mesh according to a set of representation-agnostic features selected from related literature.  The selected features become feature descriptors in MAP-Elites, providing a diverse set of high-quality terrains.  Terrains are evolved to fill the archive and generate one curriculum per generator type.   In step two, each curriculum is used to train a bipedal walker, and results compare reachable terrain difficulty and learning speed.  Our approach is illustrated in Figure \ref{fig:fig1}.

We present two main original contributions; (i) the selection of appropriate representation-agnostic feature descriptors, including coverage analysis of curricula evolved using those features, and (ii) analysis of the effects of generator type on learning performance and rate.  Results show the identification of suitable feature descriptors for representation-agnostic terrain-based curriculum learning, and suggest key differences between the  generators, particularly between direct and indirect representations in terms of map coverage.  We provide evidence that a multi-generator approach may be beneficial to the generation of curricula that promote rapid learning.

The remainder of the paper is organised as follows; Section 2 presents pertinent background research.  Section 3 describes our methodology.  Section 4 presents experiments and results, and Section 5 provides a discussion.

\section{Background}

Reinforcement Learning (RL) \cite{kober2013reinforcement} is one of the most commonly used methods to train agent behaviours, and is similar conceptually to an evolutionary algorithm which learns by continually interacting with (and being rewarded by) an environment.  In this study we use Proximal Policy Optimisation (PPO) \cite{ppo} due to its ubiquity as a benchmark method to train agents in complex environments. PPO is a policy gradient method that alternates between sampling data through the agent's interaction with the environment and optimising an objective function using stochastic gradient descent. PPOs main advantage over other policy gradient techniques is the use of multiple epochs of mini-batch updates rather than performing one gradient update per data sample.

Curriculum learning \cite{bengio2009curriculum} is a form of incremental learning \cite{howard2014evolving}, designed to address some common challenges with RL \cite{narvekar2020curriculum}.  Firstly, it allows for agents to solve very hard problems by progressively building competency in easier versions of the problem  \cite{bongard_utility_2010}.  Secondly, it provides a wealth of learning experiences to the agent which can greatly assist with generalisation \cite{KadianAbhishek2020SPDE, HoferSebastian2021SiRa,nichol2018gotta}.   Lee et al. \cite{lee_learning_2020} demonstrate both of these benefits using a particle filter to update parameters of a noise function-based terrain generator, allowing complex terrains to be navigated by a real quadruped after simulated learning.

A guided curriculum approach \cite{TiddBrendan2020GCLf} incrementally removed supporting forces from a bipedal walker's body before increasing terrain complexity and diversity.  Accelerated learning of complex environments is also evidenced with hexapod robots~\cite{9043822}. Miras demonstrated that balancing behaviour could be achieved by training on a tilted plain rather than a flat plain with objects on it, even though balance was not incorporated into the fitness function of the robot~\cite{miras_effects_2019}. Huizinga \cite{huizinga2019evolving} found that ordering of sub-tasks towards learning a difficult task is challenging, and instead proposed the Combinatorial Multi-Objective Evolutionary Algorithm (CMOEA) to simultaneously explore all orderings. This is an effective CL method when subtasks can be clearly defined (e.g. jumping, walking). Xie \cite{xie2020allsteps} demonstrated the critical role of a curriculum to train three bipedal agents to walk on stepping-stone scenarios, with final terrain complexity and learning rate being superior for curriculum approaches compared to non-curriculum RL.  Akkaya \cite{akkaya2019solving} presents a CL approach using automatic domain randomization (ADR), demonstrating vastly improved sim2real transfer compared to a non-curriculum baseline. ADR automatically expands the randomisation range parameterising a distribution over environments. Florensa \cite{florensa2018automatic} showed that using a curriculum generating network it is possible to train an agent to `perform a wide set of tasks without requiring any prior knowledge of its environment'.   Open-ended curricula \cite{zhou2021openended} can learn on terrains that are continually generated during agent learning \cite{poet}.

Procedural Content Generation (PCG) originated in computer graphics and video game design. PCG can create large volumes of high-quality content with controllable randomness, including digital objects, landscapes, levels, textures and 3D models.  PCG has been readily adopted by the machine learning community ~\cite{risi2020increasing} to create a wealth of training data.  Rich and diverse environments encourage agent learning and improve robustness \cite{heess2017emergence}, with works showing that curriculum learning can train several quadrupedal robot policies in parallel to walk over uneven terrains in simulation \cite{rudin2021learning}. Terrain generators have been previously evolved \cite{evoTG,asasas}, however these works focus only on a single representation and do not evaluate terrains in an agent-based learning context.  Also in an evolutionary context, quality-diversity algorithms \cite{pugh2016quality} present an effective method for storing and traversing a curriculum, shown in both robotics and game-playing contexts \cite{Cully_2015} \cite{9300206}.  In particular,the use of aligned feature dimensions within MAP-Elites \cite{mouret2015illuminating} can provide direction to a (stochastic) traversal algorithm.

Overall, we see that curriculum learning is a powerful technique relevant to both evolutionary algorithms and reinforcement learning.  The literature abounds with a smorgasbord of both direct and indirectly-represented generators, including various noise functions as well as CPPNs and GANs.  Additionally, we see many different feature dimensions being used to store the curriculum and permit traversal. However, we identify a significant literature gap in that these different selections of generators and features have not yet been compared.  In this study we compare popular generators and a set of selected feature descriptors for an evolved MAP-Elites based curriculum.  We attempt to inform the selection of appropriate terrain generators and feature dimensions for fellow researchers in the field.  We also demonstrate a curriculum learning approach that simultaneously supports both direct and indirect representations, opening up future work in mixed-generator curricula.

\section{Methodology}

We follow a three-stage process: (i) decide on features, (ii) evolve curricula (iii) learn on curricula.  To allow for the isolated study of generators, we omit other tasks that typically comprise a curriculum (steps, jumps, etc.) and focus purely on locomotion in rough terrains.

\subsection{Generators}

We select 5 popular and diverse generators from the literature: 3 direct (Perlin noise, Diamond Square Noise, Worley Noise) and two indirect (a GAN, and a CPPN). Each generator mapped to a 256x256 resolution pixel map which is converted to a heightmap and then to a solid terrain mesh for simulation.

\subsubsection{Perlin Noise}

Perlin noise was applied to fractal Brownian Motion (fBM). The genome was parameterised into scale [1-100], octaves [1-9], persistence [0.1-0.9], lacunarity [1-3] and a random seed [0-100].

\subsubsection{Diamond Square Noise}

We use a version of the diamond square noise algorithm \cite{diamondsq}.  The four corner points of the heightmap grid are initialised with a random value between -1 and 1 making a square. The following two steps alternate until all the values of the heightmap are assigned:

Diamond step: for each square in the grid, assign the value of the midpoint of that square to be a random percentage \emph{P} of the average of the four corner points, plus a random value \emph{R} between -1 and 1.

Square step:  for each diamond in the grid, assign the value of the midpoint of the diamond to be a random percentage \emph{P} of the average of the four corner points, plus a random value \emph{R} between -1 and 1.

After each iteration of these steps, the range of the random value \emph{R} is reduced using the formula:

\[
\frac{-1}{(steps*D+1)} \leq R \leq \frac{1}{(steps*D+1)}
\]

With $\emph{D} \in [1,10]$, and \emph{steps} corresponding to the number of iterations of the above two steps performed.  \emph{P} is computed as:

\[
Z\leq P \leq 100-Z, Z \in [0,50]
\]

The genome is represented by a random seed, the four initial corner values, percentile (\emph{Z}), and level (\emph{D}).

\subsubsection{Worley Noise}

Worley Noise is typically used to generate procedural textures, and is implemented following \cite{worley}. The genome for the Worley terrain generator consists of a seed for randomisation, the number of feature points $\emph{N} \in [2,400]$ and the index of the point sorted by ascending distance to the current point $\emph{D} \in [0, \frac{N}{4}]$.

\subsubsection{GAN}

The Deep Convolutional GAN (DCGAN) model was trained on real terrain heightmap data with identical resolution to the humanoid in simulation. The original data was in pointcloud format obtained from the OpenTopography website. The ground plane was semantically segmented, and the resulting points interpolated into multiple heightmap patches at the desired resolution. This provided a large amount of high resolution data (30000 256x256 patches) which was used to train the model. The latent vector fed into the generator contained 50 values which formed the genome of the GAN generator.

\subsubsection{CPPN}

The CPPN terrain generator has 2 inputs corresponding to row and column of the heightmap, and one output for the height value. All remaining CPPN settings were taken from the PicBreeder paper \cite{picbreeder}.

\subsection{Features}

We explore a range of generic terrain descriptors for use as MAP-Elites feature descriptors that allow both direct and indirect generators to be compared. Features were aligned to terrain traversability to provide continuity to the final generated curricula, with enough variation to create diversity in the population.  Features originate from a range of non-RL, non-evolutionary fields, mainly from surveying where they are used to categorise real terrains. We use:

\begin{enumerate}
    \item Terrain Ruggedness Index (TRI) \cite{tri} - measures the average difference between a pixel and its 8 neighbouring pixels.
    \item Topographic Position Index (TPI) \cite{tpi} - measures the difference between each pixel and the mean of its 8 neighbouring pixels
    \item Roughness \cite{roughness} - measures the maximum difference between a pixel and its 8 neighbouring pixels
    \item Traversability Estimation Model implemented in \cite{traversability} - A trained model that predicts the traversability of terrains by outputting a traversability map. This model had been trained on different types of terrains to the terrain generators used in this paper. The orientation input to the model was set to 0 as this was the direction the humanoid traversed.
\end{enumerate}

Once generated, a terrain mesh is tagged with values for each feature descriptor.  Settings for kernel size are determined in Section 4.  Roughness, TPI and TRI use a stride length of 2.  The average of the output from each feature was taken as the overall measure.  Initial experimentation found that an archive discretisation of 50 bins per descriptor presented meaningful but achievable differences between terrains in neighbouring cells.

\subsection{Evolutionary Algorithm}

We evolve curriculums for each generator, and compare different pairs of feature dimensions.  Per treatment, we run MAP-Elites for 5000 generations, with 100 random initial genomes.  Per generation, 20 new terrains were generated by randomly selecting from the current members of the archive, mutating, generating feature values, and adding back into the archive as in Figure \ref{fig:fig2}.

\begin{figure}[h!]
\centering
\includegraphics[width=1\columnwidth]{images/MAP_ELITES.png}
\caption{Overview of library generation.  Features must be calculate in simulation before the terrain can be added to the library.}
\label{fig:fig2}
\end{figure}

For noise emitters, each gene in the genome had a mutation probability of 0.35, and mutation altered the allele by $\pm$ a value taken from a normal distribution with covariance set to 10\% of its range.  CPPN mutation followed NEAT \cite{picbreeder}.  The GAN's latent space was stored in 50 variables, which with P=0.07 were mutated by a value taken from a normal distribution with covariance set to 10\% of its range (heuristically determined).

Fitness is set to 1.0 - the difficulty of the generated terrain.  Difficulty is assessed using the base policy of the Bipedal Walker that is trained on flat ground\cite{coumans2017pybullet} and simulated in PyBullet. Difficulty of a terrain is calculated as the average distance travelled out of the best 5 of 20 attempts with random joint initialisations, normalised between 0 and 1. The 5 best were taken as some initialisations were too extreme and caused a large amount of noise in the fitness estimation.  When replacing individuals in a MAP-Elites cell, we kept the lowest difficulty terrain and deleted the higher difficulty one.  This generated a smoother curriculum in terms of fitness, whilst also removing many impossible terrains.  A second step removed the remaining impossible terrains, following Algorithm 1.  Representative terrains for two generators (CPPN and Perlin noise) that achieve difficulties of $\approx$0.1, $\approx$ 0.5, and $\approx$ 0.9 are shown in Figure \ref{fig:fig3}.

\begin{algorithm}[h!]
\caption{Check terrain traversability.}
\begin{algorithmic}
\Procedure{Check Terrain}{$hm$}\Comment{Terrain heightmap}
\State $threshold\gets \frac{robot Height}{3}$\Comment{max incline in metres}
\State $k\gets 26$\Comment{Approx. step length of robot}
\State Initialise  differences result [hm.rows-k,hm.cols-k]
\For{r in [0, hm.rows - kSize)}
\For{c in [0, hm.cols - kSize)}
\State differences[r,c]$\gets$

maximum\_difference(hm[r:r+k,c:c+k])
\EndFor
\EndFor
\State \textbf{return} $max(differences) \leq threshold$
\EndProcedure
\end{algorithmic}
\end{algorithm}

\begin{figure}[ht!]
\begin{center}
\centering
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/cppn_1.png} }
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/cppn_5.png} }
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/cppn_9.png} }\\
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/noise_1.png} }
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/noise_5.png} }
 \subfloat[]{\includegraphics[width=0.3\columnwidth]{images/difficulty_pergen/noise_9.png} }\\
\end{center}
\caption[]{Representative pixel maps with approximate difficulties of 0.1, 0.5, 0.9 respectively (L-R) for (a)-(c) an indirect representation (CPPN) and (d)-(f) a direct representation (Perlin noise).}
\label{fig:fig3}
\end{figure}

\section{Experimentation}
\subsection{Experiment 1: Features and Curriculum}

Initial experimentation compared feature descriptors with different kernel sizes to measure their alignment to terrain difficulty (fitness). This alignment was ideal for constraining the solution search space to produce a strong traversability gradient in the MAP-Elites grid. Without a strong gradient, creating an effective curriculum becomes difficult. Roughness, TPI and TRI had similar results with TPI being most correlated. Results are shown in Figure \ref{fig:fig4}. We used TPI in learning experiments as it was most aligned, and use Roughness since TPI and TRI (k = 30) were too similar (r = 0.94/0.95) to produce sufficient map diversity. A kernel size of 30 was chosen as it had consistently high correlations to terrain difficulty.  These features could then be used for arbitrary meshes with confidence that they were meaningful metrics.  The traversability model ("Traversability") had the weakest correlation to difficulty so this was not used further.

\begin{figure}[ht!]
\centering
\includegraphics[width=0.95\columnwidth]{images/correlation_heatmap_cppn.png}
\includegraphics[width=0.95\columnwidth]{images/correlation_heatmap_noise.png}
\caption[]{Heatmaps showing Pearson correlation of all considered terrain characterisation features including ground truth: terrain difficulty, for (a) CPPN and (b) Perlin noise. }
\label{fig:fig4}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=0.9\columnwidth]{images/upload.png}
\caption[]{Feature coverage of each generator after each experiment.}
\label{fig:fig5}
\end{figure}

\begin{figure*}[ht!]
\centering
\includegraphics[width=2\columnwidth]{images/new_images/allcoveragefigs_landscape.png}
\caption[]{MAP-Elites archive coverage for three combinations of features: TPI/TRI (row 1), roughness/TRI (row 2), roughness/TPI (row 3) for Diamond Square noise (a), Perlin noise (b), Worley noise (c), GAN (d) and CPPN (e) terrain generators after evolution. Colour corresponds to fitness, measured as the average performance of the base policy humanoid traversing each terrain.  Dark colours represent fitter behaviours.}
\label{fig:fig6}
\end{figure*}

The coverage of each terrain generator is shown in Figure \ref{fig:fig5}.  Pairwise feature comparisons are shown in Figure \ref{fig:fig6}.  All coverage maps show a strong gradient in terrain traversability, with it declining as the map feature values increase. This suggests the chosen features are very representative of traversability. Some variation exists between the different features, but the most significant basis of variation is in the euclidean distance between feature values and (0,0). Each terrain generator presents a characteristic shape and amount of coverage on the map. Table 1 shows the comparative coverage between generators, with CPPN having significantly higher coverage, and able to create more difficult and more diverse terrains than the other generators.    Interestingly the other indirect emitter, GAN, presents the lowest coverage of all. The GAN was trained on real terrains and therefore can only output similar meshes.  The CPPN is unrestricted in this manner.  Perlin noise shows the strongest diagonal mapping to the feature dimensions and presents the best direct encoding, followed by Worley noise.  Diamond-square noise can be seen to struggle to generate reasonable values for TPI.  Feature ranges were identical between generator coverage maps for fair comparison. While the total terrains in each map vary significantly based on coverage, adjusting feature ranges to account for this does not help learning as terrains with comparable features are similarly traversable and are skipped.

\begin{table}[h!]
\centering
\caption{Coverage percent of total map for feature combinations with each terrain generator; maps from Figure \ref{fig:fig5}.}
\begin{tabular}{|p{2.3cm}| c c c|}
      \hline
 Terrain Generator & Roughness/TPI & Roughness/TRI & TPI/TRI \\ [0.5ex]
 \hline
 CPPN & 23.16\% & 15.52\% & 14.92\% \\
 \hline
 GAN & 2.48\% & 2.48\% & 1.48\% \\
 \hline
 Perlin Noise & 3.92\% & 3.92\% & 2.80\% \\
 \hline
 Worley Noise & 2.88\% & 1.84\% & 2.12\% \\
 \hline
 Diamond Square & 3.12\% & 1.36\% & 1.88\% \\
 \hline \hline
 Combined & 23.64\% & 15.56\% & 15.56\% \\ [1ex]
 \hline
\end{tabular}

\label{table:1}
\end{table}

\subsection{Experiment 2: Bipedal Walker Learning}

\begin{figure}[ht!]
\centering
\subfloat[]{\includegraphics[width=1.0\columnwidth]{images/11.png}}\\
\subfloat[]{\includegraphics[width=1.0\columnwidth]{images/22.png}}
\caption[]{Illustrative performance comparison between (a) the different terrain generators, and (b) CPPN terrains between the proposed map-based curriculum algorithm and incrementally increasing roughness with standard CL, showing the most difficult terrain completed at the current training iteration. Plots finish when 30k epochs is reached or IT\&E has excluded all terrains. Averaged over 10 trials, with confidence shown for $\pm$1 standard error.}
\label{fig:fig7}
\end{figure}

In the second experiment we carry out a typical learning experiment on the evolved curricula.  We trained the walker from the base policy using the PyTorch PPO implementation from SpinningUp \cite{SpinningUp2018} with default parameters with a maximum of 30,000 epochs.  The base policy provides the prior for IT\&E.  Traversal of the map is the same Gaussian Process (GP) technique as described by Cully et al. \cite{gaussianprocess}, using the same recommended parameters: $\rho$ = 0.4, $\alpha$ = 0.9, $\kappa$ = 0.05, ${\sigma^2}_{noise}$ = 0.001, and the same covariance function (Matern kernel). PPO trains for 40 epochs at a time until the humanoid reaches the 90\% traversability threshold (90\% of maximum fitness achieved) or when there had been 100 traversability evaluations on the terrain. Once this condition is met, the GP model finds the highest fitness (easiest) terrain from evaluations with the updated PPO model. However, unlike the original IT\&E, our algorithm removes terrains that are estimated by the model to be easier than the terrain just trained on at each update. This ensures that the algorithm incrementally increases difficulty. This also avoids the problem of the curriculum visiting previously easy terrains that may appear hard after learning on complex environments which require a significant change in policy. The learning process ends when all terrains are used. To assess the performance of each terrain generator's curriculum map, the hardest terrain successfully traversed at each training epoch was recorded. This is shown in Figure 7, and includes training using a combined map with the highest fitness terrain chosen when grid squares overlap between generators. Hardest terrains are measured in terms of how large their feature values are, which is the euclidean feature distance from (0,0). CPPN and Combined significantly outperform the other four generators with very similar large coverage profiles. CPPN and combined are similar in performance.  Diamond Square Noise and the GAN perform similarly, and have similar coverage maps. Worley and Perlin Noise also obtain a similar final difficulty, and have similarly shaped coverage maps although Perlin covers more of the archive. Learning speed is another key determinant of generator performance --- Fig.\ref{fig:fig7}. We measure this by designating terrains at a certain Euclidean distance from the top-right corner of the map, and recording the iteration at which the walker learns one of those terrains.  Table \ref{table:2} shows some interesting results.  First, the Perlin curriculum learns much faster than the Worley curriculum, despite covering comparable archive area.  The high regularity of Perlin Noise may account for this with policies that are more generalisable between Perlin Noise terrains.  Second, although CPPN and combined maps achieve comparable difficulty, a combined map that starts on higher-fitness noise terrains before transitioning to CPPN terrains achieves difficulty milestones much faster than the single CPPN generator.  The result suggests that combining different generators in a single curriculum may be beneficial, however investigating is out of scope for the current study.  Finally, the effect of the map creation and map-traversal algorithm on performance against classic curriculum learning with a single feature (Figure \ref{fig:fig7}(b)). CPPN terrains from the map were sorted by their roughness, and every fifth terrain was used, similar to standard CL by incrementally increasing a difficulty parameter. This trained much slower and did not succeed on terrains as difficult as in the map-based algorithm.

\begin{table*}[ht!]
\centering
\caption{Showing learning speed: the number of training epochs required to learn a designated terrain a given \% away from the maximum possible map distance (70.71).}
\begin{tabular}{|p{2.3cm}| c c c c c c c c c c|}
      \hline
 Terrain Generator & 5\% & 10\% & 15\% & 20\% & 25\% & 30\% & 35\% & 40\% & 45\% & 50\% \\ [5.0ex]
 \hline
 CPPN & \textbf{40} & 80 & \textbf{120} & \textbf{240} & 960 & 4720 & \textbf{7000} &  \textbf{9880} & \textbf{13280} & -  \\
 \hline
 GAN & \textbf{40} & 80 & 1000 & - & - & - & - & - & - & - \\
 \hline
 Perlin Noise & \textbf{40} & \textbf{40} & 160 & 320 & 960 & 8680 & - & - & - & -  \\
 \hline
 Worley Noise & \textbf{40} & 80 & 480 & 2640 & 4160 & 14080 & - & - & - & - \\
 \hline
 Diamond Square & 80 & 160 & 1280 & - & - & - & - & - & - & - \\
 \hline \hline
 Combined & 80 & 120 & 160 & 320 & \textbf{440} & \textbf{3720} & 7280 & 10280 & 15040 & \textbf{29000} \\ [1ex]
 \hline

\end{tabular}
\label{table:2}
\end{table*}

\section{Discussion}

In this paper we investigated the utility of different feature descriptors for map-based curricula.  Using a ground truth of how hard a terrain was for a pre-trained bipedal walker, we showed how several feature descriptors taken from the literature allow for the assessment of meaningful difficulty purely from a terrain mesh.  Using pairwise combinations of features as MAP-Elites feature dimensions, we then showed the effect of feature dimension selection on archive coverage and fill pattern.

In a second step, we used our selected feature descriptors to build a MAP-Elites-based curriculum for different terrain generation algorithms, including direct and indirect representation.  Using our representation-agnostic feature descriptors, we could directly compare archive properties.  We show that CPPNs present the best coverage, and make the most difficult terrains learnable by the agent. GANs were most limited in terms of coverage, with noise generators taking up the middle ground.  Perlin and Worley noise in particular presented useful coverage patterns. Interestingly, Perlin noise allowed for much faster learning (achievement of terrain milestones) than Worley.  Even more interestingly, we see that a combined map presents significantly faster learning than a pure CPPN map, as it learns first on fitter (easier) terrains (where cells are shared with multiple generators) before transitioning to pure CPPN terrains that the other generators cannot reach.

Future work will investigate this, as well as exploring the effects of hyperparameter tuning (all parameters were set to default), as well as the generalisability of these findings to other high-impact problems.  However, purely in terms of learning about how agent-based curriculum RL works, this paper presents several important contributions for the research community.  Future work will also apply the methods presented to the training of real robots, where the effect of generator types and the map-based curriculum may be able to contribute in narrowing the sim2real gap in addition to increasing learning speed. Moreover, game-playing agents could benefit from this curriculum technique. Further exploration of feature descriptors should be conducted in different curriculum contexts to measure their influence on policy performance. We hope this will lead to more terrain generator methods used in curriculum learning, as well as more principled selection of generator.

\bibliographystyle{ACM-Reference-Format}

redpajama_set_name: RedPajamaArXiv


text:
KC to SL, Berger enjoying senior season
By Jori Kapla on March 14, 2019 • ( 1 Comment )
Considering the circumstances, life doesn't get much better than this for Michael Berger.
This past summer, he was living in Kansas City when he found out his 9-year-old cousin had cancer and his grandmother was getting divorced.
"It was pretty much my decision to move," Berger said. "So I moved in with my grandma and my mom also moved in with us."
Berger enrolled at Sheboygan Lutheran for his final year of high school while his father continued to work in Kansas City. The two will be reunited this weekend in Madison, as Berger and the Crusaders try to win the Division 5 boys state basketball championship.
"This is something I've been waiting for," Berger said. "I've always wanted to play in a state tournament. I've prepared my whole life to play in this type of game by dedicating my life to play basketball."
Even better for Berger, who has never played at state before, his cousin is cancer free.
"I'm pretty sure he's all good now," Berger said. "It's pretty crazy how everything is working out. I get to play basketball and my family is doing great right now."
Berger's mother got a job at Lutheran – his father graduated there – and he says leaving Kansas City was not that hard.
"I knew that I was coming here to be around family and most of my family lives up here anyway," he said.
Berger did not know anyone from the team, however, but started playing basketball with them in August. He slowly started to realize state was a possibility.
"Everybody started saying how good we were and stuff, so I prepared myself to think we were going to make it," Berger said. "That was just the mindset I had."
However, Berger was not a starter and was a back-up to freshman guard Casey Verhagen.
Some seniors would not be happy about that. Not Berger.
"He's playing a role as a back-up point guard to a freshman, but Michael's always been super respectful of Casey and our program," Lutheran coach Nick Verhagen said. "He adjusts and adapts to a whole new role and recognizes his value. When his name is called, he jumps in and does exactly what needs to be done. He's a huge reason we got over Hustisford (in the sectional semifinal) the other day. He stepped in during a really hostile situation and we were down a bunch, and he played really well on both ends of the court, got us back in the game, and lifted us to a double-digit lead because of his leadership and his experience."
Berger has appeared in every game and is averaging 3.7 points a contest.
Casey Verhagen averages 10.5 points per game and recently set the school record for most assists in a season.
"I think it's awesome to have him as a senior," Casey said. "A freshman starting over (a senior) is not easy, but I think he's embraced his role and pushes us in practice."
Tagged as: 2019, SL-STATE, Sports March 12 - 25 2019
Remember Dekker's shot? Current Crusaders do
Do the Macks even have a chance?
lawoman15 says:
This is my Grandson and I am so proud oh his contribution unions to Lutheran High School boys basketball Varsity State Champion Team!