Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments

The capability of reinforcement learning (RL) agent directly depends on the diversity of learning scenarios the environment generates and how closely it captures real-world situations. However, existing environments/simulators lack the support to systematically model distributions over initial states and transition dynamics. Furthermore, in complex domains such as soccer, the space of possible scenarios is infinite, which makes it impossible for one research group to provide a comprehensive set of scenarios to train, test, and benchmark RL algorithms. To address this issue, for the first time, we adopt an existing formal scenario specification language, SCENIC, to intuitively model and generate interactive scenarios. We interfaced SCENIC to Google Research Soccer environment to create a platform called SCENIC4RL. Using this platform, we provide a dataset consisting of 36 scenario programs encoded in SCENIC and demonstration data generated from a subset of them. We share our experimental results to show the effectiveness of our dataset and the platform to train, test, and benchmark RL algorithms. More importantly, we open-source our platform to enable RL community to collectively contribute to constructing a comprehensive set of scenarios.

Introduction
Deep reinforcement learning (RL) has emerged as a powerful method to solve a variety of sequential decision-making problems, including board games [31,32], video games [24,36], and robotic manipulation [18]. These successes rely heavily on good simulation environments and benchmarks [2,3,5,7,34]. However, regardless of a long history of RL benchmarks, the existing RL environments/simulators often fail to generate realistic scenarios portraying a rich variation of environment dynamics. This lack of diversity severely limits the ability of reinforcement learning, primarily resulting in lack of generalization: one of the biggest challenges in deep reinforcement learning [4,5,20,21]. To be able to train robust agents that generalizes over different variations of the environment dynamics, we must train RL agents on scenarios that represent the desired variation.
In literature, several techniques have been adopted to generate a rich variation of learning scenarios [4,5]; however, none of them provides a support to easily model and generate interactive scenarios. To properly benchmark the capabilities of trained RL agents, we need a systematic way to model and generate scenarios with varying levels of difficulties, which are determined by opponents' initial conditions and a fined-grained control over their behaviors. Furthermore, collecting quality demonstration data (e.g. to train RL agents or benchmark imitation learning algorithms) becomes expensive because humans have to play the game numerously to generate data, potentially with errors.
Hence, there is a clear need to support users to intuitively model and generate various scenarios at their demands for accurate benchmark and quality dataset collection.
In this paper, we introduce our platform, Scenic4RL, which offers a systematic and transparent way of specifying and controlling environment dynamics. We adopt an existing formal scenario specification language, SCENIC [9,10], to programmatically model and generate scenarios to benchmark RL algorithms. SCENIC provides an intuitive abstraction over environment behaviors to easily model interactive scenarios. We interfaced SCENIC to Google Research Football [19], an stochastic environment that requires the agents to learn strategies at multiple levels of abstraction, from learning to pass, to high level tactics, and presents theoretically infinitely many variations of the environment dynamics (e.g., player positions and behaviors).
Our contributions are as follows: • We introduce a platform which supports intuitive modeling and generation of scenarios programmatically using SCENIC that enables creating diverse scenarios and easy generation of quality demonstration data, at users' demand.
• We introduce a dataset consisting of i) 13 mini-game scenarios along with reusable model and behavior libraries ii) semi-expert stochastic policies (and data generated from them) written in SCENIC for select 5 scenarios, and iii) 18 test scripts to evaluate generalization • We evaluate three use cases of our platform and the dataset by benchmarking scenarios, behavior cloning with demonstration data from our stochastic policies, and testing generalization.

Google Research Football Simulator
The Google Research Football (GRF) simulator provides an environment to train and test RL agents to play soccer. The environment setup is as the following. All the players on the field are controlled by GRF's built-in, rule-based AI bots and RL agents. The simulator dynamically determines which of the RL team players are to be controlled by RL agents based on their vicinity to the ball. GRF provides 11 offense scenarios to benchmark RL agent performance.

Scenario Specification Language: Scenic
SCENIC is an object-oriented, probabilistic programming language which is designed to easily model and generate scenarios. In SCENIC, a scenario is a distribution over spatio-temporal configuration of agents and objects. Its natural English-like syntax and semantics let users intuitively specify complex spatial and temporal relations among objects (see Sec. 3 for an example). Furthermore, SCENIC allows users to construct object model, action, and behavior libraries and simply reference these from SCENIC programs. This feature expedites the modeling process because users do not have to encode a scenario from scratch. We cover the libraries in detail in Sec. 4.1. Finally, users can specify distributions over states and player behaviors, making a SCENIC program much more expressive. Prior distribution is specified over an object's initial states like position and behavior according to its role. A scenario is generated by sampling an initial state at start and sampling an action for each object during simulation run time according to their specified behavior. If users instantiates an object but do not specify any initial states or behavior, those are sampled from the prior distribution on the object. For further details about the language please refer to [10].
3 The SCENIC4RL Platform for Reinforcement Learning

The Need for Scenario Specification Language for RL
The capability of an agent depends largely on the quality of the scenarios it is trained on. However, the existing simulators typically do not provide a systematic way to generate rich-diverse scenarios, limiting the abilities of reinforcement learning algorithms. For example, the GRF's scenario python program, shown in Figure 1(c), used to generate the Pass and Shoot scenario ( Fig. 1(a)), initiates There is no support to position the players intuitively by specifying their geometric relations (e.g. "there is a yellow player ahead of a blue player"). Rather the user has to precisely compute initial positions. Also, there are no easy-to-use interface to modify the opponent behaviors. Moreover, the dynamics of the existing environments often lack transparency, which can make reasoning about them difficult in some cases, e.g., it is often not clear how different (or, similar) the train and test environments are [5]. To alleviate these problems, we propose our platform SCENIC4RL , which provides an extensive control over the environment dynamics to generate scenarios on user-demand via SCENIC programs. SCENIC4RL offers a number of advantages over traditional simulators/environments: which makes the scenario programs interpretable. 6. Transparency: Being able to specify/modify environment dynamics programmatically also improves transparency of the generated scenarios, e.g., users can reason about the difference/similarity of their train and test environments with more clarity.

Modeling Scenarios with SCENIC
Formally, a scenario is a Markov Decision Process (MDPs) [33] defined as a tuple (S, A, p, r, ρ 0 ), with S denoting the state space, A the action space, p (s |s, a) the transition dynamic, r (s, a) the reward function, and ρ 0 the initial state distribution. Given the state and action spaces as defined by the GRF environment, a SCENIC program defines (i) the initial state distribution, (ii) the transition dynamics (specifically players' behaviors), and (iii) the reward function. Hence, users can exercise extensive control over the environment with SCENIC.
Modeling Initial State Distribution Users can intuitively specify initial state distributions with SCENIC's high-level syntax. For example, refer to the full SCENIC program in Fig. 1(d) which describes a more generalized version of the Pass and Shoot scenario modelled in Fig. 1(c). In line 12-22, the initial state distribution is specified. The SCENIC's spatial syntax highlighted in yellow let users to intuitively model spatial relations among objects, rather than having to hand-code positions as in the GRF's scenario. Here, Left represents the yellow team, Right the blue, and the two following abbreviated capital letters indicate the player role. As discussed in Sec. 2.2, object models (e.g. players, regions, and directions) that are undefined in the program are referenced from the model library, expediting the modeling process. For players instantiated without specification as in line 12 and 22, their state and behavior default to prior distribution, which saves repetitive state specification.
Modeling Transition Dynamics One can flexibly modify transition dynamics of the environment by specifying the behaviors of any players using SCENIC. Take the same example SCENIC program in Fig. 1(d) as above. Line 1-10 models two new behaviors. A behavior can invoke another behavior(s) with syntax do, succinctly modeling a behavior in a hierarchical manner. Users can assign distribution over behaviors as in line 2. The interactive conditions are specified using try/interrupt block as in line 5-10. Semantically, the behavior specified in the try block is executed by default. However, if any interrupt condition is satisfied, then the default behavior is paused and the behavior in the interrupt block is executed until completion and then the default behavior resumes. These interrupts can be nested with interrupt below has higher priority. In such case, the same semantics is consistently applied.
Rewards SCENIC has a construct called monitor, which can be used to specify reward functions. The reward conditions in the monitor is checked at every simulation step and updates the reward accordingly. Figure 2 shows an overview of the overall architecture of SCENIC4RL. The architecture can be divided into two parts, i) RL interface, through which the RL algorithms interact with SCENIC4RL and ii) the SCENIC Server, which executes a SCENIC program and governs the simulation by interacting with the underlying simulator. We implemented our interface using OpenAI Gym API [3], which allows SCENIC4RL to be used seamlessly with all the existing standard RL frameworks.

Architecture
For each simulation/episode, the SCENIC server first samples an initial state from the SCENIC program to start a new scenario in the GRF simulator, and updates its internal model of the world (e.g., player and ball positions). From then on, a round of communication occur between the RL algorithm and SCENIC server, with the RL interface at the middle. At each timestep, the gym interface takes in the action(s) for the RL agent and passes them to the SCENIC server. The SCENIC server in turn computes actions for all the remaining non-RL players-the players not controlled by the RL agent-and then executes all these actions (of both the RL and non-RL players) in the simulator. The SCENIC server then receives the observation and reward from the simulator, updates the internal world state, and then passes them back to the RL algorithm. This interaction goes on till any terminating conditions as specified in the scenario script is satisfied.

SCENIC Libraries
As discussed in Sec. 2.2, SCENIC libraries consist of three types: model, action, and behavior libraries.
The SCENIC libraries significantly reduces users' time to model complex scenarios by helping them re-use the set of models, actions, and behaviors in the libraries, rather than having to write scenario from scratch.
The model library defines the state space. It defines players with distribution over their initial state according to their roles and GRF's AI bot is assigned by default to all player behavior. These prior distribution over the initial state and behavior can be overwritten in the SCENIC program. The model library also defines region objects such as goal and penalty box regions as well as directional objects in compass directions. The action library defines the action space as determined by the GRF simulator. These action space consists of movement actions in eight compass directions, long/short/high pass, shoot, slide, dribble, and sprint.
The behavior library consists of behaviors and helper functions that represent widely used basic skills in soccer. These behaviors include give-and-go, evasive zigzag dribble to avoid an opponent's ball interception, dribbling to a designated point and shooting, shooting towards the left or right corner of the goal, etc. Additionally, the behavior library also include useful helper functions such as identifying nearest opponent or teammate, whether there is an opponent near the running direction of a dribbler, etc. Please refer to our open-sourced repository for more details.  and a test scenario to assess generalization (rightmost). The RL team is yellow and the opponent, blue. The assigned opponent behaviors are highlighted with light blue arrows. Uniformly random distribution is assigned over a specific region for each player. These regions are highlighted boxes.

SCENIC programs
Please refer to our Supplement for detailed descriptions of all our scenarios and policies.
New Benchmark Scenarios We modelled 13 new benchmark scenarios. Nine of them are defense scenarios which are nice complement to GRF's offense-only scenarios. The remaining four scenarios are additional distinct offense scenarios that enriches GRF's benchmark scenarios. We varied the initial state distribution and assigned GRF's AI bot behavior by default. However, if the AI bot did not exhibit the behaviors we expected, we modelled the behaviors of environment agents to create such scenarios. For example, a few of our defense scenarios with specified behaviors are visualized in Fig. 3. In these scenarios, the opponent AI bot did not exhibit expected behaviors so we specified behaviors in the SCENIC programs as described in the sub-captions. In 1vs1 scenario, the opponent AI bot was not aggressive enough to attack the goal and was easily pushed out of the penalty box. In 2vs2 scenario, the opponent initially with the ball did not quickly advance the ball to its teammate closer to the goal post via high pass. In 3vs3, the opponent player with the ball on the leftmost side of the field did not cross the ball to its teammates.

Scenarios for Testing Generalization
We provide scripts to test generalization of GRF's 5 academy scenarios and all of our 13 new benchmark scenarios from our dataset. The scripts test how robust the trained agents are to perturbations in the initial state. We assigned distribution over the initial state while keeping the formation of players and their behaviors in each scenario intact. For example, for testing generalizability of an RL agent trained in the Pass and Shoot scenario (Fig. 1a), we instantiated the yellow and the blue players on the symmetric right side of the field instead of the left and kept the other initial state distribution the same (Fig. 3d).
Probabilistic Scenic Policies and Offline Dataset SCENIC4RL also allows users to incorporate their domain knowledge by writing probabilistic policies, which can be used as a means to generate rich demonstration data. Users can benefit from using our behavior library to write their policies, instead of having to write them from scratch. In our work, we present simple SCENIC policies for three GRF academy scenarios [19] and two of our new benchmark scenarios in which a proximal policy optimization (PPO) algorithm [29] had relatively poor training performance. These policies are not to be considered "expert", as they do not solve the scenarios consistently. However, these simple policies can be considered as guidance/ semi-expert policies, which can be utilized for better training. We then generate very small datasets of demonstration from these policies, each containing 10K samples collected from the successful (i.e. the agent scores) trajectories. Each sample is a < o, a > pair, i.e., specifying the action a the SCENIC policy took on the observation o. In our experiments we show that, such small offline dataset can expedite the RL training significantly.

Towards Crowd Sourced Dataset
A current limitation of our benchmark dataset is that it does not provide a comprehensive set of scenarios and policies that cover all realistic soccer scenarios and policies on the field. In fact, there are infinitely many different possible scenarios. We address this issue by open-sourcing our platform and dataset, https://github.com/scenic4RL/scenic4rl/tree/v1, to allow other researchers to contribute their own scenarios and policies as SCENIC programs. A good crowdsourced dataset will cover much more scenarios than a research group can generate. In fact, the effort to crowdsource a dataset in RL domain has been low due to the lack of support for environment modeling. We share our platform to encourage it.

Evaluation
In this section, we evaluate our platform SCENIC4RL and the proposed datasets using three RL usecases. First, we benchmark our proposed scenario dataset-using the widely used reinforcement learning algorithm PPO [29]-to enable researchers to compare their algorithms. Then, we evaluate these PPO trained agents using our test dataset, which tests their generalization capabilities against varying initial state distribution. Finally, we evaluate the usefulness of the proposed SCENIC policies by showing significant improvement in training performance.

Experimental Setup
We run PPO on a single GPU machine (NVIDIA T4) with 16 parallel workers on Amazon AWS. Unless otherwise specified all the PPO training are run for 5M timesteps and repeated for 3 different seeds. For all the experiments, we use the stacked Super Mini Map representation for observations The error bars represent 95% bootstrapped confidence intervals -a 4x72x96 binary matrix representing positions of players from both team, the ball, and the active player-and the scores as rewards, i.e., +1 when scoring a goal and −1 upon conceding, from [19]. Similar to the academy scenarios from [19], we also conclude a game when one of the following happens: either of the team scores, ball goes out of the field, or, the ball possession changes. For further details, including hyperparameters and network architecture, we refer readers to the Supplementary Materials (Section Details on Experimental Setup and Training).

Mini-game Scenario Benchmark
We benchmark our mini-game scenarios by training agents with PPO for 5M timesteps. Figure 4 shows the average goal differences for all the scenarios. Note that we propose both offense and defense scenarios. For all these scenarios, we end the game if one of the teams score. Hence, the goal difference can range between -1 to +1. For the offense scenarios, a well trained agent is supposed to score consistently achieving a goal difference close to +1. On the other hand, a well trained agent should achieve a goal difference close to 0 for successfully defending the opponents. From the graph it can be seen that the proposed scenarios offer a varied levels of difficulties. For example, PPO consistently achieves goal difference of around 0.7 for the EASY CROSSING scenario, but barely learns anything for HARD CROSSING. In case of the defense scenarios, the results also show a varied range of difficulty, GK VS OPPONENT scenario being be easiest. Due to the stochasticity of the GRF environment [19], we observed variations in training performance, with the most variation for the 11 VS GK scenario, reaching 0.93 for one run while failing to learn anything for the other two runs.

Testing for Generalization
As discussed in Sec. 4.2, we introduce a dataset of test scripts to evaluate agents for their robustness against varying initial conditions. Fig. 5 compares the trained agents' performance in training and test scenarios. As expected, we observe a noticeable drop of performance in the GRF academy and offense scenarios (Fig. 5a). However, for defense scenarios, the drop was not as significant. To further investigate, we evaluated random RL policy performance in the test scenarios as shown in Fig. 5b.
Comparing the random and the trained RL agent's policies in the test scenarios, we observed that the trained RL agent noticeably outperforms the random policy across all test scenarios. This clearly indicates that the trained RL agent policy is generalizing to unseen test scenarios. Note that unlike most of the offense scenarios in Fig. 5a with fixed initial state, all the defense scenarios assigned distributions over the initial state. We conjecture that this difference helped the RL agent to generalize better to unseen yet similar defense scenarios.

Pretraining with Semi-expert Demonstrations
In these section we evaluate the SCENIC policies and the demonstration data generated by them, as discussed in 4.2. For each scenario, we first use behavior cloning to train an agent using 8K randomly (a) Offense and select GRF academy scenarios (b) Defense scenarios Figure 5: Evaluation of PPO agents' generalization against varying initial conditions. For most of the academy and offense scenarios we observe a significant drop in performance. However, for the defense scenarios the difference in train and test scenarios is not that significant. Hence, we compare the trained agents further with random agents in the test environment and observe clear sign of generalization. Figure 6: Performance of PPO agents trained with and without any expert data, along with the performance of corresponding SCENIC and behavior cloned policies. We see significantly better training for three of the scenarios, while the rest two achieves comparable performance, highlighting the usefulness of the proposed SCENIC policies.
selected samples from the 10K samples provided in the demonstration dataset . We then continue the training using PPO for 5M timesteps. Figure 6 compares the training performance of these agents against the agents that were trained with PPO only. We notice that, even with such a low volume of demonstration data, we can train much better agents and can solve scenarios which were otherwise unsolved. The experimental results thus suggests, with probabilistic SCENIC policies we can generate rich quality demonstration data to substantially increase training performance, which can be particularly useful in practice for environments like GRF which requires a huge amount of compute resource.

Related work
Improving Environment Dynamics for Generalization: In literature, several techniques have been adopted to generate a rich variation of learning scenarios, primarily to promote or, ensure generalization. Techniques such as changing background with natural videos [38], introducing sticky actions [22] have been attempted, but are not robust enough. To ensure generalization, [21] and [30] generated training and testing scenarios by randomly sampling from different regions of parameter space. Similar to supervised learning, the use of separate train and test sets have also been adopted [25,5,4,17], typically using techniques such as Procedural Content Generation, which has traditionally been used to automatically generate levels in video games. However, most of these focus on discrete domain, typically the dataset generation process is opaque, and it can be difficult to quantify or, reason about how different (or, similar) these train and test sets are.
Formal Scenario Specification Languages A few scenario specification languages have been proposed in the autonomous driving domain since the introduction of SCENIC. Paracosm language [23] models dynamic scenarios with reactive and synchronous model of computation. The Measurable Scenario Description Language (M-SDL) [8] shares common features as SCENIC to model interactive scenarios. In contrast, however, SCENIC provides a much higher-level, probabilistic, declarative way of modeling, and it is not specific to autonomous driving domain.
Recently, various RL benchmarks designed for various purposes have been proposed. Cobbe et al. [5] presented a suite of game-like environments where the train and test environments differ for evaluating generalization performance of RL agents. Kurach et al. [19] presented a Google Football simulator to study curriculum learning and multi-agent RL. Ray et al. [26] provided a Safety Gym for measuring progress towards RL agents that satisfy the safety constraints. D4RL [13] and RL Unplugged [14] have been proposed to evaluate and compare offline RL algorithms, and Yu et al. [37] proposed Meta-world to study meta-and multi-task RL.

Conclusion & Future works
In this work we present SCENIC4RL: A platform for generation of diverse scenarios for Reinforcement Learning programmatically, using the SCENIC scenario specification language. We present datasets of mini-game scenarios, behaviors depicting common soccer short-term strategies, probabilistic SCENIC policies and the demonstration dataset generate from them. Lastly, our evaluation demonstrates that our platform and the dataset can be applied to train, test, and benchmark RL algorithms.
Limitations and future directions. To the best of our knowledge SCENIC4RL is the first attempt to utilize formal scenario specification language to model environment dynamics in the domain of RL.
There are a number of interesting future directions that we plan to pursue.
• Interfacing other simulators: Currently, our platform only supports the Google Soccer Environment. In future we plan to include more simulators.
• As we are adding a layer over the base GRF simulator, we also add an overhead. In future we plan to decrease this overhead. Please refer to the Supplementary Materials for detailed experiment on SCENIC4RL performance.
• Multi-Agent Learning: While our platform does not restrict us to only single-agent RL settings, our current training experiments/benchmark do not include multi-agent training. In future, we would like to introduce scenarios and benchmarks for multi-agent setting.
• Multi-Task Learning: Currently we only train RL agents on a single scenario. However, many soccer skills are transferable. Hence, we want to experiment how we can solve multiple scenarios simultaneously in a multi-task setting • Use of Formal Analysis Techniques. Using SCENIC4RL it is possible to implement verification and systematic testing techniques (e.g., see [11,12]).
Negative Social Impact. Because our work concerns simulated soccer environment and has no relation to humans in the real world, we do not foresee any negative social impact of this work.

A.1 Motivation
The dataset using SCENIC4RL platform is created to (i) benchmark training performance, (ii) test generalizability, and (iii) generate semi-expert demonstration data for reinforcement learning (RL) algorithms in Google Research Football (GRF) Environment [19]. More importantly, we open-source our SCENIC4RL platform which enables researchers to intuitively model distributions over initial states and transition dynamics over RL and non-RL (i.e. environment) players to generate complex scenarios in GRF.
We share this platform for two reasons. First, despite advances in RL simulation environments, the lack of support to easily model and generate dynamic, interactive scenarios has become a bottleneck. Consequently, the benchmark scenarios have been restrictive (e.g. only varying the initial state distributions), and researcher had to rely on humans to generate quality demonstration data. In response, for the first time, we adopt an existing scenario specification language, SCENIC [9,10], to support researchers to intuitively model such complex environment. In our SCENIC4RL platform, researchers can use SCENIC's high-level syntax and semantics to quickly create complex environment with a fine-grained control over players' behaviors (i.e. transition dynamics). Secondly, because the space of all possible scenarios is infinite in the soccer environment and, therefore, it is impossible for one research group to provide a comprehensive set of scenarios. We share our platform to encourage the RL community to collectively contribute to constructing a comprehensive set of benchmark scenarios and rich demonstration data using SCENIC.
This dataset is created by Abdus Salam Azad, Edward Kim, Kimin Lee, and Qiancheng Wu, who are currently researchers at UC Berkeley. This work is not funded by any specific organization.

A.2 Composition
The dataset consists of 38 SCENIC programs and 5 sets of demonstration data that are generated from a subset of the programs. Here, scenarios are equivalent to SCENIC programs because SCENIC is a probabilistic programming language. Hence, a scenario encoded in SCENIC naturally becomes a program.
Of the 36 SCENIC programs, (i) 13 programs encode new offense and defense benchmark scenarios that the GRF environment does not provide, (ii) 18 programs model scenarios to test generalizability of RL agents trained from our 13 new benchmark scenarios and 5 GRF's benchmark scenarios with high (i.e. average scoring rate of 0.7 per game in each scenario) RL agent training performance, and (iii) 5 SCENIC programs specifying semi-expert RL stochastic policy to generate demonstration data. For these policies, we selected 5 offense scenarios from our new benchmark and GRF's benchmark scenarios where the proximal policy optimization (PPO) algorithm [29] performed relatively poorly, and scripted policy customized to each scenario. Hence, we provide 5 sets of demonstration data where each set consists of 10K samples collected from the successful (i.e. scores a goal) trajectories. Each sample is a < o, a > pair, i.e., specifying the action a the SCENIC policy took on the observation o. In our benchmark scenarios, we used the same reward function as GRF environment, which is to score (i.e. +1 if scores, -1 if scored, 0 otherwise).
Our benchmark scenarios and semi-expert policies are stochastic. This is because SCENIC is a probabilistic programming language, meaning users can specify distribution over states and transition dynamics. Hence, our provided scenarios and policies are expressive, capturing variety of initial state and transition dynamics that share certain spatial and temporal structures.
This dataset does not contain all possible benchmark scenarios or demonstration data in GRF environment since the space of possible scenarios are infinite. The scenarios and demonstration data we provide are only a subset which we modeled and generated using our domain knowledge of soccer.
Our dataset is self-contained, meaning we did not rely on any external sources. As to our SCENIC4RL platform, we relied on three external open-sourced codebase: SCENIC, GRF environment, and OpenAI gym environment API [3]. Our dataset does not contain any data that might be considered confidential, and it does not relate to people nor contain any data that might be offensive.

A.3 Collection Process
The SCENIC programs in our dataset is manually modelled by us using our domain knowledge of soccer. We modelled our benchmark scenarios such that each scenario is qualitatively distinct. Broadly, there are two different scenario categories: offense and defense, both of which are crucial for solving the environment. Within each category, we made distinction in three ways. First, the number of players are varied. Second, if scenarios share the same number of players, then their initial state distributions are varied to create different scenarios. Lastly, if the first two features of the scenarios are the same, then we specified distinct transition dynamics to players to qualitatively differentiate scenarios by either (a) modeling transition dynamics in SCENIC or (b) assigning rule-based AI bots provided by GRF environment to control players.
We provide 18 SCENIC programs to test generalization of (i) five GRF benchmark scenarios where PPO algorithm had high (i.e. average scoring rate of 0.7 per game in each scenario) training performance as reported in [19]) and all of our new benchmark scenarios from our dataset. The scripts tests how robust the trained agents are to perturbations in the initial state. We assigned distribution over the initial state while keeping the formation of players and their transition dynamics in each scenario intact.
As to SCENIC programs specifying stochastic semi-expert RL policies, we chose 5 offense scenarios in which PPO performed relatively poorly (i.e. average scoring rate less than 0.7 goal per mini-game as specified by the scenarios) from our new benchmark and GRF's benchmark scenarios. We scripted a stochastic semi-expert RL policy customized to each of the five offense scenarios using our domain knowledge of soccer.
Our benchmark and test scenarios and semi-expert stochastic RL policies are executed in GRF environment using our SCENIC4RL platform, which interfaces the SCENIC and the GRF simulator. During simulation, the SCENIC program statically samples an initial state at start, and the SCENIC program and an RL algorithm dynamically samples players' actions according to specified transition dynamics in the program. Specifically, the SCENIC program, the RL algorithm (via OpenAI gym environment API), and the GRF simulator communicate the following information at every simulation timestep; the simulator sends the ground truth state information of the world such as each player position, direction, role, ball possession, etc., while the scenic program and the RL algorithm takes this ground truth and samples actions for each RL and non-RL players. These sampled actions are sent to the simulator to be executed for 1 simulation timestep, and this communication continues until termination conditions are satisfied. We used the same termination conditions as GRF environment: (a) ball goes out of the field, (b) either team scores, and (c) change in the ball possession across teams. For the demonstration data generation process, the RL agent is substituted with our manually scripted policy (i.e. transition dynamics). In such case, only the SCENIC program and the GRF simulator interacts to generate the demonstration data. For more detail, refer to Section 3.3 of our main paper.
We sample specific instances of scenarios (i.e. concrete initial state and transition dynamics) in accordance with the type of distributions specified in the SCENIC program. Currently, SCENIC supports a uniformly random distributions on discrete, categorical, and continuous variables and a normal distribution. No human entity was involved in these benchmark scenario or demonstration data generation process, since the generation is automatically handled by our SCENIC4RL platform. The benchmark scenarios are generated for 5 million simulation timesteps, equivalent to the benchmark setting provided in the GRF paper [19]. As to demonstration data, since we only collected successful demonstration, the generation time varied widely according the accuracy of the policies, ranging from one to four hours to collect 10k demonstrations per set.

A.4 Data Pre-processing
For our demonstration dataset, we scripted customized policy for each of the five offense scenarios as discussed in Sec. A.2. We only collected successful trajectories where our policy ended in the RL team scoring against the opponent team. The non-successful demonstration data are discarded. Our code to generate this pre-processed demonstration data is open-sourced.
This non-successful demonstration can result from two causes. First, our scripted semi-expert policy may not always result in scoring. Second, the GRF environment has inherent stochasticity [19] that occasionally either disregards or do not fully execute sampled actions from our SCENIC programs.
This stochasticity is intentionally embedded in the simulator by the Google Research group to provide richer environment to RL agents.

A.5 Uses
We showcase the three use cases of our dataset and the SCENIC4RL platform: (i) benchmark training performance, (ii) test generalizability, and (iii) collect demonstration data to warm-start an RL agent. The experiment results for these use cases are included in the Evaluation Section of our main paper. Our dataset and the SCENIC4RL platform is new and, hence, it has not been used for any other task previously. Currently, our scenario and demonstration generation code implementation has been geared toward only single RL agent setting. We intend to extend our code base to accommodate multi-agent setting for our future work.

A.6 Distribution
Our dataset and the SCENIC4RL platform are publicly released on June, 2021 with BSD 3-Clause License. The code implementation for our platform, dataset generation, and experimentation as well as our dataset can be downloaded from https://github.com/scenic4RL/scenic4rl/tree/v1. Our code base requires users to install the GRF simulator [19], the SCENIC [10] repo, and the OpenAI's gym environment [3] API which are all open-sourced and free of charge.

A.7 Maintenance
The dataset and the platform will be hosted at UC Berkeley. All questions and comments can be sent to Abdus Salam Azad (salam_azad@berkeley.edu) and Edward Kim (ek65@berkeley.edu). All changes to the dataset and the platform will be announced through our mailing list. Those who would like to sign up should send an email to scenic4rl@lists.berkeley.edu. All the code and dataset will continue to be supported unless otherwise communicated through our mailing list. At this point, we do not have any expiration date to our code nor dataset.
We encourage users to contribute to enhancing the platform and to construct a comprehensive scenarios to benchmark, train, and test generalizable RL agents. They can do so by creating pull requests with their modifications and explanations. Upon review, they will be merged into our repo.

B Description of Proposed Scenarios and Policies
In this section we provide brief descriptions of all of the scenarios in our dataset.

B.1 New Benchmark Scenarios
In general, all our scenarios have the following three termination conditions: (i) ball goes off the field, (ii) change in ball possession across teams, (iii) one of the team scores. If any of these conditions are satisfied, then the scenario will terminate in simulation.

B.1.1 Offense Scenarios
In all of our four offense scenarios as shown in Fig. 7, we explicitly modelled the initial state distribution in using SCENIC and implicitly specified the behaviors to environment players by assigning the rule-based AI bots provided by Google Research Football (GRF) to control all non-RL players.
Hard Crossing: A very common scenario in real soccer games: 2 of our players along are guarded by 3 of the opponent players, in an interleaved manner, along the line of the penalty box. Another of our player at the edge of the field is attempting a cross.
11 vs GK: Our team, with a full lineup of eleven players in a traditional 4-4-2 formation, needs to score against the opponent goalkeeper.
Avoid, Pass, and Shoot: Two of our players, one starting on the middle of the right half and the other inside the penalty box, tries to score. One opponent defender starts between our players to intercept direct pass. Easy Crossing: An easy crossing scenario involving two of our players against opponent defender and goalkeeper in the penalty box.

B.1.2 Defense Scenarios
Like the offense scenarios, we assigned rule-based AI bots provided by GRF by default to control non-RL players in many of our defense scenarios as shown in Fig. 8, 9, 10. However, if the AI bots do not exhibit expected behavior for our modelled scenarios, we specified non-RL players' behaviors in SCENIC. For these scenarios with specified behaviors in SCENIC, their behaviors are highlighted with light blue arrows.
Goalkeeper vs Opponent: This scenario is designed to train an RL agent to be a defensive goalkeeper when it has to face an opponent one-on-one.
Defender vs Opponent with Hesitant Dribble: The opponent dribbles, stop, then dribbles again in a repeated manner.
3 vs 3 Side Build Up Play: Instead of crossing, the opponent player on the side builds up a play by short passing to its teammates.

B.2 Testing Generalization
For all the new benchmark scenarios in our dataset as well as for the selected five GRF's scenarios, we generalized those scenarios to test the generalizability of the trained RL agent. Our test scenarios are juxtaposed to corresponding scenarios in Fig. 7, 8,9,10. We modelled these test scenarios by either (i) adding distribution over the initial state or (ii) creating a symmetric opposite formation.

B.3 Semi-Expert Stochastic Policies
We selected five scenarios from GRF's and our benchmark scenarios. The selected GRF's scenarios are shown in Fig. 11. The following are the brief descriptions of policies encoded in SCENIC for each scenario.
Pass and Shoot with a Goal Keeper: We randomly choose one of the two policies for the RL agent.
In the first policy, the player dribbles to the penalty area and shoots once inside it. In the second policy, the player passes the ball to the teammate, who will then dribble towards the goal and shoot.
Easy Counterattack: The first player will pass the ball to the right midfielder. Then, the player with the ball will run into the penalty area, and if there is an opponent player on the way, the player will pass the ball to the nearest teammate. The player will shoot at a corner of the goal once inside the penalty area.
Run to Score with a Goal Keeper: The player with the ball will first sprint towards the goal and turn slightly to either left or right randomly to evade the opponent goalkeeper's interception. Once the player bypasses the goalkeeper, or is inside the penalty area, the player will shoot.
Avoid Pass and Shoot: The player decides to go towards 3 suitable regions of scoring: left edge/ middle/ right edge of the right goal post, by keeping as much distance possible to the opponent defender. At each time step it decides one of the three destination location. It first predicts its next position for all the three suitable destinations and pick the direction which keeps it farthest of the opponent. If the player comes near the defender it passes the ball to its teammate and if it can successfully go near the goal post, it attempts shooting.

vs Goal Keeper:
The player with the ball runs towards the right goalpost, if it reaches near the goal post it attempts to shoot. If the opponent goal keeper comes near (i.e. within seven meters) our player before it can reach near the right goal post, it stops running and shoots immediately. † The DEFENSE 3VS3 WITH CROSS scenario doesn't conclude a game upon a change in ball possession, unlike other scenarios

C Details on Experimental Setup and Training
We use the OpenAI Baselines' [6] implementation of PPO. The training was run for 5M timesteps with 16 parallel workers. All of our experiments are run on g4dn.4xlarge instances on Amazon AWS: a machine with a single NVIDIA T4 gpu, 16 virtual cores and 64GB RAM.
Network architecture & Hyperparameters For the PPO training, we first experimented with the network architecture and hyperparameters from [19] and was able to reproduce their result. [19] did an extensive search to select their hyperparameters and hence we decided to use the same for our experiments. The architecture we used from [19] is similar to the architecture introduced in [? ], with the exception of using four big blocks instead of three. The parameters for behavior cloning is shown in Table 2. For the GRF academy scenarios the behavior cloning algorithm is run for 16 epochs while for the offense scenarios it was run for 5 epochs.

D Interface details and Reproducibility
Our Platform follows the widely used OpenAI Gym API [3]. For sample usage, we refer readers to our github repository: https://github.com/scenic4RL/scenic4rl/tree/v1. The github repository contains detailed description of all our API with examples. It also contains necessary scripts to reproduce the results shown in paper.

E Performance
As we are adding an additional layer over the GRF simulator, we wanted to measure how much overhead we are adding over the base GRF simulator. We selected five GRF academy scenarios and ran a simulation of 20K timesteps with a random policy both in the GRF simulator and in SCENIC4RL. The simulation was ran sequentially, i.e., no parallelism was used. Across the scenarios, the GRF simulator took an average of 74.28 seconds for executing a simulation of 20K timesteps, while the SCENIC4RL interface took 222.07 seconds, showing a 2.99x drop in speed. Some of this overhead is inevitable however, we believe there are ways to speed up SCENIC4RL . For example, as we change the initial state every episode/simulation: we update the Python scenario file used by the GRF simulator for each episode/simulation. We plan to modify GRF interface to avoid such disk-access each simulation to speed up SCENIC4RL among other performance improvements. The scenarios we used for the experiments are namely: i) Empty Goal, ii) Empty Goal Close, iii) Pass and Shoot with Keeper, iv) Run, Pass, and Shoot with Keeper, and v) Run to Score with Keeper.


extfieldsofstudy:
Mathematics
Computer Science

text:
Using Ordinal Data to Assess Distance Learning

There is some disagreement on whether Likert scale data should be treated as ordinal or continuous. This paper treats Likert data as ordinal, uses non-parametric hypothesis testing, and clustering to validate those variables that have significant results from hypothesis testing.

adoption or even opposition, both methods would have worked together well during April and May of 2020. It also provided many students with experiences that, while not fully adopting formative assessment or flipped classrooms, contained some components of both. The purpose of the project was to create a survey that identified student a itudes about elements of flipped classrooms or formative assessments and determine if they had a statistically significant effect on outcomes. In actionable terms, it could isolate the tolerable or even enjoyable aspects of distance learning and bring those into the classroom, even if the classroom will never be flipped.

R ELATED W ORK
While there seems to be a consensus that flipped classrooms and formative assessment should be where education is headed, there is still research being done on both topics. As online learning is both experiencing growth in academic and business sectors, research to reduce the a rition rate and improve online education's capabilities are considered incredibly important. The concept of individual learning styles, while popular outside of those studying education, have been shown to have li le to no effect on educational outcomes. A question referring to learning styles was included on the survey, and it will be briefly discussed to demonstrate the methods of analysis for this project.

Flipped Classrooms
A flipped or inverted classroom model can be simplistically defined as when "activities traditionally conducted in the classroom become home activities" Formative assessment is a crucial component of flipped classrooms' success in improving academic performance (Zainuddin, 2016). The two would likely have to be implemented concurrently, as research suggests that current testing methods would prevent flipped classroom adoption from being fully successful (Rotellar, 2016). Rotellar also discussed the high cost in time to faculty while they develop the new classroom environment, and that many students initially resist the transition. However, even the most driven of students (in this experiment, medical students), after participating in a flipped classroom environment, find that it is a richer environment for learning than the traditional classroom (Martinelli, 2017). Even in an elementary mathematics course, the flipped classroom has demonstrated it can encourage be er outcomes (Lai, 2015).
The most recent papers emerging on the subject have been almost entirely qualitative in nature and use the required period of distance learning as their petri dish. One such paper found that online education in China was still able to follow a flipped classroom model and maintain student and teacher satisfaction (Yen, 2020). A paper from a Saudi Arabian university discusses how the training systems to implement flipped classrooms are not in place, but could provide a vital service during this time (Guraya, 2020).

Formative Assessment
One paper precisely defined formative assessment with its five primary goals: "1. Clarifying and sharing learning intentions and criteria for success; 2. Engineering effective classroom discussions and other learning task that elicit evidence of student understanding; 3. Providing feedback that moves learnings forward; 4. Activating students as instructional resources for one another; and 5. Activating students as the owners of their own learning." (Black, 2008) Because it would replace summative assessment, which relies primarily on exams to determine a student's grade, there may be pushback with the belief that it allows students to take an easier path. However, it actually requires students to set their own standards of behavior and engage in self-monitoring behavior (Sadler, 1989) in addition to the five criteria above. Returning to the issue nearly a decade later, Sadler emphasized the feedback loop that is crucial for formative assessment to work (Sadler, 1999). As online learning becomes more common, developing learning communities with peers is difficult; performative assessment, as a process, facilitates interactions between geographically divided students (Gikandi, 2011).
The most recent research uses tech platforms like Moodle to help students in their self-assessments (McCallum, 2020). The students in McCallum's experiment reported that they felt more confident in their abilities to monitor and steer their own progress. It seems to be a valuable skill to build, especially during a time when the adults instructing them are not quite sure what might happen.

Learning Styles
The primary area of research that involves learning styles seems to be an a empt to build a platform or AI tutor that can tailor instruction to the student. In a paper that integrated formative assessment and learning style analysis, formative assessment was found to be effective, as was learning style (Wang, 2006).
However, the only group that did succeed with their learning style were judged to be the reflective and observational learners; it seems quite possible that those students might have taken to self-assessment more quickly.
One paper discusses how self-reporting is no match for objective measurement: a study showed that self-reporting on what method is most comfortable had no correlation with methods that actually worked (Kirschner, 2016). Another study was unable to find any evidence of learning style efficacy, but was able to uncover study design errors in papers that found the theory effective (Pashler, 2009).

Self-Motivation in Online Learning
Research has been ongoing on the high a rition rates of online learning platforms. A primary factor is self-motivation, but motivation can be divided into extrinsic and intrinsic motivations. The paper suggested that external incentives to continue learning needed to be matched with a conscious effort to reduce student uncertainty and anxiety, which greatly affected intrinsic motivation (Chen, 2010). Another paper focused on the positive effects a competent instructor can have on an online learner's motivation, and the negative consequences that result from an imcompetent instructor (Selvi, 2010). Specific to mathematics instruction of children, research demonstrated the training in emotion regulation had positive educational effects (Cartwright, 2018).

Survey Design and Collection
The questionnaire consisted primarily of 5-point Likert scale items. The first question asked in what year of high school the survey-taker was enrolled. The next thirteen questions were all of the Likert scale types. The Google Form had the numbers 1 through 5 selectable, and "Strongly Disagree" was associated with 1, while "Strongly Agree" was placed by the 5. Every Likert scale question began with the following request: "Please state to what degree you agree or disagree with the following statement." Many Likert scale questions reversed the order of a positive response to add to the validity of the design. The criticisms of that construction are primarily from confusing the survey-taker or exhausting their patience, which was unlikely given the short length and simplicity of the questionnaire.
The final four questions asked students to estimate their time spent studying and their math grades before and during distance learning. As was noted above, students do misreport, but the error that comes with self-reporting was likely not a weakness in this particular survey. Some schools stated they would not lower grades after beginning distance learning. Good work could improve grades, but falling behind would not harm them. The change in study hours and grades provided continuous variables to use with the ordinal data.
Each question was designed to be associated with a principal feature of formative assessment or flipped classrooms. The questionnaire's goal was to be short and easy to understand. Thus, each question aimed to be easy for the student while providing insight into how these concepts worked in a distance learning environment. The ideal survey results would provide individual responses useful for research and form useful groupings in the project's analysis phase. The full survey is available in the appendices

Summary of Survey Results
Seventy-one student responses came from the Kansas City, Denver, and Sea le metro areas. When the first completed surveys arrived, all public high schools had begun distance learning. Due to anonymity, there is no tally of the geographic distribution of responses, but there is an even distribution of grade levels. For the same reason, no efforts to ensure that the sample was representative of the United States population were feasible. Likely no rural students responded to the questionnaire. As the link's distribution came from one source in each metro area, it is possible respondents were more similar to one another than a random sample would have been.

Method of Analysis
The primary analysis focused on the distribution of the answers to each question and the statistical analysis and post-hoc tests performed on each question's responses. The results for each question will not be included for brevity's sake; only interesting or demonstrative analyses will be reported. To demonstrate the process, Question 9 on the survey is as follows: "Please state to what degree you agree or disagree with the following statement: online learning is not my preferred learning style:" and asks for an ordinal response. By using the calculated difference between the pre-distance learning math grade and grade during distance learning as a dependent variable, I could perform the Kruskal-Wallis test. The results were: Kruskal-Wallis chi-squared = 6.755, df = 4, p-value = 0.1494 As learning styles were criticized for not having an effect, it holds true in this experiment as well.
All of the other questions performed be er, but some had other issues. Question 3 asked if the change in style of the math class made the student feel less confident in their ability to succeed in math.
Kruskal-Wallis chi-squared = 9.0497, df = 4, p-value = 0.05987 Due to uncertainty with sample size and a potential for an overly small or large group to throw off the test statistics, >90% was set as a threshold for rejecting the null hypothesis of the Kruskal-Wallis test. However, after running the Benjamini-Hochberg pairwise test on each group, the adjusted p-values went from significant to insignificant.
Finally, a "waffle plot" was made for each Likert item to see if the distribution might be grounds to dismiss a significant finding. Questions 10 and 14 were discarded because more than half the plot consisted of one Likert response. All other waffle plots will be included in the appendices. Here is Question 14:

R ESULTS
The following six results were found to be statistically significant and relevant to the project's theme of looking for characteristics of formative assessment or flipped classrooms. Though "Strongly Disagree" is weakly represented, the Kruskal-Wallis test is robust enough to handle that when the other four are more evenly distributed.

Question 4 Analysis: Dependent Variable GPA Change
The Kruskal-Wallis test chi-squared value was 20.844 with a p-value of 0.0003.

The Dunn Kruskal-Wallis pairwise test did not show any strange information.
The test for effect size returned an epsilon-squared statistic of .255, considered to be of large magnitude. The test for effect size returned an epsilon-squared statistic of .262, considered to be of large magnitude.
The test for effect size returned an epsilon-squared statistic of .424, considered to be of large magnitude.
The test for effect size returned an epsilon-squared statistic of .213, considered to be of large magnitude.
The test for effect size returned an epsilon-squared statistic of .354, considered to be of large magnitude.

Question 13 Analysis with GPA as Dependent Variable
Kruskal-Wallis chi-squared = 30.473, df = 4, p-value = 3.92e-06 The Dunn Kruskal-Wallis pairwise test did not show any strange information.
The test for effect size returned an epsilon-squared statistic of .354, considered to be of large magnitude.

Summary of Findings
Questions 4, 5, 6, 7, 12, and 13 all were found to have statistical significance, or in Kruskal-Wallis terms, were found to have at least one group with stochastic dominance over the others. Listing out these six Likert items: Q4: my math teacher has done an excellent job developing an online curriculum for us .

Q5: I no longer receive the math support I need when I began distance learning.
Q6: to keep up in class, I feel the need to use other online learning tools to study.
Q7: I feel that the lack of peer interaction has caused my math performance to suffer.
Q12: online exams cause me more anxiety than in-person exams did. Finally, Question 13's response would be very different if they were primarily graded on a summative assessment or formative assessment. Close to half "Agreed" or "Strongly Agreed".
To explore just a bit further, I performed a Polychoric Correlation test, which is suitable for ordinal variables, to judge the relationships between these variables. Question 13 is highly correlated with a correlation coefficient of .9 to Question 7, which implies that as responses move towards a classroom that is heavily based on formative assessment, so does the need to use online tools. This correlation only makes sense if there are some students engaging in the formative assessment model. The highest correlation on the table is between 5 and 7, which implies that as more students agree that their grades are suffering from a lack of peer interaction, more will also feel as though they lack support. This could be indicative of a flipped classroom model, where feedback from classmates is extremely valuable. The visualization of this Correlation Table and also a Factor Analysis and a Cluster diagram will be in the appendices.

Limitations
There were several limitations with this study. The sample size was on the border of being too small, and may have affected some of the statistical tests.
Samples were pulled from three locations with a high chance that the survey was answered by someone similar to another person who responded, simply because people passed the questionnaire along to friends, who tend to be similar. The Kruskal-Wallis test is difficult to interpret beyond acknowledgment of difference, but the correlations did provide some suggestions for how they relate to each other. The visualizations in the appendix makes the existence of the relationship easier to acknowledge, but there is more that could be done. Finally, several fascinating questions about how the different grades differed in their responses were impossible due to the sample size.

Future Work
The question of "partially applied" flipped classroom and formative assessment methodologies is an interesting one, and the choices made to investigate it were sound. With a chance to send out a slightly modified survey, a larger sample size, some indicators that the sample was representative of the United States population, and some machine learning clustering algorithms, the analysis would likely return a rewarding answer. Please state to what degree you agree or disagree with the following statement: online exams cause me more anxiety than in-person exams did. *

C ORRELATION , F ACTOR A NALYSIS , AND C LUSTER P LOTS
Please state to what degree you agree or disagree with the following statement: my math teacher has made it clear that while exams are important, our completed coursework and involvement will do more for my final grade. * Please state to what degree you agree or disagree with the following statement: it is more difficult for me to study before exams or start homework early now than it was during inclass instruction. * Before you began distance learning, estimate how many hours per week would you spend on math? * Since beginning distance learning, estimate how many hours per week do you spend on math? *

Your answer
In general, before distance learning, what would estimate your average math grade to be? *