We propose PIGLeT : a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language. We factorize PIGLeT into a physical dynamics model, and a separate language model. Our dynamics model learns not just what objects are but also what they do : glass cups break when thrown, plastic ones don't. We then use it as the interface to our language model, giving us a unified model of linguistic form and grounded meaning. PIGLeT can read a sentence, simulate neurally what might happen next, and then communicate that result through a literal symbolic representation, or natural language. Experimental results show that our model effectively learns world dynamics, along with how to communicate them. It is able to correctly forecast what happens next given an English sentence over 80% of the time, outperforming a 100x larger, text-to-text approach by over 10%. Likewise, its natural language summaries of physical interactions are also judged by humans as more accurate than LM alternatives. We present comprehensive analysis showing room for future work. As humans, our use of language is linked to the physical world. To process a sentence like the robot turns on the stove, with a pan on it (Figure 1) we might imagine a physical Pan object. This meaning representation in our heads can be seen as a part of our commonsense world knowledge, about what a Pan is and does. We might reasonably predict that the Pan will become Hot  and if there's an Egg on it, it would become cooked . As humans, we learn such a commonsense world model through interaction. Young children learn to reason physically about basic objects by manipulating them: observing the properties they have, Language Model PIGLeT t t+1 The robot turns on the stove, with a pan on it. and how they change if an action is applied on them (Smith and Gasser, 2005). This process is hypothesized to be crucial to how children learn language: the names of these elementary objects become their first real words upon which other language is scaffolded (Yu and Smith, 2012). In contrast, the dominant paradigm today is to train large language or vision models on static data , such as language and photos from the web. Yet such a setting is fundamentally limiting, as suggested empirically by psychologists' failed attempts to get kittens to learn passively (Held and Hein, 1963). More recently, though large Transformers have made initial progress on benchmarks, they also have frequently revealed biases in those same datasets, suggesting they might not be solving underlying tasks (Zellers et al., 2019b). This has been argued philosophically by a flurry of reThe robot throws the vase onto the coffee table. cent work arguing that no amount of language form could ever specify language meaning (McClelland et al., 2019; Bender and Koller, 2020; Bisk et al., 2020); connecting back to the Symbol Grounding Problem of Harnad (1990). In this paper, we investigate an alternate strategy for learning physical commonsense through interaction, and then transferring that into language. We introduce a model named PIGLeT , short for P hysical I nteraction as G rounding for L anguag e T ransformers. We factorize an embodied agent into an explicit model of world dynamics, and a model of language form. We learn the dynamics model through interaction . Given an action heatUp applied to the Pan in Figure 1, the model learns that the Egg on the pan becomes Hot and Cooked , and that other attributes do not change. We integrate our dynamics model with a pretrained language model, giving us a joint model of linguistic form and meaning . The combined PIGLeT can then reason about the physical dynamics implied by English sentences describing actions, predicting literally what might happen next. It can then communicate that result either symbolically or through natural language, generating a sentence like The egg becomes hot and cooked. Our separation between physical dynamics and language allows the model to learn about physical commonsense from the physical world itself, while also avoiding recurring problems of artifacts and biases that arise when we try to model physical world understanding solely through language. We study this through a new environment and evaluation setup called PIGPeN , short for P hysical I nteraction G rounding P air e d with N atural Language. In PIGPeN , a model is given unlimited access to an environment for pretraining, but only 500 examples with paired English annotations. Models in our setup must additionally generalize to novel unseen' objects for which we intentionally do not provide paired language-environment supervision. We build this on top of the THOR environment (Kolve et al., 2017), a physics engine that enables agents to perform contextual interactions (Fig 2) on everyday objects. Experiments confirm that PIGLeT performs well at grounding language with meaning. Given a sentence describing an action, our model predicts the resulting object states correctly over 80% of the time, outperforming even a 100x larger model (T5-11B) by over 10%. Likewise, its generated natural language is rated by humans as being more correct than equivalently-sized language models. Last, it can generalize in a zero-shot' way to objects that it has never read about before in language. In summary, we make three key contributions. First , we introduce PIGLeT , a model decoupling physical and linguistic reasoning. Second , we introduce PIGPeN , to learn and evaluate the transfer of physical knowledge to the world of language. Third , we perform experiments and analysis suggesting promising avenues for future work. We introduce PIGPeN as a setting for learning and evaluating physically grounded language understanding. An overview is shown in Figure 2. The idea is that an agent gets access to an interactive 3D environment, where it can learn about the world through interaction  for example, that objects such as a Vase can become Broken if thrown. The goal for a model is to learn natural language meaning grounded in these interactions. Task definition. Through interaction, an agent observes the interplay between objects o 2 O (rep-resented by their attributes) and actions a 2 A through the following transition: { o 1 , . . . , o N } | {z } ~ o , state pre-action  a ! To encourage learning from interaction, and not just language, an agent is given a small number of natural language annotations of transitions. We denote these sentences as s ~ o , describing the state pre-action, s a the action, and s ~ o 0 the state post-action respectively. During evaluation, an agent will sometimes encounter new objects o that were not part of the paired training data. We evaluate the model's transfer in two ways: a . PIGPeN -NLU. A model is given object states ~ o , and an English sentence s a describing an action. It must predict the grounded object states ~ o 0 that result after the action is taken. b . PIGPeN -NLG. A model is given object states ~ o and a literal action a . It must generate a sentence s ~ o 0 describing the state post-action. We next describe our environment, feature representation, and language annotation process. We use AI2-THOR as an environment for this task (Kolve et al., 2017). In THOR, a robotic agent can navigate around and perform rich contextual interactions with objects in a house. For instance, it can grab an Apple , slice it, put it in a Fridge , drop it, and so on. The state of the Apple , such as whether it is sliced or cold, changes accordingly; this is not possible in many other environments. In this work, we use the underlying THOR simulator as a proxy for grounded meaning. Within THOR, it can be seen as a complete' meaning representation (Artzi et al., 2013), as it fully specifies the kind of grounding a model can expect in its perception within THOR. Objects. The underlying THOR representation of each object o is in terms of 42 attributes; we provide a list in Appendix B. We treat these attributes as words specific to an attribute-level dictionary; for example, the temperature Hot is one of three possible values for an object's temperature; the others being Cold and RoomTemp . Actions. An action a in THOR is a function that takes up to two objects as arguments. Actions are highly contextual, affecting not only the arguments but potentially other objects in the scene (Figure 2). We also treat action names as words in a dictionary. Filtering out background objects. Most actions change the state of only a few objects, yet there can be many objects in a scene. We keep annotation and computation tractable by having models predict (and humans annotate) possible changes of at most two key objects in the scene. As knowing when an object doesn't change is also important, we include non-changing objects if fewer than two change. Exploration. Any way of exploring the environment is valid for our task, however, we found that exploring intentionally was needed to yield good coverage of interesting states. Similar to prior work for instruction following (Shridhar et al., 2020), we designed an oracle to collect diverse and interesting trajectories { ~ o , a , ~ o 0 } . Our oracle randomly selects one of ten high level tasks, see Appendix B for the list. These in turn require randomly choosing objects in the scene; e.g. a Vase and a Laptop in Figure 2. We randomize the manner in which the oracle performs the task to discover diverse situations. In total, we sampled 20k trajectories. From these we extracted 280k transitions (Eqn 1's) where at least one object changes state, for training. We select 2k action state-changes from trajectories held out from the training set. We select them while also balancing the distribution of action types to ensure broad coverage in the final dataset. We are also interested in a model's ability to generalize to new object categories  beyond what it has read about, or observed in a training set. We thus select 30 objects to be unseen, and exclude these from paired environment-language training data. We sample 500 state transitions, containing only seen objects to be the training set; we use 500 for validation and 1000 for testing. Workers on Mechanical Turk were shown an environment in THOR before and after a given action a . Each view contains the THOR attributes of the two key objects. Workers then wrote three English sentences, corresponding to s ~ o , s a , and s ~ o 0 respectively. Workers were instructed to write at a particular level of detail: enough so that a reader could infer what happens next from s ~ o and s a , yet without mentioning redundant attributes.We provide more details in Appendix C. 3 Modeling PIGLeT In this section, we describe our PIGLeT model. First , we learn a neural physical dynamics model The robot is holding a glass vase. from interactions, and second , integrate with a pretrained model of language form. We take a neural, auto-encoder style approach to model world dynamics. An object o gets encoded as a vector h o 2 R d o . The model likewise encodes an action a as a vector h a 2 R d a , using it to manipulate the hidden states of all objects. The model can then decode any object hidden representation back into a symbolic form. We use a Transformer (Vaswani et al., 2017) to encode objects into vectors o 2 R d o , and then another to decode from this representation. Encoder. Objects o are provided to the encoder as a set of attributes, with categories c 1 ,..., c n . Each attribute c has its own vocabulary and embedding E c . For each object o , we first embed all the attributes separately and feed the result into a Transformer encoder T enc . This gives us (with position embeddings omitted for clarity): h o = T enc  E 1 ( o 1 ) , . . . , E c n ( o c n )  (2) Decoder. We can then convert back into the original symbolic representation through a left-to-right Transformer decoder, which predicts attributes one-by-one from c 1 to c n . This captures the inherent correlation between attributes, while making no in-dependence assumptions, we discuss our ordering in Appendix A.2. The probability of predicting the next attribute o c i +1 is then given by: p ( o c i +1 | h o , o : c i )= T dec  h o , E 1 ( o 1 ) ,..., E c i ( o c i )  (3) 3.1.2 Modeling actions as functions We treat actions a as functions that transform the state of all objects in the scene. Actions in our environment take at most two arguments, so we embed the action a and the names of its arguments, concatenate them, and pass the result through a multilayer perceptron; yielding a vector representation h a . Applying Actions. We use the encoded action h a to transform all objects in the scene, obtaining updated representations  h o 0 for each one. We take a global approach, jointly transforming all objects. This takes into account that interactions are contextual: turning on a Faucet might fill up a Cup if and only if there is one beneath it. Letting the observed objects in the interaction be o 1 and o 2 , with encodings h o 1 and h o 2 respectively, we model the transformation via the following multilayer perceptron: [ h o 0 1 ,  h o 0 2 ] = MLP apply  h a , h o 1 , h o 2  . (4) The result can be decoded into symbolic form using the object decoder (Equation 3). We train our dynamics model on ( ~ o , a , ~ o 0 ) transitions. The model primarily learns by running ~ o , a through the model, predicting the updated output state  h o 0 , and minimizing the cross-entropy of generating attributes of the real changed object ~ o 0 . We also regularize the model by encoding objects ~ o , ~ o 0 and having the model learn to reconstruct them. We weight all these cross-entropy losses equally. We discuss our architecture in Appendix A.1; it uses 3-layer Transformers, totalling 17M parameters. After pretraining our physical dynamics model, we integrate it with a Transformer Language Model (LM). In our framework, the role of the LM will be to both encode natural language sentences of actions into a hidden state approximating h a , as well as summarizing the result of an interaction ( ~ o , a , ~ o 0 ) in natural language. Choice of LM. Our framework is compatible with any language model. However, to explore the impact of pretraining data on grounding later in this paper, we pretrain our own with an identical architecture to the smallest GPT2 (Radford et al. (2019); 117M). To handle both classification and generation well, we mask only part of the attention weights out, allowing the model to encode a prefix bidirectionally; it generates subsequent tokens left-to-right (Dong et al., 2019). We pretrain the model on Wikipedia and books; details in Appendix D. We next discuss architectural details of performing the language transfer, along with optimization. English actions to vector form. Given a natural language description s a of an action a , like The robot throws the vase, for PIGPeN -NLU, our model will learn to parse this sentence into a neural representation h a , so the dynamics model can simulate the result. We do this by encoding s a through our language model, TLM , with a learned linear transformation over the resulting (bidirectional) encoding. The resulting vector h s a can then be used by Equation 4. Summarizing the result of an action. For PIGPeN -NLG, our model simulates the result of an action a neurally, resulting in a predicted hidden state  h o for each object in the scene o . To write an English summary describing what changed, we first learn a lightweight fused representation of the transition, aggregating the initial and final states, along with the action, through a multilayer perceptron. For each object o i we have: h  o i = MLP  ([ h o i ,  h o 0 i , h a ]) . We then use the sequence [ h  o 1 , h  o 2 ] as bidirectional context for our our LM to decode from. Additionally, since our test set includes novel objects not seen in training, we provide the names of the objects as additional context for the LM generator (e.g. Vase, Laptop'); this allows a LM to copy those names over rather than hallucinate wrong ones. Importantly we only provide the surface-form names, not underlying information about these objects or their usage as with few-shot scenarios in the recent GPT-3 experiments (Brown et al., 2020)  necessitating that PIGLeT learns what these names mean through interaction. Modeling text generation allows us to incorporate a new loss function, that of minimizing the log-likelihood of generating each s ~ o 0 given previous words and the result of Equation 5: p ( s post i +1 | s ~ o 0 , 1: i ) = TLM ( h  o 1 , h  o 2 , s ~ o 0 , 1: i ) . using h o i as the corresponding hidden states. For PIGPeN -NLU, where no generation is needed, optimizing Equation 5 is not strictly necessary. However, as we will show later, it helps provide additional signal to the model, improving overall accuracy by several percentage points. We test our model's ability to encode language into a grounded form ( PIGPeN -NLU), and decode that grounded form into language ( PIGPeN -NLG). We first evaluate models by their performance on PIGPeN -NLU: given objects ~ o , and a sentence s a describing an action, a model must predict the resulting state of objects ~ o 0 . We primarily evaluate models by accuracy; scoring how many objects for which they got all attributes correct. We compare with the following strong baselines: a . No Change: this baseline copies the initial state of all objects ~ o as the final state ~ o 0 . b . GPT3-175B (Brown et al., 2020), a very large language model for few-shot' learning using a prompt. For GPT3, and other text-to-text models, we encode and decode the symbolic object states in a JSON-style dictionary format, discussed in Appendix A.4. c . T5 (Raffel et al., 2019). With this model, we use the same text-to-text' format, however here we train it on the paired data from PIGPeN . We consider varying sizes of T5, from T5-Small  the closest in size to PIGLeT , up until T5-11B, roughly 100x the size. d . (Alberti et al., 2019)-style. This paper originally proposed a model for VCR (Zellers et al., Model Accuracy (%) Val Test Overall Seen Unseen No Change 27.4 25.5 29.9 24.0 t e x t t o t e x t GPT3-175B (Brown et al., 2020) 23.8 22.4 22.4 21.4 T5-11B (Raffel et al., 2019) 68.5 64.2 79.5 59.1 T5-3B 66.6 63.3 77.1 58.7 T5-Large 56.5 54.1 69.2 49.1 T5-Base 56.0 53.9 69.2 48.8 T5-Small 39.9 36.2 57.0 38.0 BERT s t y l e Alberti et al.2019, Pretrained Dynamics 61.3 53.9 71.4 48.1 Alberti et al.2019 9.7 6.8 16.2 3.7 G&D2019, Pretrained Dynamics 43.8 35.3 60.9 26.9 G&D2019 15.1 11.3 23.1 7.3 PIGLeT 81.8 81.1 83.8 80.2 Attribute-level accuracy (Test-Overall,%) size distance mass Temperature isBroken 8-way 8-way 8-way 3-way boolean 83.2 84.1 96.3 86.0 94.8 73.7 77.0 89.5 84.2 94.7 83.9 88.9 94.3 95.4 98.1 81.6 90.0 94.0 95.6 98.4 81.8 84.6 94.3 96.3 95.8 81.1 87.5 93.6 96.1 96.5 82.2 84.9 93.8 89.6 93.5 87.7 87.6 97.5 93.4 97.5 53.4 43.6 84.0 88.1 95.1 83.0 86.9 94.0 93.7 97.4 68.6 47.3 82.2 88.3 95.8 92.3 91.9 99.2 99.8 99.0 Table 1: Overall results . 2019a), where grounded visual information is fed into a BERT model as tokens; the transformer performs the grounded reasoning. We adapt it for our task by using our base LM and feeding in object representations from our pretrained object encoder, also as tokens. Our object decoder predicts the object, given the LM's pooled hidden state. This is pretrained dynamics, we also consider a version without a randomly initialized dynamics model. e . (Gupta and Durrett, 2019)-style. Thiso paper proposes using Transformers to model physical state, for tasks like entity tracking in recipes. Here, the authors propose decoding a physical state attribute (like isCooked ) by feeding the model a label-specific [CLS] token, and then mapping the result through a hidden layer. We do this and use a similar object encoder as our (Alberti et al., 2019)-style baseline. We discuss hyperparameters in Appendix A.3. Results. From the results (Table 1), we can draw several patterns. Our model, PIGLeT performs best at getting all attributes correct; doing so over 80% on both validation and test sets, even for novel objects not seen during training. The next closest model is T5-11B, which scores 68% on validation. Though when evaluated on objects seen' during training it gets 77%, that number drops by over 18% for unseen objects. On the other hand, PIGLeT has a modest gap of 3%. This suggests that our approach is particularly effective at connecting unpaired language and world representations. At Model Accuracy (val;%) PIGLeT , No Pretraining 10.4 PIGLeT , Non-global MLP apply 72.0 PIGLeT , Global MLP apply 78.5 PIGLeT , Global MLP apply , Gen. loss (6) 81.8 PIGLeT , Symbols Only (Upper Bound) 89.3 Table 2: Ablation study on PIGPeN -NLU's validation set. the other extreme, GPT3 does poorly in its few-shot' setting, suggesting that size is no replacement for grounded supervision. PIGLeT also outperforms BERT style' approaches that control for the same language model architecture, but perform the physical reasoning inside the language transformer rather than as a separate model. Performance drops when the physical decoder must be learned from few paired examples (as in Gupta and Durrett (2019)); it drops even further when neither model is given access to our pretrained dynamics model, with both baselines then underperforming No Change.' This suggests that our approach of having a physical reasoning model outside of an LM is a good inductive bias. In Table 2 we present an ablation study of PIGLeT 's components. Of note, by using a global representation of objects in the world (Equation 4), we get over 6% improvement over a local representation where objects are manipulated independently. We get another 3% boost by adding a generation loss, suggesting that learning to generate summaries helps the model better connect the world to language. Last, we benchmark how much headroom there is on PIGPeN -NLU by evaluating model performance on a symbols only' version of the task, where the symbolic action a is given explicitly to our dynamics model. This upper bound is roughly 7% higher than PIGLeT , suggesting space for future work. Next, we turn to PIGPeN -NLG: given objects ~ o and the literal next action a , a model must generate a sentence s ~ o 0 describing what will change in the scene. We compare with the following baselines: a . T5. We use a T5 model that is given a JSON-style dictionary representation of both ~ o and a , it is finetuned to generate summaries s ~ o 0 . b . LM Baseline. We feed our LM hidden states h o from our pretrained encoder, along with its representation of a . The key difference between it and PIGLeT is that we do not allow it to simulate neurally what might happen next  MLP apply is never used here. Size matters. Arguably the most important factor controlling the fluency of a language generator is its size (Kaplan et al., 2020). Since our LM could also be scaled up to arbitrary size, we control for size in our experiments and only consider models the size of GPT2-base (117M) or smaller; we thus compare against T5-small as T5-Base has 220M parameters. We discuss optimization and sampling hyperparameters in Appendix A.3. Evaluation metrics. We evaluate models over the validation and test sets. We consider three main evaluation metrics: BLEU (Papineni et al., 2002) with two references, the recently proposed BERTScore (Zhang et al., 2020), and conduct a human evaluation. Humans rate both the fluency of post-action text, as well as its faithfulness to true action result, on a scale from  1 to 1 . Results. We show our results in Table 3. Of note, PIGLeT is competitive with T5 and significantly outperforms the pure LM baseline, which uses a pretrained encoder for object states, yet has the physical simulation piece MLP apply removed. This suggests that simulating world dynamics not only allows the model to predict what might happen Model BLEU BERTScore Human (test; [ 9 1 , 1] ) Val Test Val Test Fluency Faithfulness T5 46.6 43.4 82.2 81.0 0.82 0.15 LM Baseline 44.6 39.7 81.6 78.8 0.91 -0.13 PIGLeT 49.0 43.9 83.6 81.3 0.92 0.22 Human 44.5 45.6 82.6 83.3 0.94 0.71 Table 3: Text generation results on PIGPeN -NLG, showing models of roughly equivalent size (up to 117M parameters). next, it leads to more faithful generation as well. We show two qualitative examples in Figure 4, covering both PIGPeN -NLU as well as PIGPeN -NLG. In the first row, the robot empties a held Mug that is filled with water. PIGLeT gets the state, and generates a faithful sentence summarizing that the mug becomes empty. T5 struggles somewhat, emptying the water from both the Mug and the (irrelevant) Sink . It also generates text saying that the Sink becomes empty, instead of the Mug. In the second row, PIGLeT correctly predicts the next object states, but its generated text is incomplete  it should also write that the mug becomes filled wtih Coffee. T5 makes the same mistake in generation, and it also underpredicts the state changes, omitting all changes to the Mug . We suspect that T5 struggles here in part because Mug is an unseen object. T5 only experiences it through language-only pretraining, but this might not be enough for a fully grounded representation. The language models that perform best today are trained on massive datasets of text. However, this has unintended consequences (Bender et al., 2021) and it is unlike how children learn language, with children learning novel words from experience (Carey and Bartlett, 1978). The large scale of our pretraining datasets might allow models to learn to perform physical-commonsense like tasks for wrong reasons, overfitting to surface patterns rather than learning meaningful grounding. We investigate the extent of this by training a zero-shot' version of our backbone LM on Wikipedia and books  the only difference is that The sink is now empty. In this setting, not only must PIGLeT learn to ground words like mug,' it must do so without having seen the word mug' during pretraining. This is signifi-cant because we count over 20k instances of Mug' words (including morphology) in our dataset. We show results in Figure 5. A version of PIGLeT with the zero-shot LM does surprisingly well  achieving 80% accuracy at predicting the state changes for Mug  despite never having been pretrained on one before. This even outperforms T5 at the overall task. Nevertheless, PIGLeT outperforms it by roughly 7% at unseen objects, with notable gains of over 10% on highly dynamic objects like Toaster s and Sink s. work, we study language grounding and commonsense commonsense reasoning at the representation and concept level. The aim is to train models that learn to acquire concepts more like humans, rather than performing well on a downstream task that (for humans) requires commonsense reasoning. Thus, this work is somewhat different versus other 3D embodied tasks like QA (Gordon et al., 2018; Das et al., 2018), along with past work for measuring such grounded commonsense reasoning, like SWAG, HellaSWAG, and VCR (Zellers et al., 2018, 2019b,a). The knowledge covered is different, as it is self-contained within THOR. While VCR, for instance, includes lots of visual situations about what people are doing, this paper focuses on learning the physical properties of objects. Zero-shot generalization . There has been a lot of past work involved with learning zero-shot': often learning about the grounded world in language, and transferring that knowledge to vision. Techniques for this include looking at word embeddings (Frome et al., 2013) and dictionary defini-tions (Zellers and Choi, 2017). In this work, we propose the inverse. This approach was used to learn better word embeddings (Gupta et al., 2019) or semantic tuples (Yatskar et al., 2016), but we consider learning a component to be plugged into a deep Transformer language model. Past work evaluating these types of zero-shot generalization have also looked into how well models can compose concepts in language together (Lake and Baroni, 2018; Ruis et al., 2020). Our work considers elements of compositional-ity through grounded transfer. For example, in PIGPeN -NLG, models must generate sentences about the equivalent of dropping a dax', despite never having seen one before. However, our work is also contextual, in that the outcome of dropping a dax' might depend on external attributes (like how high we're dropping it from). Structured Models for Attributes and Objects . The idea of modeling actions as functions that transform objects has been explored in the computer vision space (Wang et al., 2016). Past work has also built formal structured models for connecting vision and language (Matuszek et al., 2012; Krishnamurthy and Kollar, 2013), we take a neural approach and connect today's best models of language form to similarly neural models of a simulated environment. In this paper, we presented an approach PIGLeT for jointly modeling language form and meaning. We presented a testbed PIGPeN for evaluating our model, which performs well at grounding language to the (simulated) world. We thank the reviewers for their helpful feedback, and the Mechanical Turk workers for doing a great job in annotating our data. Thanks also to Zak Stone and the Google Cloud TPU team for help with the computing infrastructure. This work was supported by the DARPA CwC program through ARO (W911NF-15-1-0543), the DARPA MCS program through NIWC Pacific (N66001-19-2-4031), and the Allen Institute for AI.
objective abstain abstain method abstain result abstain abstain result abstain abstain abstain result method abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain other method abstain abstain result abstain abstain objective objective method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method method abstain method abstain abstain abstain method abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method other other method other method other other other objective method other method other abstain other other method method method other other other