Semi-Supervised Detection of Extreme Weather Events in Large Climate Datasets
The detection and identification of extreme weather events in large scale climate simulations is an important problem for risk management, informing governmental policy decisions and advancing our basic understanding of the climate system. Recent work has shown that fully supervised convolutional neural networks (CNNs) can yield acceptable accuracy for classifying well-known types of extreme weather events when large amounts of labeled data are available. However, there are many different types of spatially localized climate patterns of interest (including hurricanes, extra-tropical cyclones, weather fronts, blocking events, etc.) found in simulation data for which labeled data is not available at large scale for all simulations of interest. We present a multichannel spatiotemporal encoder-decoder CNN architecture for semi-supervised bounding box prediction and exploratory data analysis. This architecture is designed to fully model multi-channel simulation data, temporal dynamics and unlabelled data within a reconstruction and prediction framework so as to improve the detection of a wide range of extreme weather events. Our architecture can be viewed as a 3D convolutional autoencoder with an additional modified one-pass bounding box regression loss. We demonstrate that our approach is able to leverage temporal information and unlabelled data to improve localization of extreme weather events. Further, we explore the representations learned by our model in order to better understand this important data, and facilitate further work in understanding and mitigating the effects of climate change.
This work presents a novel 3D CNN architecture for climate event detection that combines an unsupervised auto-encoder reconstruction loss with YOLO like bounding box prediction. The approach is trained and evaluated on a large-scale, simulated climate dataset labeled by a costly heuristic approach called TECA. For the most part, the paper is nicely written (minor comments below) and addresses an important and well motivated problem. The authors provide sufficient model details to allow reproduction (although public code would be preferred). I find the experiments a bit unconvincing (see below) but appreciate the attention to model capacity (via number of parameter) when comparing the 2D and 3D model variants. I am concerned that the evaluation may be insufficient to assess the effectiveness of this method. An IoU threshold of 0.1 allows for many rather poor detections to count as true positives. If the goal of this work is to count the number of such instances this is defensible, but for localization this seems overly loose.Furthermore, the 3D CNN architecture (which is one of the the core novelties of this work compared to past efforts) does not seem capable of producing variable sized boxes (as noted in the last paragraph of page 7), which I imagine results in poorer performance at higher IoU thresholds (as many of the weather events appear small). The experiments also feel inconclusive about the effect of temporal modeling and semi-supervision. The temporal component does not seem to matter in the supervised settings (2D 51.45 mAP - 3D 51.00 mAP) but improves somewhat in the semi-supervised case (2D 51.11 mAP - 3D 52.92 mAP). Whereas the additional unlabeled data seems to hurt in the 2D case but improve results for the 3D model. Could the authors provide confidence intervals for these numbers? I would like to see further discussion of these trends especially with respect to the effect of the loss weights (alpha, beta, and gamma). I also note that it is not clear if both the 2D and 3D models were trained for equivalent time periods (seems like no from last paragraph of page 7). Could a plot of training and validation accuracy for each model be presented for comparison? Finally, is there any baseline approach the authors could report or compare too? Without one, it is difficult to evaluate the performance of the approach with respect to the difficulty of the problem. Preliminary Rating: I think this is an interesting paper that is well motivated but feel the experiments as presented do not seem adequate to support any conclusive trends. I would like to see the mAP trends across a wider range of IoU values and further discussion of training procedure, loss weight settings, and reasons for lack of bounding box variability in the 3D model (as stated above). Clarification: In the paper you say *While climate models are run on a 3D grid, with the vertical dimension corresponding to 30 levels; we only consider surface quantities (i.e. 2D data) in this study.* Could you elaborate on what the surface quantities correspond to? Is it the highest cloud level? Minor notes: Please provide years for Prabhat et al. references rather than a and b. Footnote in 4.2 could be inline text with similar space. 4.3 second paragraph the word table is not capitalized like elsewhere. 4.3 4th paragraph the word section is not capitalized like elsewhere. Edit: I appreciate the authors responding to my questions but still feel the relatively poor localization performance at stricter IoU thresholds fails to justify the complexity of the approach. I encourage the authors to continue pursuing this line of research.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Semi-Supervised Detection of Extreme Weather Events in Large Climate Datasets
The detection and identification of extreme weather events in large scale climate simulations is an important problem for risk management, informing governmental policy decisions and advancing our basic understanding of the climate system. Recent work has shown that fully supervised convolutional neural networks (CNNs) can yield acceptable accuracy for classifying well-known types of extreme weather events when large amounts of labeled data are available. However, there are many different types of spatially localized climate patterns of interest (including hurricanes, extra-tropical cyclones, weather fronts, blocking events, etc.) found in simulation data for which labeled data is not available at large scale for all simulations of interest. We present a multichannel spatiotemporal encoder-decoder CNN architecture for semi-supervised bounding box prediction and exploratory data analysis. This architecture is designed to fully model multi-channel simulation data, temporal dynamics and unlabelled data within a reconstruction and prediction framework so as to improve the detection of a wide range of extreme weather events. Our architecture can be viewed as a 3D convolutional autoencoder with an additional modified one-pass bounding box regression loss. We demonstrate that our approach is able to leverage temporal information and unlabelled data to improve localization of extreme weather events. Further, we explore the representations learned by our model in order to better understand this important data, and facilitate further work in understanding and mitigating the effects of climate change.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This work presents a novel 3D CNN architecture for climate event detection that combines an unsupervised auto-encoder reconstruction loss with YOLO like bounding box prediction. The approach is trained and evaluated on a large-scale, simulated climate dataset labeled by a costly heuristic approach called TECA. For the most part, the paper is nicely written (minor comments below) and addresses an important and well motivated problem. The authors provide sufficient model details to allow reproduction (although public code would be preferred). I find the experiments a bit unconvincing (see below) but appreciate the attention to model capacity (via number of parameter) when comparing the 2D and 3D model variants. I am concerned that the evaluation may be insufficient to assess the effectiveness of this method. An IoU threshold of 0.1 allows for many rather poor detections to count as true positives. If the goal of this work is to count the number of such instances this is defensible, but for localization this seems overly loose.Furthermore, the 3D CNN architecture (which is one of the the core novelties of this work compared to past efforts) does not seem capable of producing variable sized boxes (as noted in the last paragraph of page 7), which I imagine results in poorer performance at higher IoU thresholds (as many of the weather events appear small). The experiments also feel inconclusive about the effect of temporal modeling and semi-supervision. The temporal component does not seem to matter in the supervised settings (2D 51.45 mAP - 3D 51.00 mAP) but improves somewhat in the semi-supervised case (2D 51.11 mAP - 3D 52.92 mAP). Whereas the additional unlabeled data seems to hurt in the 2D case but improve results for the 3D model. Could the authors provide confidence intervals for these numbers? I would like to see further discussion of these trends especially with respect to the effect of the loss weights (alpha, beta, and gamma). I also note that it is not clear if both the 2D and 3D models were trained for equivalent time periods (seems like no from last paragraph of page 7). Could a plot of training and validation accuracy for each model be presented for comparison? Finally, is there any baseline approach the authors could report or compare too? Without one, it is difficult to evaluate the performance of the approach with respect to the difficulty of the problem. Preliminary Rating: I think this is an interesting paper that is well motivated but feel the experiments as presented do not seem adequate to support any conclusive trends. I would like to see the mAP trends across a wider range of IoU values and further discussion of training procedure, loss weight settings, and reasons for lack of bounding box variability in the 3D model (as stated above). Clarification: In the paper you say *While climate models are run on a 3D grid, with the vertical dimension corresponding to 30 levels; we only consider surface quantities (i.e. 2D data) in this study.* Could you elaborate on what the surface quantities correspond to? Is it the highest cloud level? Minor notes: Please provide years for Prabhat et al. references rather than a and b. Footnote in 4.2 could be inline text with similar space. 4.3 second paragraph the word table is not capitalized like elsewhere. 4.3 4th paragraph the word section is not capitalized like elsewhere. Edit: I appreciate the authors responding to my questions but still feel the relatively poor localization performance at stricter IoU thresholds fails to justify the complexity of the approach. I encourage the authors to continue pursuing this line of research.
assistant
This work presents a novel 3D CNN architecture for climate event detection that combines an unsupervised auto-encoder reconstruction loss with YOLO like bounding box prediction. The approach is trained and evaluated on a large-scale, simulated climate dataset labeled by a costly heuristic approach called TECA. For the most part, the paper is nicely written (minor comments below) and addresses an important and well motivated problem. The authors provide sufficient model details to allow reproduction (although public code would be preferred). I find the experiments a bit unconvincing (see below) but appreciate the attention to model capacity (via number of parameter) when comparing the 2D and 3D model variants. I am concerned that the evaluation may be insufficient to assess the effectiveness of this method. An IoU threshold of 0.1 allows for many rather poor detections to count as true positives. If the goal of this work is to count the number of such instances this is defensible, but for localization this seems overly loose.Furthermore, the 3D CNN architecture (which is one of the the core novelties of this work compared to past efforts) does not seem capable of producing variable sized boxes (as noted in the last paragraph of page 7), which I imagine results in poorer performance at higher IoU thresholds (as many of the weather events appear small). The experiments also feel inconclusive about the effect of temporal modeling and semi-supervision. The temporal component does not seem to matter in the supervised settings (2D 51.45 mAP - 3D 51.00 mAP) but improves somewhat in the semi-supervised case (2D 51.11 mAP - 3D 52.92 mAP). Whereas the additional unlabeled data seems to hurt in the 2D case but improve results for the 3D model. Could the authors provide confidence intervals for these numbers? I would like to see further discussion of these trends especially with respect to the effect of the loss weights (alpha, beta, and gamma). I also note that it is not clear if both the 2D and 3D models were trained for equivalent time periods (seems like no from last paragraph of page 7). Could a plot of training and validation accuracy for each model be presented for comparison? Finally, is there any baseline approach the authors could report or compare too? Without one, it is difficult to evaluate the performance of the approach with respect to the difficulty of the problem. Preliminary Rating: I think this is an interesting paper that is well motivated but feel the experiments as presented do not seem adequate to support any conclusive trends. I would like to see the mAP trends across a wider range of IoU values and further discussion of training procedure, loss weight settings, and reasons for lack of bounding box variability in the 3D model (as stated above). Clarification: In the paper you say *While climate models are run on a 3D grid, with the vertical dimension corresponding to 30 levels; we only consider surface quantities (i.e. 2D data) in this study. * Could you elaborate on what the surface quantities correspond to? Is it the highest cloud level? Minor notes: Please provide years for Prabhat et al. references rather than a and b. Footnote in 4.2 could be inline text with similar space. 4.3 second paragraph the word table is not capitalized like elsewhere. 4.3 4th paragraph the word section is not capitalized like elsewhere. Edit: I appreciate the authors responding to my questions but still feel the relatively poor localization performance at stricter IoU thresholds fails to justify the complexity of the approach. I encourage the authors to continue pursuing this line of research.