BMC Medical Research Methodology
Experts' perceptions on the use of visual analytics for complex mental healthcare planning: an exploratory study
Erin I. Walsh1,2,
Younjin Chung ORCID: orcid.org/0000-0002-2055-68453,
Nicolas Cherbuin1 &
Luis Salvador-Carulla3
BMC Medical Research Methodology volume 20, Article number: 110 (2020) Cite this article
Health experts including planners and policy-makers face complex decisions in diverse and constantly changing healthcare systems. Visual analytics may play a critical role in supporting analysis of complex healthcare data and decision-making. The purpose of this study was to examine the real-world experience that experts in mental healthcare planning have with visual analytics tools, investigate how well current visualisation techniques meet their needs, and suggest priorities for the future development of visual analytics tools of practical benefit to mental healthcare policy and decision-making.
Health expert experience was assessed by an online exploratory survey consisting of a mix of multiple choice and open-ended questions. Health experts were sampled from an international pool of policy-makers, health agency directors, and researchers with extensive and direct experience of using visual analytics tools for complex mental healthcare systems planning. We invited them to the survey, and the experts' responses were analysed using statistical and text mining approaches.
The forty respondents who took part in the study recognised the complexity of healthcare systems data, but had most experience with and preference for relatively simple and familiar visualisations such as bar charts, scatter plots, and geographical maps. Sixty-five percent rated visual analytics as important to their field for evidence-informed decision-making processes. Fifty-five percent indicated that more advanced visual analytics tools were needed for their data analysis, and 67.5% stated their willingness to learn new tools. This was reflected in text mining and qualitative synthesis of open-ended responses.
This exploratory research provides readers with the first self-report insight into expert experience with visual analytics in mental healthcare systems research and policy. In spite of the awareness of their importance for complex healthcare planning, the majority of experts use simple, readily available visualisation tools. We conclude that co-creation and co-development strategies will be required to support advanced visual analytics tools and skills, which will become essential in the future of healthcare.
Visual analytics are useful decision-making aids for complex healthcare systems
Two thirds of experts understand the importance of visual analytics
Over half of experts expect more advanced visual analytics tools
User-friendliness is key to expert engagement with visual analytics
Co-development of future visual analytics tools may address low expert uptake
In order to monitor, evaluate, and plan for the future of mental healthcare systems, health experts including policy-makers require a deep knowledge of a constantly changing landscape of service distribution, management, demand, and outcomes. This is a vital challenge given the high-stakes nature of healthcare decision-making [1]. Understanding healthcare systems is important to health experts for their decision-making in policy, funding, and quality healthcare delivery. This requires reliable evidence, and the growing availability of healthcare data constitutes important evidence for better decision-making.
Visual analytics has great potential to develop evidence from healthcare data of increasing volume and complexity and support evidence-informed decision-making [2]. Visual analytics refers to the science of analytical reasoning facilitated by interactive visual interfaces [3]. It combines automated algorithms (e.g., machine learning) with visualisation techniques for human interaction and cognition in the process of data analysis. Data visualisation is the graphical representation of data, undertaken to illustrate relationships and patterns. It can be used to draw out tacit knowledge that can be incorporated into expert knowledge to improve the decision-making process [4]. Thus, visual analytics capitalises on complex data, enabling hypothesis generation, hidden pattern identification, interest expression, insight in data, evidence development, and communication for action. This involves missing value imputation, pattern identification and prediction, as well as statistical description and inference.
Individual or combined visualisations such as histograms, line graphs, bar charts, scatter plots, networks, and/or geographical maps are widely used to provide overviews, descriptions, and statistical summaries of data [5]. Visual analytics also plays an important role in exploratory and predictive analyses for complex and nonlinear pattern information to support actionable decision-making (e.g., policy changes). The application of visual analytics as a tool for complex data analysis and decision support can be found in healthcare. Examples include a cohort clustering analysis using disk-like visualisation in public healthcare [6], and an associative service pattern analysis using grid map visualisation in mental healthcare [7]. However, a lack of understanding, availability, development, and application of visual analytics methods answering complex questions persists in the process of evidence development and decision-making [8, 9].
In mental healthcare systems research and policy, there is a critical need to study effective ways of developing and using visual analytics to support the analysis of complex data for decision-making [9]. This involves methodological development of visual analytics methods with experts to allow appropriate understanding, reasoning processes and skills in discovering complex information, and generating evidence-informed knowledge. A recent review has examined mental health expert usage of visual analytics in published studies [9]. However, in face-to-face interviews with Australian health experts, only one quarter reported referring to academic journals when gathering evidence for policy with the majority turning to other organisations or colleagues [10]. This indicates that experience and knowledge among health experts may not be the same as that contained in scholarly research publications. This may contribute to an opportunity-cost arising from a disconnect between the availability of advanced visual analytics tools in the scholarly literature - which can keep pace with complex data analysis - and their actual utilisation in decision-making.
There is currently a lack of information on the experience of experts engaging with visual analytics and their expectations for visual analytics tools that may be developed for better data analysis and decision-making processes. This may be because literature searches are largely insensitive to this information, which is better revealed by qualitative and self-report measures (e.g., [11]). Surveys and interviews are undertaken in such circumstances to explore the extent and nature of a research activity, prior to larger scale literature review or data collection [12]. This approach can be the first step toward an ongoing process where domain experts are involved in development and iterative improvement of a system or tool, known as co-development [13]. Co-development has demonstrated benefits in healthcare decision-making [4], and in the current setting may leverage insight to produce more useful visual analytic methods and equip health experts with skills for applying novel visualisation techniques. To support the application of a co-development approach, we need to improve our understanding of how healthcare experts engage with and perceive complex data visualisations.
The aims of this exploratory study were: (1) to examine the real-world experience that experts in mental health have with visual analytics; (2) to investigate how well widely available visual analytics techniques meet their needs; and (3) to codify their understanding and expectations to establish priorities for the future development of visual analytics tools of practical benefit to mental healthcare policy and decision-making. In order to guide further research, we also examine whether experience, needs, and priorities for future development differ based on the expert's current working area (policy, research, or service delivery) and the degree of data complexity. To our knowledge, this is the first study to date to explore expert understanding and expectations of visual analytics for complex mental healthcare system planning.
Participants were sampled from an expert pool of policy-makers, health agency directors, and researchers with direct experience of using visual analytics tools for complex mental healthcare systems planning. This was achieved via snowball sampling through two networks in Australia and in Europe that included experts of mental health policy research and in mental health economics:
The VIDEA (VIsual and DEcision Analytics) Lab - a visual analytics service research hub based at the Australian National University; and
The European PECUNIA project (ProgrammE in Costing, resource use measurement and outcome valuation for Use in multi-sectoral National and International health economic evaluAtions) consortium.
A total of 151 email invitations were sent at the end of May 2019. Forty respondents completed the survey (27% response rate from 151 invitations). Participants included chief executive officers, professors, and international project directors. They had a median of 15 years (1–55 year range, Standard Deviation = 12 years) working experience dealing with mental healthcare systems data. The most common areas of work were research (32%), policy (25%), and service provision (10%). Others reported working in the area of health economics (7.5%), defence (2.5%), communication (2.5%), advocacy (2.5%), and teaching (2.5%). The rest (15.5%) was either unspecified or fell within the broad category of 'health'. Respondents were mostly based in Australia (55%) with others being located in Europe (Spain 20%, United Kingdom 2.5%, Denmark 2.5%, France 2.5%, Austria 2.5%, Switzerland 2.5%, and the Netherlands 2.5%). Eleven percent of respondents did not specify their country of origin.
Respondents followed a link from invitation emails to complete an online questionnaire via the Qualtrics survey platform. Survey questions are presented alongside results, and the full questionnaire is included in the supplementary materials. In order to establish expert experience, data challenges, and visual analytic needs, questions were divided into three sections: (1) Demographics (role, area of work, years of expertise, institution and country); (2) Mental healthcare systems data (data types, aspects, analyses, outcomes, and complexity); and (3) Visual analytics (experience, preferences, need for new or advanced tools, and willingness to learn new techniques). The questionnaire concluded with an open-ended invitation for any further comments. Data collection was closed on 31st of July, 2019.
All analyses were undertaken in R [14], using the packages 'tm (v0.6–2)' [15], 'stringr (v1.10)' [16], 'LSAfun (v0.5.1)' [17], and 'ggplot2 (v 2_3.1.0)' [18].
Quantitative analysis consisted of descriptive statistics (frequencies and ranks) and Chi-square (χ2) analyses. In order to quantify expert experience and needs, categorical responses were explored in terms of descriptive statistics (frequencies and ranks). Chi-square (χ2) analysis was undertaken to explore whether or not expert expectations and requirements for visual analytics differed on the basis of work area and perceived data complexity. This analysis can indicate whether or not the differences should be taken into account in future work. To account for unbalanced cell sizes and problems arising from cells with < 5 members [19], the sparser responses in categorical variables were collapsed into 'other' categories. For these analyses, area of work was collapsed into a categorical variable (policy, research, and service provision/other). Degree of data complexity was treated as an ordinal variable due to strong negative skew (toward high complexity) and constrained rating scale (1–5). Visualisation techniques that respondents have the most experience working with were collapsed into categories of basic graphs or charts, geographical maps, and 'other'. Monte Carlo simulation for significance [20] at 10000 replicates was undertaken for significance testing.
Power analysis was also undertaken, using the 'pwr package' in R. We chose a conservatively large power requirement to account for the increased likelihood of type II error (failing to reject a null hypothesis which is true) given the comparatively small sample size [21]. Alpha (α; evidence strength before one rejects the null hypothesis) was set at 0.05, sample size (n) was set to our available sample of 40, and degrees of freedom were set based on the number of categories in both dependent and independent variables. Power calculations for χ2 indicate that our planned analyses were sensitive to large or medium-to-large sizes only [22]. Specifically, the minimum detectable effect size with our data is 0.49 (with n = 40, two degrees of freedom, alpha = 0.05, power = 0.8). Inference based on statistical significance under these conditions can be meaningful if a large effect size is present, but should be interpreted with caution because it may miss smaller trends in the data [19].
Open-ended responses were explored through qualitative synthesis and text mining. For text mining, corpora were created from open-ended responses by removal of stop words, case, punctuation and stemming (e.g., "service", "services", become "service*") using Porter's algorithm [23]. The asterisk symbol (*) indicates word truncation, e.g., 'service*' corresponds to 'service', 'services', 'serviced', etc. Word frequencies were used to identify key terms. Neighbourhood analysis was undertaken to find the terms conceptually most related to "health", while taking into account baseline word frequency in the English language.
Mental healthcare systems data
Participant experience with mental healthcare systems data is summarised in Fig. 1. Participants were most involved with demography and resource utilisation (Fig. 1a). They tended to be involved across the stages of data collection, analysis, and application to planning (Fig. 1b), with a focus on descriptive analysis (Fig. 1c). The most common outcomes of their involvement were policy guidance and future planning (Fig. 1d).
Participant experience with mental healthcare systems data. This figure summarises responses to multiple-choice questions in Section 2 of the questionnaire. In a, b, c, and d, bold label indicates provided options as suggested categories, lighter weight indicates categories entered by respondents under "Other: please specify" of each question
In open-ended responses to "What have you used analytical approaches in your mental healthcare systems data for?" the most frequent term was "service", and the closest correlate of the concept of "health" was "cost-effect*". This indicated a general focus on managing healthcare service planning and delivery within budgetary constraints. On a 5-point Likert scale, participants rated their data as very complex (33%) or complex (22.5%), indicating that mental healthcare systems data was predominantly complex. The highest ranked contributor to this complexity was the data structure (e.g., complex interactions between resources, services, patients, and agents within or between practices and catchments) as seen in Table 1.
Table 1 Participant ranking on the source of complexity in their data
Sixty-five percent of participants indicated that visual analytics is important (rather than neutral or unimportant) to their field. Participants' experience with visual analytics is summarised in Fig. 2. Based on their analytical experience in Fig. 1c for descriptive, statistical, and geospatial analyses, basic graphs and geographical maps were identified as the most commonly applied and preferred visualisations (Fig. 2a and b). In open-ended responses to "What did you use this visualisation method for?" the most frequent terms were "trend" and "service*". The closest correlated terms to the concept of "health" were "hotspot" (a form of visualisation) and "coplot" (software for producing visualisations [24]), indicating a tendency to use visualisations to track spatial or temporal trends.
Participant experience with visual analytics. This summarises responses to multiple-choice questions in Section 3 of the questionnaire
Current experience and needs
Figure 3 depicts overall participant ranking of their experience with visualisation methods across a set of criteria as defined in the questionnaire: applicability (the degree to which the visualisation is meaningful for analysis and subsequent decisions); acceptability (user-friendliness and likelihood of uptake); practicability (ability to effectively implement and interpret); and efficiency (capacity to clearly summarise large and complex data or model results). Participants who experienced visualisation methods rated these criteria primarily as "very adequate" or "adequate", and none rated them as inadequate for understanding data in general. However, only 25% indicated current visual analytics approaches were adequate (rather than neutral or inadequate) to inform healthcare decisions. Given the options 'yes', 'maybe' and 'no', 55% indicated that more advanced visual analytics tools were needed for their data analysis, and a majority of participants (68%) were willing to learn new visual analytics approaches.
Participant rating of visualisation tools in terms of applicability, acceptability, practicability and efficiency
In responses to a final open-ended invitation for further comments, the most frequently used terms were "data", "complex", and "understand". The closest correlated terms to the concept of "health" were "maker" and "system", which are indicative of a focus on expert knowledge integration with healthcare data visualisation. An exemplified response is:
"Visual analytics is essential to understand complex systems such as mental health care data. I think it is necessary to deep into this area to improve user-friendliness and to provide a better overall understanding by political decision makers."
The closest correlated terms to the concept of "visual*" were "analyt*", "barrier*", and "educ*", indicating an understanding that more development and user education are needed. This sentiment is exemplified in the response:
"I find that the lack of accessible, effective visual analytics means many decision makers lack the ability to understand and interpret the data and make decisions informed by the data. I find that many policy and planning processes are based more on anecdotal data and subjective "understandings" of service systems rather than data, even where data exists (noting that data collection in mental health is often inconsistent in both quality and extent)."
χ2 tests indicated that participants from different areas of work (e.g., policy, research or service provision) did not significantly differ in their ratings of the adequacy of currently available visual analytics for informing healthcare decisions, the importance of visual analytics for decision making, the need for new visual analytics tools for decision making, or their willingness to learn these new tools (Table 2). Similarly, perceived data complexity was not significantly associated with expectations of visual analytics as a tool for complex mental healthcare system planning (Table 2). This may suggest that there are no differences in expectations depending on what area the experts work or how complex they perceive their data to be. An important caveat is that the analysis is only powered to detect comparatively large differences between groups.
Table 2 Statistical associations of area of work and data complexity with expectations of visual analytics
This exploratory study has uncovered several promising avenues for further research and development of new and advanced visual analytics methods. The study results have demonstrated that experts in mental healthcare systems are aware of the importance of visual analytics. However, the health experts revealed that their approach to visual analytics was generally restricted to descriptive analysis with simple and familiar visualisations. Lack of understanding and low user-friendliness of advanced tools appeared to be major barriers to their application in the process of decision-making. Consequently, our results showed a demand for further co-development of effective and accessible visual analytics tools of practical benefit to mental healthcare policy planning. Further quantitative analysis revealed that these expectations do not largely differ on the basis of area of work (e.g., policy, research, or service provision) or data complexity. This suggests that the development of a general approach for engaging with health experts should be favoured in order to develop better visual analytics methods and increase their use. Thus, further work may benefit from focusing on expert understanding and needs, which can be achieved through co-development of both tools and skills.
User-friendliness was an underlying theme for health experts in both their current experience and requirements for the development of future visual analytics tools. This theme is vital to overcome the major barriers to investment in information parsing and decision-making processes, including time pressure [10], less developed expertise [25], poor visual literacy [26], and low trust in the data, analysis, or software developers underlying visualisation [27]. Thus, a co-development framework can help to achieve user-friendliness in the development of visual analytic tools. Fisher and Green [28] provided such a framework for the co-development of visual analytics in translational cognitive science and highlighted the applicability of the approach across projects and disciplines. There is some preliminary evidence of the benefits of this approach in practice, e.g., Freebairn and Atkinson [27] developed a dynamic simulation model for physical health planning and noted the co-development process led to greater expert uptake of the tool.
The mental health experts indicated working with complex data, yet predominantly reported using relatively simple descriptive statistics methods for their data analysis. They also rated these methods using basic graphs, charts, and geographical maps as highly applicable, acceptable, practical, and efficient. This may be symptomatic of the predominance of descriptive analysis applied by our sample driving reliance on simpler visualisation methods integrated into commonly used software such as Microsoft Excel, a software trend that has been observed in the context of disease epidemiology [29]. It is possible that while the data and its information processing are complex, the key information can be adequately visualised using familiar motifs (e.g., error bars are visually the same regardless of whether they are calculated via parametric or comparatively complex non-parametric iterative approaches). Most likely, the inherent reduction of complexity necessitated by simpler visualisations may meet the needs of speed of interpretability [10] and user-friendliness, at the cost of obscuring multidimensionality and hidden pattern information available in the underlying data [30]. Consistent, structured design and education to expand visual literacy can be effective in increasing expert engagement with more advanced visualisations and visual analytics tools [1]. However, such strategies have yet to achieve a widespread impact on expert utilisation of more complex approaches. For a shift to match complex data with more information-dense and non-standard visualisations, there is a need of further research on the degree of information loss from expert preference for visual simplicity and consequences for how decisions are made.
This study has a number of limitations. Healthcare systems expertise was a restrictive participation criterion. A more comprehensive scope not limited to mental healthcare would have provided a broader perspective on the experts' perceptions on visual analytics and tools. However, it is important to consider that mental healthcare is at the core of integrated, multi-sectoral and complex approaches in healthcare systems research [31, 32], thus is a start point for this exploratory research. While representing a strong cross-section of senior mental healthcare systems perspectives (e.g., policy, provision and research) in Australia and in Europe, the response rate and small sample size limit generalisability. A reliance on self-report is vulnerable to recall bias and precludes more detailed investigation of participant engagement with visual analytics. However, the low response rate is a common problem to the majority of on-line surveys, and low response rates - generally above 20% - have demonstrated consistent and accurate results in previous studies [33, 34]. Although the sample size was sufficient for qualitative analyses (e.g., [35]), our quantitative comparison across area of work and degree of data complexity could only detect large effect sizes. This does suggest the absence of major differences between experts in policy, research, and service provision. Future research is required to establish whether there are subtle but meaningful differences in the experiences and expectations of experts depending on their area of work and complexity of data they work with.
This study examined the perceptions and the real-world experience that experts in mental healthcare planning have with visual analytics tools. We investigated how well current visualisation techniques meet the expert needs and established suggestions for future development of visual and decision analytics, using a targeted online survey and combination of qualitative and quantitative analysis. Notably, this study was the first examination of self-report expert experience on the topic of using visual analytics in mental healthcare systems research and policy. Our preliminary findings indicated that, despite a clear need to extract information from highly complex data, experts tend to utilise visualisations that are most familiar to them, widely understood, and not necessarily the most appropriate. This undermined full utilisation of the depth of available evidences and risks leading to a shallow basis for mental healthcare decision-making. Positively, we found that mental health experts are aware of the value of visual analytics and open to further developments to enhance their applicability. We suggest that co-development may be the most fruitful approach in future research intended to develop effective and useful visual analytics tools for complex mental healthcare systems data.
The datasets used and analysed during the current study are available from the corresponding author on reasonable request.
VIsual and DEcision Analytics
PECUNIA:
ProgrammE in Costing, resource use measurement and outcome valuation for Use in multi-sectoral National and International health economic evaluAtions
VA:
Ola O, Sedig K. Beyond simple charts: design of visualizations for big health data. Online J Public Health Inform. 2016;8(3):e195.
Caban JJ, Gotz D. Visual analytics in healthcare–opportunities and research challenges. Journal of the American Medical Informatics Association. 2015;22(2):260–62.
Keim D, Andrienko G, Fekete J-D, Görg C, Kohlhammer J, Melançon G. Visual analytics: definition, process, and challenges. Berlin: Information visualization: Springer; 2008. p. 154–75.
Gibert K, García-Alonso C, Salvador-Carulla L. Integrating clinicians, knowledge and data: expert-based cooperative analysis in healthcare decision support. Health Res Policy Syst. 2010;8(1):28.
Preim B, Lawonn K. A survey of visual analytics for public health. Computer graphics forum: Eurographics; 2019;39(3):1–35.
Alemzadeh S, Hielscher T, Niemann U, Cibulski L, Ittermann T, Völzke H, et al. Subpopulation discovery and validation in epidemiological dataEuroVA@EuroVis; 2017.
Chung Y, Salvador-Carulla L, Salinas-Pérez JA, Uriarte-Uriarte JJ, Iruin-Sanz A, García-Alonso CR. Use of the self-organising map network (SOMNet) as a decision support system for regional mental health planning. Health Res Policy Syst. 2018;16(1):35.
Chishtie JA, Babineau J, Bielska IA, Cepoiu-Martin M, Irvine M, Koval A, et al. Visual analytic tools and techniques in population health and health services research: protocol for a scoping review. JMIR Res Protoc. 2019;8(10):e14019.
Chung Y, Bagheri N, Salinas-Perez JA, Smurthwaite K, Walsh E, Furst M, et al. Role of visual analytics in supporting mental healthcare systems research and policy: a systematic scoping review. Int J Inf Manag. 2020;50:17–27.
O'Donoughue Jenkins L, Kelly PM, Cherbuin N, Anstey KJ. Evaluating and using observational evidence: the contrasting views of policy makers and epidemiologists. Front Public Health. 2016;4:267.
Benjamin Martz Jr W, Shepherd MM. Testing for the transfer of tacit knowledge: making a case for implicit learning. Decis Sci J Innov Educ. 2003;1(1):41–56.
Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5(1):69.
Morisset C, Sanchez D. On building a visualisation tool for access control policies. Cham: International Conference on Information Systems Security and Privacy: Springer; 2018.
Team RC. R: a language and environment for statistical computing; 2013.
Feinerer I. Introduction to the tm package text Mining in RRetrieved March. 2018;1; 2019.
Wickham H. Stringr: modern, consistent string processing. R J. 2010;2(2):38–40.
Günther F, Dudschig C, Kaup B. LSAfun-an R package for computations based on latent semantic analysis. Behav Res Methods. 2015;47(4):930–44.
Wickham H. ggplot2: elegant graphics for data analysis: springer; 2016.
VanVoorhis CW, Morgan BL. Understanding power and rules of thumb for determining sample sizes. Tutor Quant Methods Psychol. 2007;3(2):43–50.
Hope AC. A simplified Monte Carlo significance test procedure. J R Stat Soc Ser B Methodol. 1968;30(3):582–98.
Kim H-Y. Statistical notes for clinical researchers: type I and type II errors in statistical decision. Restor Dent Endod. 2015;40(3):249–52.
Cohen J. Statistical power analysis for the behavioral sciences: Routledge; 2013.
Book Google Scholar
Willett P. The porter stemming algorithm: then and now. Program. 2006;40(3):219–23.
Bravata DM, Shojania KG, Olkin I, Raveh A. CoPlot: a tool for visualizing multivariate data in medicine. Stat Med. 2008;27(12):2234–47.
Hannus M, Hyönä J. Utilization of illustrations during learning of science textbook passages among low-and high-ability children. Contemp Educ Psychol. 1999;24(2):95–123.
Herráez A, Costa MJ. Biochemical visual literacy with constructive alignment: outcomes, assessment, and activities; 2013.
Freebairn L, Atkinson J-A, Kelly PM, McDonnell G, Rychetnik L. Decision makers' experience of participatory dynamic simulation modelling: methods for public health policy. BMC Med Inform Decis Mak. 2018;18(1):131.
Fisher B, Green TM, Arias-Hernández R. Visual analytics as a translational cognitive science. Top Cogn Sci. 2011;3(3):609–25.
Carroll LN, Au AP, Detwiler LT, T-c F, Painter IS, Abernethy NF. Visualization and analytics tools for infectious disease epidemiology: a systematic review. J Biomed Inform. 2014;51:287–98.
Lanters CA, Fantke P. Structuring complex results using network maps and hierarchical charts. Procedia CIRP. 2018;69:441–6.
Salvador-Carulla L, Haro J, Ayuso-Mateos J. A framework for evidence-based mental health care and policy. Acta Psychiatr Scand. 2006;114:5–11.
Furst MA, Gandré C, López-Alberca CR, Salvador-Carulla L. Healthcare ecosystems research in mental health: a scoping review of methods to describe the context of local care delivery. BMC Health Serv Res. 2019;19(1):173.
Mealing NM, Banks E, Jorm LR, Steel DG, Clements MS, Rogers KD. Investigation of relative risk estimates from studies of the same population with contrasting response rates and designs. BMC Med Res Methodol. 2010;10(1):26.
Morton SM, Bandara DK, Robinson EM, Carr PEA. In the 21st century, what is an acceptable response rate? Aust N Z J Public Health. 2012;36(2):106–8.
Richardson M, Domingos P. Learning with knowledge from multiple experts. Proceedings of the 20th international conference on machine learning (ICML-03); 2003.
We thank the following respondents, who provided their names for attribution and valuable responses to the survey. Alexander Lim, Amber Shuhyta, Anna Brooks, Annette Erlangsen, Bianca Calabria, Carlos Pereira Rodriguez, Clara Ha, Denise Riordan, Elizabeth Moore, Federico Alonso-Trujillo, Harry Lovelock, Helen Benassi, Helen Killaspy, Ilaria Montagni, John Acs, John Mendoza, Jose A. Salinas-Perez, Judit Simon, Kerry Hawkins, Kinley Wangdi, Lilisbeth Perestelo Perez, Maria Luisa Rodero Cosano, Maria Rubio-Valera, Nasser Bagheri, Paul Mayers, Phil Batterham, Pilar Campoy, Ruben Drost, Sarah Pollock, Scott Henderson, Simon Viereck, Sue Lukersmith, and Wei Du. We also thank another seven participants, who anonymously provided precious responses to the survey. We specially thank Mencia R. Gutierrez-Colosia for coordinating the submission of this survey in Europe, and the contribution of other members of the PECUNIA Group. The PECUNIA project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement (No. 779292). The authors would also like to acknowledge Dr. Richard Terrett for proof reading assistance.
The study was supported by the 2019 Excellence in Population Health Research Award (Research School of Population Health, Australian National University).
Centre for Research on Ageing, Health and Wellbeing, Research School of Population Health, College of Health and Medicine, Australian National University, Canberra, Australia
Erin I. Walsh & Nicolas Cherbuin
PHXchange (Population Health Exchange), Research School of Population Health, College of Health and Medicine, Australian National University, Canberra, Australia
Erin I. Walsh
Centre for Mental Health Research, Research School of Population Health, College of Health and Medicine, Australian National University, 63 Eggleston Road, Acton, ACT, 2601, Australia
Younjin Chung & Luis Salvador-Carulla
Younjin Chung
Nicolas Cherbuin
Luis Salvador-Carulla
EIW contributed conceptually to the study, contributed to questionnaire design, conducted analysis, and led the drafting of the manuscript. YC conceptually to the study, contributed to questionnaire design, managed questionnaire distribution, and contributed substantially to the analysis and drafting of the manuscript. NC contributed conceptually to the study design and contributed to the drafting of the manuscript. LSC contributed conceptually to the study, provided guidance and contacts for questionnaire distribution, and contributed to the drafting of the manuscript. All authors have read and approved the final manuscript.
Correspondence to Younjin Chung.
All participants provided written, informed consent. The ethical aspects of this study have been approved by the Australian National University Human Research Ethics Committee (Protocol 2019/069).
Informed consent for publication of individual responses was obtained from all respondents. Those participants listed in the acknowledgements were presented with the choice to participate anonymously and by providing their names in informed written consent to be named in the acknowledgements.
Additional file 1.
The online survey questionnaire used via Qualtrics platform for the data collection of this study.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Walsh, E.I., Chung, Y., Cherbuin, N. et al. Experts' perceptions on the use of visual analytics for complex mental healthcare planning: an exploratory study. BMC Med Res Methodol 20, 110 (2020). https://doi.org/10.1186/s12874-020-00986-0
DOI: https://doi.org/10.1186/s12874-020-00986-0
Expert experience
Complex data analysis
Mental healthcare systems
Evidence-informed decision-making
Submission enquiries: bmcmedicalresearchmethodology@biomedcentral.com