<title>Introduction</title> <p>The World Wide Web is now recognized as an important source of information in supporting the practice of evidence-based medicine [##REF##8963525##1##] and consumer health care decision making [##REF##11720961##2##]. While much research focuses on the impact of information retrieval on clinical decision making, there has been little examination of how online searching influences the way consumers make health-related decisions.</p> <p>Many studies have examined the quality of online health care consumer information [##REF##12020305##3##], the tools and initiatives developed to promote health literacy [##REF##15561782##4##], as well as the characteristics of websites and search engines that influence the way consumers perceive and utilize information [##REF##12893614##5##,##REF##12615824##6##]. Of particular relevance to understanding the way consumers use online health-related information, past studies have examined consumers’ familiarity with health vocabulary [##REF##17478414##7##], their information appraisal [##REF##15155597##8##] and search query reformulation skills [##REF##16221944##9##], the way they perceive and assess Web-based health information [##UREF##0##10##, ####REF##15224975##11##, ##REF##16492004##12##, ##REF##14713661##13####14713661##13##], the types of online sources they trust [##REF##14517112##14##], the patterns of use and barriers experienced while using online resources [##UREF##1##15##], and how access to online information influences the way they interact with health care professionals [##REF##16464924##16##, ####REF##12885689##17##, ##REF##16344419##18####16344419##18##].</p> <p>Studies have also shown that people are an important source of influence among consumers with a health-related concern. In a randomized controlled trial conducted by Lorig et al, patients with back pain who had access to an email discussion group demonstrated greater improvement in pain and made less physician visits than those without access [##REF##11926853##19##]. Patients with breast cancer participating in electronic support groups are reported to have reduced rates of depression and lessened reactions to pain [##REF##12569591##20##].</p> <p>Little, however, is known about whether consumers are actually able to improve their understanding of health issues after searching the Web. In addition, little is known about the extent to which social feedback affects the way consumers develop their understanding of health issues. This prospective experiment tests the following hypotheses: (1) consumers can improve their accuracy in answering health care questions after searching tested online resources, and (2) consumers’answers to health care questions are influenced by feedback with other consumers’ answers.</p>
<title>Methods</title> <title>Study Design</title> <p>A convenience sample of 227 undergraduate students was recruited from the University of New South Wales (UNSW). Subjects were asked to use a specific online search engine to answer six consumer health questions. People with Internet access who had previously used an online search engine were recruited by announcements via student email lists, posters, leaflets, weekly student magazines, and a UNSW research news website. Upon completion of the study, subjects were entered into a draw for one of 100 movie tickets. Ethics approval was obtained from the Human Research Ethics Advisory Panel at UNSW.</p> <p>A pre/post protocol was used in this study. Subjects recorded their pre- and post-search answers to each question and their confidence in these answers. After answering each question post-search, subjects were presented with a summary of the post-search answers provided by previous subjects and were asked to answer the question again (##FIG##0##Figure 1##).</p> <p>
                    
                </p> <p>Each question and the expected correct answer are shown in ##TAB##0##Table 1##. All scenario questions were randomly allocated. There were four options to answer a question: “yes,” “no,” “conflicting evidence,” and “don’t know.” Confidence was measured by a 4-point Likert scale from “very confident” to “not confident.” The questions ranged in difficulty and topic in order to cover a spectrum of health care consumer questions. They were developed in consultation with a general practitioner and two academics from the School of Public Health and Community Medicine at UNSW. Agreement was reached on the “correct” answer and the location of the best evidence sources for each question. A pilot test with three members of the general public tested the questions for interest and readability. Two additional pilots of five people each were conducted to confirm that it was possible to locate documentary evidence required to answer the questions correctly.</p> <p>The search engine retrieved documents from tested resources known to have high relevance in answering health-related questions [##REF##16403716##21##]. These resources are PubMed [##UREF##2##22##], MedlinePlus [##UREF##3##23##], and HealthInsite [##UREF##4##24##]. Overall, subjects were advised to spend about 10 minutes for each question and to use only the provided search system to answer the questions. To prevent subjects from visiting external websites during the experiment, the navigational bar on the Web browser was hidden once the subject logged on to the study website.</p> <title>Data Analysis</title> <p>Subjects’ searches and their selected documents, pre-/post-search answers and confidence, post-feedback responses, time taken from answering the question pre-search to answering post-search, and responses to the pre-search and post-search questionnaire were logged during the experiment. Responses to questions were coded as “correct,” “don’t know,” or “incorrect” according to the predetermined answers for each question. All cases in which subjects did not conduct a search before providing an answer or seeking the social feedback, did not answer the question post-search, or answered “don’t know” post-search were removed from the data analysis.</p> <p>The test for difference between proportions was used to compare differences between subjects’ pre-search, post-search, and post-feedback answers and to compare changes in confidence in answers pre- and post-search. The chi-square test was used to examine whether there was a statistically significant relationship between subjects’ confidence in their post-search answers and their tendency to change answers after feedback with other subjects’ answers. The McNemar test was used to examine the direction of change in pre- and post-feedback answers.</p>
<title>Results</title> <title>Subjects and Sample</title> <p>After data exclusion (##FIG##1##Figure 2##), the study consisted of 211 subjects who made 928 responses, 1606 searches, and 3019 document accesses. ##TAB##1##Table 2## presents demographic attributes and self-rated search skills and frequency of searching the Web for general topics and health-related issues. Overall, subjects on average took 361 seconds (SD 281.2) to search, made 1.73 (SD 1.391) searches, and accessed 3.25 (SD 3.067) documents to answer a question.</p> <p>
                    
                </p> <title>Impact on Decision Accuracy</title> <p>As shown in ##TAB##2##Table 3##, most subjects, 56.5% (95% CI: 53.3-59.6), answered correctly both before and after searching, which was termed right-right (RR). This was followed by 25.5% (95% CI: 22.8-28.4) who improved their answers after searching, wrong-right (WR), 13.3% (95% CI: 11.2-15.6) who never answered correctly, wrong-wrong (WW), and 4.7% (95% CI: 3.6-6.3) who went from right to wrong (RW).</p> <p>The test for difference between proportions shows that there was a statistically significant improvement (21%) in the percentage of correct answers before and after searching(pre-search 61.2% [95% CI: 58.0-64.3]; post-search 82.0% [95% CI: 79.4-84.3]; <italic>z</italic>= −1.21, <italic>P &lt;</italic>.001). There was also a marginal significant improvement in the percentage of correct answers before and after feedback with other subjects’ answers (pre-feedback 82.0% [95% CI: 79.4-84.3]; post-feedback 85.3% [95% CI: 82.9-87.5]; <italic>z</italic>= −1.95, <italic>P =</italic>.051; ##TAB##3##Table 4##).</p> <title>Impact of Confidence</title> <p>
                    ##TAB##4##Table 5## shows that the most frequently self-reported change in confidence for all responses before and after searching was “increased confidence” (WW 51.9% [95% CI: 42.5-61.0], WR 54.0% [95% CI: 46.3-61.6], RW 40.4% [95% CI: 27.6-54.7], RR 71.1% [95% CI: 67.4-74.6]).</p> <p>More than half of subjects (55.6%; 95% CI: 37.3-72.4) who did not know the answer pre-search and answered incorrectly post-search (DW) reported that they were confident or very confident with their incorrect post-search answer (##TAB##5##Table 6##). In fact, 82.0% (95% CI: 75.5-87.1) of subjects who were incorrect post-search reported being confident or very confident with their post-search answer (##TAB##6##Table 7##). Although ##TAB##6##Table 7## shows that the proportion of subjects with highly confident correct answers (ie, confident or very confident) significantly increased after searching (pre-search 61.6% [95% CI: 57.6-65.5]; post-search 95.5% [95% CI: 93.8-96.8]; <italic>z</italic>= –15.60, <italic>P &lt;</italic>.001), the proportion of subjects with highly confident incorrect answers also increased after searching (pre-search 55.3% [95% CI: 50.1-60.3]; post-search 82.0% [95% CI: 75.5-87.1]; <italic>z</italic>= –6.75, <italic>P &lt;</italic>.001).</p> <title>Impact of Social Feedback</title> <p>Those who were not as confident in their post-search answers were 28.5% more likely than those who had higher levels of confidence to change their answer after feedback with other subjects’ post-search answers (not confident / somewhat confident 34.4% [95% CI: 23.9-46.6]; confident / very confident 5.9% [95% CI: 4.5-7.7]; <italic>χ</italic>
                    <sup>2</sup>
                    <sub>1</sub>= 66.65, <italic>P &lt;</italic>.001; ##TAB##7##Table 8##). Those who changed their answer after feedback were more likely to change it from wrong to right than from right to wrong (McNemar <italic>χ</italic>
                    <sup>2</sup>
                    <sub>1</sub>= 15.25, <italic>P &lt;</italic>.001; ##TAB##8##Table 9##).</p>
<title>Discussion</title> <p>This research demonstrates that while health care consumers can improve the accuracy of their answers to health care questions after searching quality online resources, their confidence in answers is not a good indicator of the answer being correct. Further, consumers who are not confident in their answers after searching are more likely to be influenced to change their views after feedback with other consumers’ answers.</p> <p>Results of this study for nonclinically trained users are in line with studies that reported search engines can improve the ability of clinically trained users to answer questions [##REF##15684126##25##,##REF##11055299##27##,##REF##11971889##28##]. The 21% improvement in accuracy between pre-search and post-search answers reported in this study corresponds with the study conducted by Hersh et al [##REF##11971889##28##], which found that 66 medical and nurse practitioner students were able to improve their answers to a set of five clinical questions by up to 20% before and after using Medline. Our improvement rate also corresponds with the 21% improvement reported for clinicians who used the same search engine to answer eight clinical scenario questions in a controlled laboratory setting (pre-search correct 29% [95% CI: 25-33]; post-search correct 50% [95% CI: 46-54]; <italic>z</italic>= 9.58, <italic>P</italic>&lt; .001) [##REF##15684126##25##].</p> <p>Findings from this research and previous studies have shown that confidence is not always a good indicator of decision accuracy [##REF##15800302##26##,##REF##12224810##29##]. The observation that 55.6% (95% CI: 37.3-72.4) of subjects in this study who did not know the answer before searching reported being confident or very confident in their incorrect post-search answers (DW) concurs with the result reported by Westbrook et al [##REF##15800302##26##], which found that among clinicians who did not know the answer before searching and were incorrect after searching (DW), 60% of doctors and 52% of clinical nurse consultants reported being confident or very confident in their incorrect post-search answer. This has implications for large-scale national surveys (such as those conducted by the Pew Research Center), which often use confidence as a metric to infer public opinion. In addition, confidence often shapes the way people make decisions (eg, in the form of the overconfidence bias [##UREF##5##30##,##UREF##6##31##]), and studies have shown that people can experience cognitive biases while searching for online information to answer questions [##REF##17600097##32##]. These biases, such as the anchoring and order effects, can influence the way people attend to and process information to make a decision. More research is needed to help users assess the impact of their levels of confidence and understand how their confidence might be shaping their beliefs and ability to attend to new information.</p> <p>Our findings on the impact of social feedback also concur with studies that report people are one of the important sources of information that influence clinicians’ and health care consumers’ actions when confronted with a clinical or health-related concern [##REF##11926853##19##,##REF##12569591##20##,##UREF##7##33##, ####REF##16404470##34##, ##REF##11010346##35##, ##REF##11259994##36####11259994##36##]. With the role of the Internet as a social network, typified by the growing interest in sites like Wikipedia, FaceBook, and MySpace, we can envisage more consumers seeking health-related information and advice from online peer networks. However, there appears to be no prior study that has evaluated the health care impact of the social feedback that is possible through such websites. In addition, it is now clear that it is not sufficient to just provide access to reliable online resources for health care consumers. The decisions consumers make are shaped by their confidence and by the influence of their peers and broader social community. Our research suggests that connecting consumers to trustworthy and relevant networks of human resources could be a significant addition to online health resources. As consumers play an increasingly active role in managing their health, it is important not to underestimate the extent to which online search engines and online peer networks can influence the way people manage their health care.</p>
<title>Background</title> <p>The World Wide Web has increasingly become an important source of information in health care consumer decision making. However, little is known about whether searching online resources actually improves consumers’ understanding of health issues.</p> <title>Objectives</title> <p>The aim was to study whether searching on the World Wide Web improves consumers’ accuracy in answering health questions and whether consumers’ understanding of health issues is subject to further change under social feedback.</p> <title>Methods</title> <p>This was a pre/post prospective online study. A convenience sample of 227 undergraduate students was recruited from the population of the University of New South Wales. Subjects used a search engine that retrieved online documents from PubMed, MedlinePlus, and HealthInsite and answered a set of six questions (before and after use of the search engine) designed for health care consumers. They were then presented with feedback consisting of a summary of the post-search answers provided by previous subjects for the same questions and were asked to answer the questions again.</p> <title>Results</title> <p>There was an improvement in the percentage of correct answers after searching (pre-search 61.2% vs post-search 82.0%, <italic>P &lt;</italic>.001) and after feedback with other subjects’ answers (pre-feedback 82.0% vs post-feedback 85.3%, <italic>P =</italic>.051).The proportion of subjects with highly confident correct answers (ie, confident or very confident) and the proportion with highly confident incorrect answers significantly increased after searching (correct pre-search 61.6% vs correct post-search 95.5%, <italic>P &lt;</italic>.001; incorrect pre-search 55.3% vs incorrect post-search 82.0%, <italic>P &lt;</italic>.001). Subjects who were not as confident in their post-search answers were 28.5% more likely than those who were confident or very confident to change their answer after feedback with other subjects’ post-search answers (<italic>χ</italic>
                        <sup>2</sup>
                        <sub>1</sub>= 66.65, <italic>P &lt;</italic>.001).</p> <title>Conclusions</title> <p>Searching across quality health information sources on the Web can improve consumers’ accuracy in answering health questions. However, a consumer’s confidence in an answer is not a good indicator of the answer being correct. Consumers who are not confident in their answers after searching are more likely to be influenced to change their views when provided with feedback from other consumers.</p>
<p>The authors would like to thank Dr. David Thomas and Dr. Ilse Blignault for their assistance in subject recruitment and the development of the health care consumer case scenarios. This research was supported by the Australian Research Council SPIRT grant and APAI scholarship C00107730. The Quick Clinical search engine used in the study was developed with support from NHMRC project grant 300435 and the NHMRC development grant 300591. None of the funding sources had any role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; or the preparation, review, or approval of the manuscript.</p>
<fig id="figure1" position="float"><label>Figure 1</label><caption><p>Screen capture of feedback provided to subjects after answering a question post-search</p></caption></fig> <fig id="figure2" position="float"><label>Figure 2</label><caption><p>Data exclusion procedure</p></caption></fig>
<table-wrap id="table1" position="float"><label>Table 1</label><caption><p>Case scenarios and questions presented to subjects. A random selection of six cases was presented to each subject in the study.</p></caption><table frame="hsides" rules="groups" width="576" border="1" cellpadding="7" cellspacing="0"><col width="466" span="1"/><col width="82" span="1"/><thead><tr valign="top"><td rowspan="1" colspan="1"><bold>Case</bold><bold>Scenario and Question (Scenario Name)</bold></td><td rowspan="1" colspan="1"><bold>Expected Correct Answer</bold></td></tr></thead><tbody><tr valign="top"><td rowspan="1" colspan="1">1. We hear of people going on low carbohydrate and high protein diets, such as the Atkins diet, to lose weight. <italic>Is there evidence to support that low carbohydrate, high protein diets result in greater long-term weight loss than conventional low energy, low fat diets? </italic>(Diet)</td><td rowspan="1" colspan="1">No</td></tr><tr valign="top"><td rowspan="1" colspan="1">2. You can catch infectious diseases such as the flu from inhaling the air into which others have sneezed or coughed, sharing a straw, or eating off someone else’s fork. The reason is because certain germs reside in saliva, as well as in other bodily fluids. Hepatitis B is an infectious disease. <italic>Can you catch Hepatitis B from kissing on the cheek?</italic> (Hepatitis B)</td><td rowspan="1" colspan="1">No</td></tr><tr valign="top"><td rowspan="1" colspan="1">3. After having a few alcoholic drinks, we depend on our liver to reduce the Blood Alcohol Concentration (BAC). Drinking coffee, eating, vomiting, sleeping, or having a shower will not help reduce your BAC. <italic>Are there different recommendations regarding safe alcohol consumption for males and females?</italic> (Alcohol)</td><td rowspan="1" colspan="1">Yes</td></tr><tr valign="top"><td rowspan="1" colspan="1">4. Sudden infant death syndrome (SIDS), also known as “cot death,” is the unexpected death of a baby where there is no apparent cause of death. Studies have shown that sleeping on the stomach increases a baby’s risk of SIDS. <italic>Is there an increased risk of a baby dying from SIDS if the mother smokes during pregnancy?</italic> (SIDS)</td><td rowspan="1" colspan="1">Yes</td></tr><tr valign="top"><td rowspan="1" colspan="1">5. Breast cancer is one of the most common types of cancer found in women. <italic>Is there an increased chance of developing breast cancer for women who have a family history of breast cancer? </italic>(Breast cancer)</td><td rowspan="1" colspan="1">Yes</td></tr><tr valign="top"><td rowspan="1" colspan="1">6. Men are encouraged by our culture to be tough. Unfortunately, many men tend to think that asking for help is a sign of weakness. <italic>In Australia, do more men die by committing suicide than women</italic>? (Suicide)</td><td rowspan="1" colspan="1">Yes</td></tr><tr valign="top"><td rowspan="1" colspan="1">7. Many people use home therapies when they are sick or to keep healthy. Examples of home therapies include drinking chicken soup when sick, drinking milk before bed for a better night’s sleep, and taking vitamin C to prevent the common cold. <italic>Is there evidence to support the taking of vitamin C supplements to help prevent the common cold?</italic> (Cold)</td><td rowspan="1" colspan="1">No</td></tr><tr valign="top"><td rowspan="1" colspan="1">8. We know that we can catch AIDS from bodily fluids, such as from needle sharing, having unprotected sex, and breast-feeding. We also know that some diseases can be transmitted by mosquito bites. <italic>Is it likely that we can get AIDS from a mosquito bite?</italic> (AIDS)</td><td rowspan="1" colspan="1">No</td></tr></tbody></table></table-wrap> <table-wrap id="table2" position="float"><label>Table 2</label><caption><p>Characteristics of subjects (N = 211)</p></caption><table frame="hsides" rules="groups" width="319" border="1" cellpadding="8" cellspacing="0"><col width="171" span="1"/><col width="116" span="1"/><thead><tr valign="top"><td rowspan="1" colspan="1"><bold>Characteristic</bold></td><td rowspan="1" colspan="1"><bold>No. (%)</bold></td></tr></thead><tbody><tr valign="top"><td rowspan="1" colspan="1"><bold>Gender</bold></td><td rowspan="1" colspan="1"/></tr><tr valign="top"><td rowspan="1" colspan="1">    Female</td><td rowspan="1" colspan="1">130 (61.6)</td></tr><tr valign="top"><td rowspan="1" colspan="1">    Male</td><td rowspan="1" colspan="1">81 (38.4)</td></tr><tr valign="top"><td rowspan="1" colspan="1"><bold>Age (years)</bold></td><td rowspan="1" colspan="1"/></tr><tr valign="top"><td rowspan="1" colspan="1">    &lt;25</td><td rowspan="1" colspan="1">139 (65.9)</td></tr><tr valign="top"><td rowspan="1" colspan="1">    25 to 34</td><td rowspan="1" colspan="1">46 (21.8)</td></tr><tr valign="top"><td rowspan="1" colspan="1">    35 to 44</td><td rowspan="1" colspan="1">12 (5.7)</td></tr><tr valign="top"><td rowspan="1" colspan="1">    ≥ 45</td><td rowspan="1" colspan="1">14 (6.6)</td></tr><tr valign="top"><td rowspan="1" colspan="1"><bold>Search skill</bold></td><td rowspan="1" colspan="1"/></tr><tr valign="top"><td rowspan="1" colspan="1">    Fair or poor</td><td rowspan="1" colspan="1">46 (21.8)</td></tr><tr valign="top"><td rowspan="1" colspan="1">    Good</td><td rowspan="1" colspan="1">100 (47.4)</td></tr><tr valign="top"><td rowspan="1" colspan="1">    Very good</td><td rowspan="1" colspan="1">65 (30.8)</td></tr><tr valign="top"><td rowspan="1" colspan="1"><bold>Search frequency</bold></td><td rowspan="1" colspan="1"/></tr><tr valign="top"><td rowspan="1" colspan="1">    Once a week or less</td><td rowspan="1" colspan="1">13 (6.2)</td></tr><tr valign="top"><td rowspan="1" colspan="1">    Several times a week</td><td rowspan="1" colspan="1">198 (93.8)</td></tr><tr valign="top"><td rowspan="1" colspan="1"><bold>Health search frequency</bold></td><td rowspan="1" colspan="1"/></tr><tr valign="top"><td rowspan="1" colspan="1">    Never</td><td rowspan="1" colspan="1">9 (4.3)</td></tr><tr valign="top"><td rowspan="1" colspan="1">    Less than once a week</td><td rowspan="1" colspan="1">94 (44.5)</td></tr><tr valign="top"><td rowspan="1" colspan="1">    Once a week</td><td rowspan="1" colspan="1">52 (24.6)</td></tr><tr valign="top"><td rowspan="1" colspan="1">    Several times a week</td><td rowspan="1" colspan="1">56 (26.5)</td></tr></tbody></table></table-wrap> <table-wrap id="table3" position="float"><label>Table 3</label><caption><p>Changes in answer before and after searching (N = 928; adapted from [##REF##15684126##25##])</p></caption><table frame="hsides" rules="groups" width="541" border="1" cellpadding="7" cellspacing="0"><col width="94" span="1"/><col width="94" span="1"/><col width="177" span="1"/><col width="120" span="1"/><tbody><tr valign="top"><td rowspan="1" colspan="1"><bold>Pre-</bold><bold>Search</bold></td><td rowspan="1" colspan="1"><bold>Post-Search</bold></td><td rowspan="1" colspan="1"><bold>Percentage</bold><bold>(95% CI)</bold></td><td rowspan="1" colspan="1"><bold>Total No.</bold></td></tr></tbody><tbody><tr valign="top"><td rowspan="1" colspan="1">Right</td><td rowspan="1" colspan="1">Right</td><td rowspan="1" colspan="1">56.5 (53.3-59.6)</td><td rowspan="1" colspan="1">524</td></tr><tr valign="top"><td rowspan="1" colspan="1">Wrong</td><td rowspan="1" colspan="1">Right</td><td rowspan="1" colspan="1">25.5 (22.8-28.4)</td><td rowspan="1" colspan="1">237</td></tr><tr valign="top"><td rowspan="1" colspan="1">Wrong</td><td rowspan="1" colspan="1">Wrong</td><td rowspan="1" colspan="1">13.3 (11.2-15.6)</td><td rowspan="1" colspan="1">123</td></tr><tr valign="top"><td rowspan="1" colspan="1">Right</td><td rowspan="1" colspan="1">Wrong</td><td rowspan="1" colspan="1">4.7 (3.6-6.3)</td><td rowspan="1" colspan="1">44</td></tr></tbody></table></table-wrap> <table-wrap id="table4" position="float"><label>Table 4</label><caption><p>Correct answers by case scenario (N = 928)</p></caption><table frame="hsides" rules="groups" width="607" border="1" cellpadding="7" cellspacing="0"><col width="137" span="1"/><col width="130" span="1"/><col width="132" span="1"/><col width="152" span="1"/><thead><tr valign="top"><td rowspan="1" colspan="1"><bold>Case</bold><bold>Scenario (n)</bold></td><td rowspan="1" colspan="1"><bold>Correct Before Searching,</bold><break/><bold>No. (%)</bold></td><td rowspan="1" colspan="1"><bold>Correct</bold><bold>After Searching, No. (%)</bold></td><td rowspan="1" colspan="1"><bold>Correct After Feedback, No. (%)</bold></td></tr></thead><tbody><tr valign="top"><td rowspan="1" colspan="1">Diet (115)</td><td rowspan="1" colspan="1">38 (33.0)</td><td rowspan="1" colspan="1">72 (62.6)</td><td rowspan="1" colspan="1">79 (68.7)</td></tr><tr valign="top"><td rowspan="1" colspan="1">Hepatitis B (123)</td><td rowspan="1" colspan="1">90 (73.2)</td><td rowspan="1" colspan="1">108 (87.8)</td><td rowspan="1" colspan="1">114 (92.7)</td></tr><tr valign="top"><td rowspan="1" colspan="1">Alcohol (113)</td><td rowspan="1" colspan="1">93 (82.3)</td><td rowspan="1" colspan="1">94 (83.2)</td><td rowspan="1" colspan="1">99 (87.6)</td></tr><tr valign="top"><td rowspan="1" colspan="1">SIDS (111)</td><td rowspan="1" colspan="1">71 (64.0)</td><td rowspan="1" colspan="1">95 (85.6)</td><td rowspan="1" colspan="1">97 (87.4)</td></tr><tr valign="top"><td rowspan="1" colspan="1">Breast cancer (121)</td><td rowspan="1" colspan="1">108 (89.3)</td><td rowspan="1" colspan="1">108 (89.3)</td><td rowspan="1" colspan="1">111 (91.7)</td></tr><tr valign="top"><td rowspan="1" colspan="1">Suicide (113)</td><td rowspan="1" colspan="1">63 (55.8)</td><td rowspan="1" colspan="1">98 (86.7)</td><td rowspan="1" colspan="1">104 (92.0)</td></tr><tr valign="top"><td rowspan="1" colspan="1">Cold (111)</td><td rowspan="1" colspan="1">22 (19.8)</td><td rowspan="1" colspan="1">68 (61.3)</td><td rowspan="1" colspan="1">71 (64.0)</td></tr><tr valign="top"><td rowspan="1" colspan="1">AIDS (121)</td><td rowspan="1" colspan="1">83 (68.6)</td><td rowspan="1" colspan="1">118 (97.5)</td><td rowspan="1" colspan="1">117 (96.7)</td></tr></tbody><tbody><tr valign="top"><td rowspan="1" colspan="1">Total (928)</td><td rowspan="1" colspan="1">568 (61.2)</td><td rowspan="1" colspan="1">761 (82.0)</td><td rowspan="1" colspan="1">792 (85.3)</td></tr></tbody></table></table-wrap> <table-wrap id="table5" position="float"><label>Table 5</label><caption><p>Changes in confidence in original answer following searches (N = 905; adapted from [##REF##15800302##26##])<sup>*</sup>
                        </p></caption><table frame="hsides" rules="groups" width="541" border="1" cellpadding="7" cellspacing="0"><col width="78" span="1"/><col width="105" span="1"/><col width="100" span="1"/><col width="94" span="1"/><col width="94" span="1"/><thead><tr valign="top"><td rowspan="1" colspan="1"><bold>Change in</bold><bold>Confidence</bold></td><td rowspan="1" colspan="1"><bold>WW</bold><sup>†</sup><bold>(n = 108), No. (%)</bold></td><td rowspan="1" colspan="1"><bold>WR</bold><sup>†</sup><bold>(n = 161), No. (%)</bold></td><td rowspan="1" colspan="1"><bold>RW (n = 47), No. (%)</bold></td><td rowspan="1" colspan="1"><bold>RR (n = 589), No. (%)</bold></td></tr></thead><tbody><tr valign="top"><td rowspan="1" colspan="1">Decreased</td><td rowspan="1" colspan="1">15 (13.9)</td><td rowspan="1" colspan="1">58 (36.0)</td><td rowspan="1" colspan="1">14 (29.8)</td><td rowspan="1" colspan="1">5 (0.8)</td></tr><tr valign="top"><td rowspan="1" colspan="1">No change</td><td rowspan="1" colspan="1">37 (34.3)</td><td rowspan="1" colspan="1">16 (9.9)</td><td rowspan="1" colspan="1">14 (29.8)</td><td rowspan="1" colspan="1">165 (28.0)</td></tr><tr valign="top"><td rowspan="1" colspan="1">Increased</td><td rowspan="1" colspan="1">56 (51.9)</td><td rowspan="1" colspan="1">87 (54.0)</td><td rowspan="1" colspan="1">19 (40.4)</td><td rowspan="1" colspan="1">419 (71.1)</td></tr></tbody></table></table-wrap> <table-wrap id="table6" position="float"><label>Table 6</label><caption><p>Confidence in post-search answer for subjects who did not know answer before searching (N = 147; adapted from [##REF##15800302##26##])</p></caption><table frame="hsides" rules="groups" width="583" border="1" cellpadding="7" cellspacing="0"><col width="209" span="1"/><col width="166" span="1"/><col width="166" span="1"/><thead><tr valign="top"><td rowspan="1" colspan="1"><bold>Post-Search Confidence</bold></td><td rowspan="1" colspan="1"><bold>Wrong After Search</bold><bold>(n = 27), No. (%)</bold></td><td rowspan="1" colspan="1"><bold>Right After Search</bold><bold>(n = 120), No. (%)</bold></td></tr></thead><tbody><tr valign="top"><td rowspan="1" colspan="1">Not confident /somewhat confident</td><td rowspan="1" colspan="1">12 (44.4)</td><td rowspan="1" colspan="1">13 (1.8)</td></tr><tr valign="top"><td rowspan="1" colspan="1">Confident /very confident</td><td rowspan="1" colspan="1">15 (55.6)</td><td rowspan="1" colspan="1">107 (89.2)</td></tr></tbody></table></table-wrap> <table-wrap id="table7" position="float"><label>Table 7</label><caption><p>Comparison of confidence between pre-search and post-search right and wrong answers (N = 928)</p></caption><table frame="hsides" rules="groups" width="583" border="1" cellpadding="7" cellspacing="0"><col width="197" span="1"/><col width="106" span="1"/><col width="106" span="1"/><col width="58" span="1"/><col width="46" span="1"/><thead><tr valign="top"><td rowspan="1" colspan="1"><bold>Confidence in</bold><bold>Answer</bold></td><td rowspan="1" colspan="1"><bold>Pre-Search, No. (%)</bold></td><td rowspan="1" colspan="1"><bold>Post-Search, No. (%)</bold></td><td rowspan="1" colspan="1"><italic><bold>z</bold></italic><bold>Score</bold></td><td rowspan="1" colspan="1"><italic><bold>P</bold></italic><bold>Value</bold></td></tr></thead><tbody><tr valign="top"><td rowspan="1" colspan="1"><bold>Right answer</bold></td><td rowspan="1" colspan="1">(n= 568)</td><td rowspan="1" colspan="1">(n = 761)</td><td rowspan="1" colspan="1"/><td rowspan="1" colspan="1"/></tr><tr valign="top"><td rowspan="1" colspan="1">Not confident/ somewhat confident</td><td rowspan="1" colspan="1">208 (36.6)</td><td rowspan="1" colspan="1">34 (4.5)</td><td rowspan="1" colspan="1">14.91</td><td rowspan="1" colspan="1">&lt; .001</td></tr><tr valign="top"><td rowspan="1" colspan="1">Confident/ very confident</td><td rowspan="1" colspan="1">350 (61.6)</td><td rowspan="1" colspan="1">727 (95.5)</td><td rowspan="1" colspan="1">–15.60</td><td rowspan="1" colspan="1">&lt; .001</td></tr><tr valign="top"><td rowspan="1" colspan="1">Not provided</td><td rowspan="1" colspan="1">10 (1.8)</td><td rowspan="1" colspan="1">–</td><td rowspan="1" colspan="1">–</td><td rowspan="1" colspan="1">–</td></tr><tr valign="top"><td rowspan="1" colspan="1"><bold>Wrong answer</bold></td><td rowspan="1" colspan="1">(n= 360)</td><td rowspan="1" colspan="1">(n = 167)</td><td rowspan="1" colspan="1"/><td rowspan="1" colspan="1"/></tr><tr valign="top"><td rowspan="1" colspan="1">Not confident/ somewhat confident</td><td rowspan="1" colspan="1">154 (42.8)</td><td rowspan="1" colspan="1">30 (18.0)</td><td rowspan="1" colspan="1">6.28</td><td rowspan="1" colspan="1">&lt; .001</td></tr><tr valign="top"><td rowspan="1" colspan="1">Confident/ very confident</td><td rowspan="1" colspan="1">199 (55.3)</td><td rowspan="1" colspan="1">137 (82.0)</td><td rowspan="1" colspan="1">–6.75</td><td rowspan="1" colspan="1">&lt; .001</td></tr><tr valign="top"><td rowspan="1" colspan="1">Not provided</td><td rowspan="1" colspan="1">7 (1.9)</td><td rowspan="1" colspan="1">–</td><td rowspan="1" colspan="1">–</td><td rowspan="1" colspan="1">–</td></tr></tbody></table></table-wrap> <table-wrap id="table8" position="float"><label>Table 8</label><caption><p>Number of subjects who changed their post-search answer after feedback (N = 928)</p></caption><table frame="hsides" rules="groups" width="559" border="1" cellpadding="7" cellspacing="0"><col width="233" span="1"/><col width="130" span="1"/><col width="154" span="1"/><thead><tr valign="top"><td rowspan="1" colspan="1"><bold>Post-</bold><bold>Search Confidence</bold></td><td rowspan="1" colspan="1"><bold>Changed Answer</bold><bold>, No. (%)</bold></td><td rowspan="1" colspan="1"><bold>Did Not Change Answer</bold><bold>, No. (%)</bold></td></tr></thead><tbody><tr valign="top"><td rowspan="1" colspan="1">Not confident/ somewhat confident (n = 64)</td><td rowspan="1" colspan="1">22 (34.4)</td><td rowspan="1" colspan="1">42 (65.6)</td></tr><tr valign="top"><td rowspan="1" colspan="1">Confident/ very confident<break/>(n = 864)</td><td rowspan="1" colspan="1">51 (5.9)</td><td rowspan="1" colspan="1">813 (94.1)</td></tr></tbody></table></table-wrap> <table-wrap id="table9" position="float"><label>Table 9</label><caption><p>Changes in post-search answer before and after feedback (N = 928)</p></caption><table frame="hsides" rules="groups" width="483" border="1" cellpadding="7" cellspacing="0"><col width="161" span="1"/><col width="149" span="1"/><col width="131" span="1"/><thead><tr valign="top"><td rowspan="2" colspan="1"><bold>Before</bold><bold>Feedback</bold></td><td colspan="2" rowspan="1"><bold>After</bold><bold>Feedback</bold></td></tr><tr valign="top"><td rowspan="1" colspan="1"><bold>Right, No. (%)</bold></td><td rowspan="1" colspan="1"><bold>Wrong, No. (%)</bold></td></tr></thead><tbody><tr valign="top"><td rowspan="1" colspan="1">Right (n= 167)</td><td rowspan="1" colspan="1">122 (73.1)</td><td rowspan="1" colspan="1">45 (26.9)</td></tr><tr valign="top"><td rowspan="1" colspan="1">Wrong (n= 761)</td><td rowspan="1" colspan="1">14 (1.8)</td><td rowspan="1" colspan="1">747 (98.2)</td></tr></tbody></table></table-wrap>
<table-wrap-foot><fn id="table5fn1"><p>
                                <sup>*</sup>In 23 responses, subjects did not report a confidence rating.</p></fn><fn id="table5fn2"><p>
                                <sup>†</sup>Includes subjects who did not know the answer before searching.</p></fn></table-wrap-foot> <fn-group><fn fn-type="conflict"><p>The University of New South Wales and Enrico Coiera could benefit from the commercial exploitation of the Quick Clinical search engine or its technologies.</p></fn></fn-group>
<graphic xlink:href="jmir_v10i1e2_fig1"/> <graphic xlink:href="jmir_v10i1e2_fig2"/>
[{"label": ["10"], "surname": ["Marshall", "Williams"], "given-names": ["L", "D"], "article-title": ["Health information: does quality count for the consumer? How consumers evaluate the quality of health information materials across a variety of media"], "source": ["J Libr Inf Sci"], "year": ["2006"], "volume": ["38"], "issue": ["3"], "fpage": ["141"], "lpage": ["156"]}, {"label": ["15"], "surname": ["Anderson"], "given-names": ["JG"], "article-title": ["Consumers of e-health: patterns of use and barriers"], "source": ["Soc Sci Comput Rev"], "year": ["2004"], "volume": ["22"], "issue": ["2"], "fpage": ["242"], "lpage": ["248"], "pub-id": ["10.1177/0894439303262671"]}, {"label": ["22"], "source": ["PubMed"], "date-in-citation": ["2008 Jan 9"], "ext-link": ["http://www.pubmed.gov/"]}, {"label": ["23"], "source": ["MedlinePlus"], "date-in-citation": ["2008 Jan 9"], "ext-link": ["http://medlineplus.gov/"]}, {"label": ["24"], "source": ["HealthInsite"], "date-in-citation": ["2008 Jan 9"], "ext-link": ["http://www.healthinsite.gov.au/"]}, {"label": ["30"], "surname": ["Kahneman", "Slovic", "Tversky"], "given-names": ["D", "P", "A"], "source": ["Judgment under Uncertainty: Heuristics and Biases"], "year": ["1982"], "publisher-loc": ["New York"], "publisher-name": ["Cambridge University Press"]}, {"label": ["31"], "surname": ["Wickens", "Hollands"], "given-names": ["CD", "JG"], "source": ["Decision Making. Engineering Psychology and Human Performance. 3rd edition"], "year": ["2000"], "publisher-loc": ["Upper Saddle River, NJ"], "publisher-name": ["Prentice Hall"], "fpage": ["293"], "lpage": ["336"]}, {"label": ["33"], "surname": ["Dawes", "Sampson"], "given-names": ["M", "U"], "article-title": ["Knowledge management in clinical practice: a systematic review of information seeking behavior in physicians"], "source": ["Int J Med Inf"], "year": ["2003"], "volume": ["71"], "issue": ["1"], "fpage": ["9"], "lpage": ["15"], "pub-id": ["10.1016/S1386-5056(03)00023-6"]}]
J Med Internet Res. 2008 Jan 22; 10(1):e2