Abstract: This article discusses the ethical issues involved in consumer health informatics -specifically those issues surrounding the provision of medical information for patients on the World Wide Web. The discussion includes concerns and suggestions relating to: quality control and error avoidance,

Key words: Bioethics, Caribbean, computing, consumer health informatics, developing countries, evaluation, Latin America, patient education, progressive caution, World Wide Web

CONSUMIDOR DE SALUD INFORMÁTICA: ÉTICA, EVALUACIÓN Y ESTÁNDARES

Resumen: Este artículo discute los temas éticos involucrados en la información en salud, específicamente aquéllos en torno a la entrega de información médica a los pacientes que ingresen a la World Wide Web. La discusión abarca preocupaciones y sugerencias en relación con el control de calidad, evitar los errores, el gobierno de Internet, la responsabilidad del sitio Web y el registro de la propiedad intelectual. Se argumenta que estos temas tienen gran importancia en el mundo en desarrollo, incluyendo a Latinoamérica y el Caribe.

Palabras clave: bioética, el Caribe, computación, información en salud, países en desarrollo, evolución, Latinoamérica, educación del paciente, cautela progresiva, World Wide Web

O CONSUMIDOR DE SAÚDE INFORMÁTICA: ÉTICA, AVALIAÇÃO E MODELOS

Resumo: Este artigo discute os temas éticos envolvidos na informação em saúde, especificamente no fornecimento de informação médica aos pacientes que ingressam na World Wide Web. A discussão envolve preocupações e sugestões em relação ao controle de qualidade, evitar os erros, o governo de Internet, a responsabilidade do site Web e o registro da propriedade intelectual. Argumenta-se sobre questões e temas de grande importância no mundo em desenvolvimento, incluindo a América Latina e o Caribe.

Palavras chave: bioética, o Caribe, computação, informação em saúde, países em desenvolvimento, evolução, América Latina, educação do paciente, cautela progressiva, World Wide Web

Introduction

Attention to ethical issues in the health professions has tended to emphasize challenges related to patient care (e.g., informed or valid consent), end-of-life controversies (withdrawing and withholding treatment), access to services (allocation of resources), special procedures (organ procurement and transplantation), and so forth. While new health technologies are traditionally a locus of ethical importance, often under these same headings, the growth of medical or health informatics has begun to capture the sustained attention of clinicians, scientists and scholars (1). Consumer health informatics is an increasingly important aspect of the computerization of health practice and communication, and it, too, requires ethical analysis and scrutiny.

Indeed, the creation, implementation, maintenance and use of information resources on the Internet constitute one of the most extraordinary developments in the history of health communication. From patient discussion groups to on-line decision support to health advice and advocacy on the World Wide Web, it is clear that digital communication is profoundly altering patient and provider access to information and decision support. This has been true for some time in many countries. The growth of the Web and its widening global availability make these issues especially important for patients, clinicians and policymakers in Latin America, the Caribbean and elsewhere.

While most debates in bioethics address direct patient care, most ethical issues related to the World Wide Web concern the control of information provided to patients and prospective patients. A defining characteristic of the Web is the lack of an overriding governing or quality management authority controlling either the content that is provided to consumers or access to that information. The Internet (a) permits very broad dissemination not otherwise possible, (b) provides more-or-less unhindered access to information (for those who can afford computers and connection charges) without the intervention of a medical professional, (c) disseminates information from sources whose credentials cannot easily be checked by consumers, and (d) can provide for consumers' anonymity so that if errors are detected and corrections made after original dissemination there is no guarantee that the consumer will, or even can, be made aware of the corrections.

What therefore makes the Web simultaneously attractive and worrisome is that anyone with access to a computer and an Internet connection can "publish" (or "make public" in an original sense of the term) anything, and most anyone with computer access can read it. While some of the resulting ethical issues are similar to those involving medical information in other media (e.g., newspapers, professional journals and texts, direct physician-patient communication), most other means of information dissemination are usually accompanied by some control over content, such as independent quality standards or peer review. Such "control" often amounted, traditionally, to no more than the difficulty encountered in obtaining the information since most printed medical journals were available only in specialized libraries or by personal subscription.

This is not the case with the World Wide Web, and we therefore will focus on ethical issues that arise when control over information is the central point of contention: quality control and error avoidance, Internet governance and Web site responsibility, and intellectual property and control. Our goal is to sketch a conceptual lattice for future analysis, risk assessment, policy development and best-practice standards, assuming that such efforts should be driven in part by ethical considerations. Our focus will be on consumer health informatics, defined as a field "devoted to the development, implementation, and research on telecommunication and computer applications designed to be used by consumers to access information on a wide variety of health care topics (2)". However, many of the same issues apply when health care professionals are the consumers. We will also address the issue of a formalized Internet presence, such as a specific Web site or an Internet support group, rather than e-mail between individuals (which others have addressed (3, 4). Similarly, we will not discuss issues of confidentiality and privacy, which are tangential to issues of quality and standards, and which require sustained treatment elsewhere. We will focus on the ethical obligations of information providers, rather than consumers.

It should be emphasized that there are a number of initiatives to develop tools and guidelines to evaluate the quality of health information on the Internet (5-7). along with commentary on these efforts (8, 9). This is important, and should be viewed as providing the data and criteria on which ethical analyses must be based. Significantly, though, a thread that runs throughout all these efforts is that current research is inadequate for complete or comprehensive evaluations of consumer health informatics tools. As early as 1997 it was noted that there is a "need to conduct systematic and rigorous research on the use of informatics tools at a time when these tools are being developed without sufficient knowledge about their effects on patient decision-making, health outcomes, and related costs (10)". There is also a need to incorporate ethical concerns into any review process. It remains to be seen whether there ought to be stricter review in the developing world--on the one hand, we do not want to stifle innovation; on the other, we perhaps ought to endorse additional protections for populations that might be regarded as vulnerable.

"Progressive Caution"

Bioethics has provided conceptually and practically useful tools for addressing issues in patient care, death and dying, access to services, special procedures and other topics. Bioethics can similarly guide decisions and actions in matters related to health informatics. Attempts to balance the risks and benefits of computer use in the health professions have confronted the following challenge: How can we enjoy the benefits of improved information technology while minimizing the risks of promotionalism, unbalanced advocacy and technology fetishism? Put differently: How does one ethically optimize practice and procedures in a context of scientific uncertainty?

One approach, which we regard as compatible with requirements both to respect persons and to maximize good consequences, has been termed "progressive caution (11)". The idea is simply that innovation should be supported and encouraged in an environment that emphasizes testing, evaluation and at least a scruple of skepticism in the face of uncritical enthusiasm. As one of the authors (KWG) has written: "Medical informatics is, happily, here to stay, but users and society have extensive responsibilities to ensure that we use our tools appropriately. This might cause us to move more deliberately or slowly than some would like. Ethically speaking, that is just too bad (11)".

The principle of progressive caution seems applicable in the context of consumer health informatics as well as other areas in which the tools of the Information Age are used in the service of the traditional healing sciences. This principle underlies the following discussion of ethical concerns related to consumer health informatics, with special regard for Latin America and the Caribbean.

Risk vs. Quality

One way to determine whether a medical tool should be used is to ask whether it will improve patient care. If the answer is affirmative, then there might be a moral requirement that the tool be used (12, 13). If the answer is that the tool increases the risk of harm to patients, then, all other things being equal, we should disdain, discourage or prohibit its use. Far more common in the case of new technologies, though, is broad uncertainty about the consequences of use.

While we should not regard patient information media as medical tools or devices per se, we might suggest that they be judged in terms similar to those for judging devices. In fact, the U.S. Food and Drug Administration (FDA), the federal agency responsible for regulating medical devices, has for a number of years been reviewing its policies on medical software (14). The current policy exempts from regulation software where there is the possibility of "competent human intervention" between the provision of advice or information and any actions that directly affect a patient. In the case of most medical software, the physician is assumed to be competent to determine the validity of the advice (15). Miller and Gardner advocate retaining the criterion of competent intervention, but also recognize that risk to patients must also be considered in determining the amount and type of regulation needed (14). On this basis they would also exempt most informational and clinical decision-support software.

It is unlikely, however, that the Internet falls into the same category as clinical decision support systems, since there is no guarantee that patients obtaining advice on the Internet are competent to evaluate it, or that a physician is an intermediary. A Web site that displays false or misleading information, for instance, can produce or contribute to bad clinical outcomes when no competent clinician, scientist, or even adequately educated patient is available to filter such content. A Web site that intentionally misleads patients (perhaps by touting a proprietary device, drug or service) is especially blameworthy. In both instances (that of sloppiness and frank deception), patients may be placed at increased risk.

Evaluation of risk and quality can be exquisitely difficult. A study of instruments used to evaluate health information on the Internet found great variability in the evaluation criteria and the reliability and validity of the rating instruments themselves (16). The authors also questioned whether it was even desirable to evaluate health information, and argued that studies of health outcomes must complement quality ratings.

Many risks are worth running, but this can be determined only against a background of information about the level of risk and the magnitude of potential benefits and harms. The very foundations of informed or valid consent for clinical practice and human subjects research are based on the idea that prospective patients and subjects must be competent, uncoerced and adequately informed about risks and potential benefits. One can run a risk, as for instance of an untoward side effect of an experimental drug, and then be harmed, as when the side effect actually occurs. But whether a wrong was committed is determined in light of the adequacy of the consent process: if information about the risk was not appropriately disclosed, the subject was wronged. It is also possible that one runs a risk that is not realized, but is wronged nonetheless because of the inadequacy of the consent process (17).

Accuracy, Timeliness and Presentation

Similar insights apply to the question of Web site accuracy and quality. While improved access to health information can (a) help patients to participate in their treatments and so can lead to improved outcomes (18), and (b) "reduce the need for some unnecessary medical services (19)", such access also risks inflaming unrealistic expectations and can, paradoxically, dissuade some patients from seeking appropriate medical attention.

Both kinds of consequences may be related to Web site quality. It remains to be determined empirically if there is a correlation between high quality/accuracy and good medical outcomes on the one hand, and low quality/accuracy and poor outcomes on the other. There is some evidence that health professions students, who presumably have more knowledge than the ordinary patient, without instruction on quality filtering are not always adept at filtering the information on websites for consumers (20). It is reasonable to insist at the very least on the following criteria for evaluating Web-based consumer health information:

* Accuracy: Does the information represent the best available research on a topic? If there is uncertainty about a particular malady, intervention, treatment, etc., is this uncertainty appropriately disclosed? Does it apply to the intended patient population?

* Timeliness: Is the information regularly reviewed and evaluated for accuracy?

* Presentation, formatting and editing: Is the information presented in such a way as to minimize misunderstanding?

Each criterion must eventually be linked to a standard, or a set of public benchmarks. For instance, what are minimal acceptable levels of accuracy? How up-to-date must information be? At what educational level should information be presented? These criteria are recommended --and standards should be developed-- for two overarching ethical reasons. First, "failing to adhere to publicly defensible standards is to risk or do harm, which, without some overriding justification or reason, is unethical (11)". Second, even in the absence of harm or the risk of harm, communicators are duty-bound to regard readers, viewers, consumers, etc. with appropriate respect and thus not deceive them, patronize them or manipulate them. Adding to our earlier discussion of risks, harms and wrongs, it should also be clear that one might commit a wrong even in a risk- and harm-free environment: deception is often blameworthy even if no other ill comes of it.

Errors of many sorts are a feature of medical practice and health communication. This does not mean that all errors are benign, or that practitioners have a diminished responsibility for error avoidance. The challenge for those who communicate health information on the Internet is that it is not always clear what should be counted as an error in the first place. Unlike clinical practice, which can accommodate reasoned disagreement about the constitution of a community standard, the Internet health community has yet to evolve an overarching philosophy and associated standards. More or less clear standards can simultaneously provide a conceptual foundation for an enterprise and be the source of productive debate and disagreement--much the same as a national constitution both lays down the law of the land and provides a structure for disagreements about that law.

Suppose, for instance, that a Web site provides dietary information and guidance for people with various endocrine disorders, and that this information is based on current but often inconclusive data. (The example is intended to be hypothetical; any similarity to an actual Web site is unintentional.) Suppose further that this is an area of ongoing research. If a patient alters his or her diet as a result of consulting the Web site, and has a bad outcome, does it make sense to say that those responsible for designing and maintaining the site erred? What if subsequent research showed that the initial recommendations were incorrect or should be superseded by more recent data? Does it matter if the Web site included a disclaimer or warning? How robust a disclaimer is needed if designers acknowledge that disclaimers are often ignored, or that despite warnings people --especially sick people-- are grasping for easy fixes and will latch onto any advice that seems to hold promise? Does a lack of access to care (in a developing country, for instance) increase the desperateness of people to acquire data? The most miserable barrio is more likely to host an Internet café than a hospital.

If a physician gives advice to a patient on Monday and then learns on Thursday of evidence that he or she was mistaken, it is usually a straightforward matter to get in touch with the patient and update, modify or recant the advice. The physician giving the advice will also have a sense of the extent to which the patient understood it. If a patient misunderstands or does not enjoy a subsequent correction of medical advice, we often (though not always) may lay blame at the physician's feet. Adequate communication is just part of sound medical practice. But no such requirement accompanies Web-based advice. Although some websites request patient email addresses, ostensibly for follow-up, there is usually no requirement to provide such contact information, nor is there any guarantee that patients would be honest in the information they provide. Because users are --indeed, perhaps should be-- anonymous, it makes little sense to suggest that they be contacted if earlier information is found to be incorrect. Because there are no agreed upon standards for measuring comprehension of medical information by laypeople on the Internet, it similarly would be unhelpful at present to suggest that users be electronically queried or quizzed about their level of understanding.

While we can imagine a Web site that includes quizzes to measure understanding, or even a "registration" procedure by which subscribers would be updated about specific advice, we cannot assume that these procedures would provide adequate protection. Such measures edge so close to what we might otherwise call the practice of medicine that it would be inappropriate to adopt them on the Internet without extensive research. That is to say, it would be inappropriate to adopt such measures while passing them off as sound or credible before soundness or credibility has been determined. A 1997 report likened the Internet to a "cocktail conversation rather than a tool for effective health care communication and decision making (21)". Although today there have certainly been improvement in the development of effective health communication websites, the quality remains uneven. For this reason, health professionals may be in a unique position to assist patients with evaluating these sites, but the health professionals themselves may need instruction about how to evaluate quality (22).

Disclosures and disclaimers

If errors are an unavoidable part of medical (or any human) practice, and if the Internet can be presumed to be similarly fallible, it is important to consider whether disclosures can constitute adequate warning to users. That is, is it ethically adequate to inform Web site visitors that (1) information might be erroneous, (2) the user cannot simultaneously enjoy anonymity and be informed of new data that bear upon an earlier query, (3) there is no substitute for competent medical attention, and even (4) that the Web site has a commercial owner or sponsor whose goals include financial or commercial benefit?

Such disclosures can in principle be quite useful. Indeed, to maintain a health Web site without some form of fair warning regarding accuracy, follow-up, standards of care, and conflicts of interest is to overlook or forgo minimal criteria for any sort of professional health communication. Yet we have no evidence to show that in electronic contexts such warnings are or could be taken seriously or have any effect; this is perhaps especially true in an environment marked by quick responses, false starts, abandoned threads. Moreover, we must attend to the problem of the "computational fallacy" or "the view that what comes out of a computer is somehow more valid, accurate, or reliable than human output (23)". As long as laypeople and professionals alike regard information available from computers and/or the Web as being inherently superior to that provided by appropriately trained or expert humans, we risk severe error. One error source rarely considered in the context of consumer health informatics is that search engines that scientists and consumers use to identify Web sites of interest or importance are themselves subject to severe limitations --because the Web is "distributed, dynamic, and rapidly growing," search engines often, if not usually, miss or fail to index some two-thirds of the "indexable Web (24)". This is perhaps especially significant in the context of calls for "participatory research" and the use by patients and families of systematic reviews to assist decision making (25).

We have been discussing the disclosure of Web site properties and shortcomings. It is important to note that a disclosure is not the same as a disclaimer. A disclosure is an uncovering or making public of certain facts; a disclaimer is a renunciation of responsibility. Some Web sites and medical software include statements that combine elements of both, though there is an emphasis on the latter, i.e., on the renunciation of responsibility and perhaps even liability. It is not clear what kind of disclaimer is adequate for legal protection. What is clear on ethical grounds is that any Web-based information system intended or likely to be used by laypeople must at a minimum include a full and frank disclosure regarding accuracy, follow-up, standards of care, and conflicts of interest. These are some of the criteria that are advocated by Health on the Net (www.hon.ch) and other organizations who have taken on the task of evaluating and certifying Web site quality.

Web Site Governance and Responsibility

The task of error avoidance falls to those who finance, develop, edit, maintain and tout Internet health sites. Disclaimers and disclosures notwithstanding, the act of Web site governance occurs in the context of an evolving, albeit inchoate, standard of care.

Perhaps the most important concern to address is that people with diseases and other maladies can be especially vulnerable to harm from on-line medical information. This means they may be more susceptible to hucksterism and promotion; more willing to take irrational risks; and more inclined to harbor false hopes.

These observations point to the two overriding ethical obligations of those responsible for Web site content, namely, the duty to adhere to, or develop, quality standards, including standards for accuracy and disclosure; and the duty to adhere to or develop standards for protecting vulnerable users. Webmasters must in general be mindful of the dangers of the computational fallacy, especially with vulnerable patients or medically underserved populations. This obligation has special applicability to children and minors.

Web sites differ from books, compact disks and other media chiefly because they create the appearance and expectation of accuracy, immediacy and timeliness. While book authors, editors and publishers, for instance, have ethical obligations to be as accurate as possible, no one can reasonably fault them for a book that has become out of date, or for a patient who seeks out the book, misinterprets it, and is harmed. Health-related Internet sites either are designed for patients or are more easily available to them than most medical libraries.

The point here is not to suggest that healthrelated Web sites are inherently dangerous, or to pass judgment on the intentions of those who create and maintain them. But good intentions are not a substitute for adherence to quality standards. The current challenge is that society is just now developing those standards. As ever, scientific progress engenders ethical controversy and conflict. The best way to meet such challenges is to identify areas of concern and propose tools for "ethical best practices." As with any new technology, the need to monitor and perhaps revise quality and bestpractice standards might be very great. This might be difficult, if not onerous, bothersome, or downright vexing; but these practical impediments are not ethical challenges.

Protection and paternalism

In arguing that minors, certain patients, some developing countries and others are, by virtue of their vulnerability, entitled to special or extra protection from risk, we invite the objection that such protections are patronizing or paternalistic. The objection might take this form: "The Internet in an open society can provide unfettered access to a variety of opinions, alternative sources of information, and diverse professional and quasiprofessional advice. It is demeaning to suggest that free people should expect any sort of special treatment or protection." It might further be suggested that the sorts of measures commended here would have a chilling effect on the creation and development of salutary World Wide Web resources.

There are a number of responses to such objections. First, it should be underscored that we are scrutinizing the dissemination to patients of medical information and advice --not poultry recipes, football statistics or travelogues. When the stakes include public health, it is reasonable to insist that ethical considerations of minimizing harm, respecting persons and adhering to sound quality criteria should guide practice. Second, professional and trade organizations have not previously been shy about regulating the actions of members, be they individual practitioners, groups or corporations; it is not clear why patient communication should not be accorded a commensurate level of attention. Third, insistence on ethical standards does not constitute a call for government regulation, legal sanctions or other measures that might in themselves warrant objections on grounds of free speech. One might analogously insist that journalists explore and adhere to ethical principles without any sort of suggestion that government should have a say in newspaper content. There is no evidence that attention to ethical "best practices" would have the purported chilling effect.

To be sure, use of any medium for commercial fraud or other kinds of deception for profit should invoke at least the consideration of social control. That the Internet is sometimes used to "promote and sell fraudulent or illegal medical products" legitimately raises the prospect of regulation (26).

Property and control

The question of who owns an electronic repository or network is among the most interesting of the Information Age. When resources are pooled from a collection of disparate sources (philanthropic, corporate, academic, governmental) it can be quite difficult to decide who owns or controls the repository. We are not here addressing the problem of intellectual property, which is well studied, but that of ownership of the means of information dissemination.

Some ownership issues are easy enough: Ownership of computers and other hardware can generally be straightforwardly assigned or apportioned. Further, there are commonly acceptable means of assigning property rights to databases, networks and other entities that straddle the conceptual divide between patentable devices and intellectual property.

A somewhat more interesting challenge arises in trying to determine who or what should control a consumer health Web site. In one case, a dispute over Web site control emerged at a university-based cancer information service. As the site gained in popularity, averaging 10,000 daily "hits" from around the world, a sequence of events led to university physician-administrators taking control of the site from a non-physician who was a cofounder (27). At dispute was the question of adequate peer review, disagreement over information quality, and, derivatively, the dissemination of health information under a university's imprimatur. One can understand an institution insisting that health-related Web sites bearing the institutional logo be under institutional control. Contrarily, most institutions would clearly exceed their authority by requiring staff members (identified as such) to seek committee approval for scholarly publications or for remarks made to the news media.

Part of the tension here is that we have yet to sort out the complexities of peer review for traditional media, such as books and journals; it is no wonder that peer review will challenge us on the World Wide Web.

The ethical challenge in such cases is shaped by considerations of governance and responsibility on the one hand, and free expression on the other. It is ultimately an empirical matter whether patients face increased risk as a result of medical Web sites either controlled by non-physicians or containing information not "approved" by physicians. Evidence of such risk would weigh in favor of some degree of medical oversight, if not collaboration. We earlier disdained government control of Web sites; what is the nature of the difference between government control and institutional or professional control? Universities traditionally do not screen faculty members' publications for accuracy or the risk of danger to lay readers or consumers. Is the Internet sufficiently different to warrant such oversight?

As elsewhere in the health professions, ethical issues here will be illuminated by more and better data. There is no reason to suppose we should resolve a problem in practical ethics --any more than a problem in clinical practice-- by a priori means. This is to argue only that health professionals and scientists must count the Internet as among the entities worthy of empirical and conceptual scrutiny. Put differently, much more research is required before we can hope for closure on an ensemble of issues at the nexus of the Internet, health and ethics.

This demand is of particular importance in Latin America, the Caribbean and the rest of the developing world. As countries leap forward in telecommunications capacity, the need to assess the effect of these changes acquires a moral urgency, lest populations accustomed to traditional forms of exploitation be exploited in subtle new ways.

Conclusion and Recommendations

It is no small challenge for clinicians to synthesize and act on the latest evidence-based data. It stands to reason that patients also need some help in trying to make sense of a bewildering stew of information.

Communicating health information is rarely a straightforward matter. Physicians and nurses who talk with their patients about something as pedestrian as this morning's newspaper article on diet or cancer risks know that ignorance, hope and fear shape behavior and beliefs as often as evidence, reason and logic.

The World Wide Web has extraordinary promise as a source of patient information and advice. But like other new health technologies, we do patients no favors by touting an untested tool.

For the Web to realize its potential:

1. It must be subjected to increasingly rigorous tests of accuracy, timeliness and readability;

2. Web sites must eliminate, or minimize and disclose, actual and potential conflicts of interest;

3. Entities that sponsor or own Web sites must recognize their responsibility to attend simultaneously to quality improvement and free expression

Medical Web site sponsors and owners who are motivated mainly by profit, public relations or prestige should be regarded with the same disdain we customarily hold for health professionals who place these values ahead of patient care. It is not that fair compensation, positive community regard and respect and admiration are inappropriate for physicians or Web site owners but, rather, that we ethically optimize practice by a more appropriate ordering of values.

The concept of progressive caution seems to capture intuitions about how best to balance the risks and benefits of consumer health informatics. It remains for scientists and scholars to make clearer how to enjoy the fruits of progress while taking care not to succumb to the hucksters, boosters, bells, and whistles that experience has taught can come between people and high quality care. Failure to use the best of health information technology will be a loss to the people of Latin America and the Caribbean. Failure to use this technology responsibly will allow the entire enterprise to be seen by some populations as a cruel trick.

Received: 13-06-2005/Accepted: 27-06-2005

References

(1.) Goodman KW, (ed.) Ethics, Computing and Medicine: Informatics and the Transformation of Health Care. Cambridge: Cambridge University Press; 1998.

(2.) Jimison HB, Sher PP. Decision support for patients. In: Berner, ES, (ed.) Clinical Decision Support Systems: Theory and Practice. New York: Springer-Verlag; 1999: 139-166.

(3.) Kane B, Sands DZ. Guidelines for the clinical use of electronic mail with patients. Journal of the American Medical Informatics Association 1998;5:104-111.

(4.) Delbanco T, Sands DZ. Electrons in flight-e-mail between doctors and patients. New England Journal of Medicine 2004; 350(17):1705-07.

(5.) Ademiluyi G, Ree C, Sheard C. Evaluating the reliability and validity of three tools to assess the quality of health information on the Internet. Patient Education and Counseling 2003; 50(2):151-55.

(6.) Gagliardi A, Jadad A. Examination of instruments used to rate quality of health information on the internet: chronicle of a voyage with an unclear destination. British Medical Journal 2002; 324:569-73.

(7.) Wilson P, Risk A. How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the internet. Commentary: On the way to quality. British Medical Journal 2002; 324: 598-602.

(8.) Purcell G, Wilson P, Delamothe T. The quality of health information on the internet. British Medical Journal 2002; 324: 557-558.

(9.) Ferguson T. From patients to end users. British Medical Journal 2002; 324:555-556.

(10.) Agency for Health Care Policy and Research. Consumer Health Informatics and Patient Decision Making. AHCPR Research Report No. 98-N001, September 1997. (http://www.ahcpr.gov/research/rtisumm.htm Accessed 11 June 2005.)

(11.) Goodman KW. Bioethics and health informatics: an introduction. In: Goodman KW, (ed.) Ethics, Computing and Medicine: Informatics and the Transformation of Health Care. Cambridge: Cambridge University Press; 1998: 1-31.

(12.) Miller RA, Schaffner KF, Meisel A. Ethical and legal issues related to the use of computer programs in clinical medicine. Annals of Internal Medicine 1985; 102:529-36.

(13.) Forsström J, Rigby M. Considerations on the quality of medical software and information services. International Journal of Medical Informatics 1999; 56 (1-3):169-76.

(14.) Miller RA, Gardner RM. Summary recommendations for responsible monitoring and regulation of clinical software systems. Annals of Internal Medicine 1997; 127:842-45.

(15.) Young FE. Validation of medical software: present policy of the Food and Drug Administration. Annals of Internal Medicine 1987; 106:628-29.

(16.) Jadad AR, Gagliardi A. Rating health information on the Internet. Navigating to knowledge or to Babel? Journal of the American Medical Association 1998; 279:611-14.

(17.) Capron AM. Ethical issues in protecting research subjects. Journal of Clinical Epidemiology 1991; 44:81S-89S.

(18.) Jimison HB, Sher PP. Consumer health informatics: Health information technology for consumers. Journal of the American Society for Information Science 1995; 10:783-90.

(19.) United States General Accounting Office. Consumer Health Informatics: Emerging Issues. Washington, D.C.: United States General Accounting Office, 1996 (report number GAO/IMD-96-86).

(20.) Berner ES, McGowan JJ, Hardin JM, Spooner SA, Raszka WV, Berkow RL. A model for assessing information and retrieval skills of medical students. Academic Medicine 2002; 77:547-51.

(21.) Silberg W, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet. Journal of the American Medical Association 1997; 277:1244-45.

(22.) McGowan JJ, Berner ES. Proposed curricular objectives to teach physicians competence in using the world wide web. Academic Medicine 2004; 79:236-40.

(23.) Goodman KW. Ethical and legal issues in use of decision support systems. In: E. Berner, (ed.) Decision Support Systems in Theory and Practice. New York: Springer Verlag; 1999:217-33.

(24.) Lawrence S, Giles CL. Searching the World Wide Web. Science 1998; 280:98-100.

(25.) Bero LA, Jadad AR. How consumers and policymakers can use systematic reviews for decision making. Annals of Internal Medicine 1997; 127:37-42.

(26.) Skolnick AA. WHO considers regulating ads, sale of medical products on Internet. Journal of the American Medical Association 1997; 278:1723-24.

(27.) Hoke F. Struggle over online cancer service spurs larger medical ethics debate. The Scientist 1995; 9(7), April 3:1-7.

Eta S. Berner *, Amarinder Singh Sandhu * and Kenneth W. Goodman **

* Department of Health Services Administration. School of Health Related Professions. University of Alabama at Birmingham

** Director, Bioethics Program University of Miami

Correspondence: kgoodman@miami.edu