Who Reviews the Reviewers?

Gordon Crelinsten, MD, McGill University-Royal Victoria Hospital, National Council on Bioethics in Human Research

A few years ago, in Montreal, a research subject died during a low-risk, non-therapeutic intervention that had no potential benefit for him. The project and information disclosed in the informed consent process were reviewed and accepted by a research ethics board (REB) in an academic teaching hospital.

When this case was subjected to the scrutiny of the law, a court found that the REB was negligent in accepting the protocol as presented. Although extremely rare, death was known to occur with the proposed intervention, but disclosure of this material risk was not part of the informed consent process. The court felt that a reasonable person who is asked to participate in non-therapeutic research with no personal potential for benefit should be informed of all serious material risks, no matter how remote or rare, so that true, informed choice can occur.

Research in humans is a special privilege granted by society that carries an awesome responsibility to ensure their safety and well-being. The importance of such research is rooted in the necessity to advance knowledge.

The REB is charged with the duty of objectively evaluating research protocols with respect to research ethics guidelines, principles, and criteria.

Research is not without risk, not only to the health and safety of individuals but to the protection of confidential information; to the beliefs, customs, and social structures of collectives and communities; to the personal and professional integrity of investigators; and to the general social good.

Because of the complexity, importance, and responsibility of the task, it is appropriate to ask who reviews the reviewers?

I will explore this question with you by looking at the following areas:

· Why is this question important?

· What does a recent snapshot of Canadian research ethics review show us?

· What models exist to establish conformity of review or rules for review?

· Where do we go from here?

Why is the question important?

The research enterprise in Canada is a growing concern. The quest for knowledge, the development of skilled and inquisitive research minds, and the need to develop new techniques, new technology, and new pharmaceuticals have spawned research efforts in academic and commercial sites across this country. Each research protocol requires review, and the increasing demands on time and personnel are immense.

Not only is there a cost to ethics review, but there is a demand for expertise that cannot be assumed to be intrinsic. In itself, this expertise requires knowledge, skill, and training. REBs are comprised of individuals who are required to bring their perspective to bear on the judgment of the ethical and scientific appropriateness of human research.

After the discussion on accountability, it seems that the effectiveness, efficiency, and conformity of individual REBs are open to scrutiny.

A recent snapshot of Canadian REBs

How do REBs function in Canadian faculties of medicine? The National Council on Bioethics in Human Research (NCBHR) Working Group on Evaluation has studied the REBs affiliated with Canadian faculties of medicine. From 1990 to 1993, all 16 medical schools were visited and the structure of each REB was reviewed.

In most institutions, multiple REBs existed. In one institution, more than 30 were identified. Communication among REBs was generally poor. The sharing of experience can be important in the field of research ethics, because problems identified by one REB have been often solved or at least faced by another. The mechanisms for recording REB activity, for appealing REB decisions, for reviewing multicentre trials, and for the evaluation of merit of competing protocols varied widely within and between institutions.

The composition of REB membership and availability of education programs for investigators and REB members varied widely. Interpretations of certain aspects of research guidelines were strongly disparate. Active monitoring of ongoing research was glaringly absent.

What models exist?

What models exist to establish conformity of review or rules for review?

The NCBHR asked Louis Nicholas Fortin and Thérèse Leroux to prepare their reflections on monitoring the ethical review of medical research in Canadians.

I have briefly reviewed how informal NCBHR visits to Canadian medical faculties can help to identify local strengths and weaknesses. However, as Frances Rolleston and others have pointed out, this is weak review.

The Canadian Council on Animal Care (CCAC) has established an assessment program that is based on predetermined mandatory site visits within a formal framework of review. This system allows the development of strict, consistent criteria that can be tested and used to assess compliance.

A similar system could be devised for use in health institution accreditation. It would establish an evaluation procedure; an institution that meets the predetermined criteria or standards would be granted a certificate of attestation.

Strict and consistent criteria would be needed, and the status of a certified or accredited REB would need to be recognized by the research community, industry, and government.

Finally, there is the U.S. system, which is grounded in legislation and regulation, grants investigative powers and functions, and has the ability to impose strong and binding sanctions.

The future demands a renewed recognition of the place of ethics and ethics review in contemporary research in humans within all disciplines involved in scientific inquiry. This recognition would call for a system of ethics review which is helpful, resourceful, efficient and effective, consistent, knowledge- and value-based, accountable, and open to review and audit.

How can we achieve this goal?

REBs must realize that government, granting agencies, and the public are increasing their demands for accountability in all aspects of social endeavour, including research in humans. The need for performance measurement, outcome analysis, and standard creation will only increase. Although performance measurement is a new addition to the research lexicon, the concept of measurement is not. In fact, most research projects are based on data collection, synthesis, and measurement.

The first level of review resides within individual REBs. The REB needs to have accurate, relevant, and clearly defined data that represents its activities and can be used to measure and evaluate its performance. Each REB must routinely ask itself, "how am I doing?".

The results of this inquiry should be demanded by all participants in the exercise: government, hospital boards, industry, the public, and granting agencies.

This task may be accomplished by establishing performance indicators, a measurement tool that is used as a guide to monitor, evaluate, and improve the quality of ethics review of medical research in humans by each REB who accepts that responsibility.

For example, one indication of REB performance may be the time from submission of protocol to the development of REB recommendations, their incorporation into a revised protocol, and the acceptance of a final protocol.

Another example may be the ability to recognize promptly adverse effects of research and the ability to introduce corrective measures immediately.

If individual REBs are truly committed to the task of ensuring the safety and well-being of research subjects as well as the promotion of good science, then structured standards may be helpful to assure that these goals are met.

A standard is a statement of good practice on a specific topic, which is formulated in a way that evaluates compliance. Standards serve a variety of purposes. They can present requirements that must be met or ideals for which to aim. Standards are criteria against which current practice can be evaluated.

The Tri-Council Code of Ethical Conduct for Research Involving Humans is a series of peer-created standards which could serve as an excellent framework for the consistent internal evaluation of REB performance.

There are advantages to accepting published national standards. It is more economical to time and effort and avoids reinventing the wheel. In many cases, standards may be well-tried and recognized of value. However, if standards cannot be suitably adapted for local use, they may lack relevance and lose credibility and commitment. This may lead to failure of the whole exercise of standard setting and performance measuring.

The most successful approach to standard setting will likely involve a careful and well-considered combination of published guidance, practical experience, and local customs, practices, and ideas.

Once REBs are comfortable with themselves, scientists are open and accepting of the concept and process of peer review. REBs should understand the need for peer scrutiny of their activities and evaluate their performance against internal standards and accepted national standards.

The process of monitoring the effectiveness and efficiency of Canadian REBs could begin with formal evaluation visits. The eventual goal would be to establish a certification process, which, when granted, would attest to the REB’s compliance with predetermined criteria. These criteria would include the REB’s ability to set standards, to assess compliance, and to defend the standards, means of measurement, and any efforts to improve performance, if standards have not been met.

The question of who reviews the reviewers is important. Available models can be studied to ensure adequate review. REBs and their members are involved in serious business that demands excellence and accountability.

Discussion

Jack Corman, Institutional Review Board Services: Who would review the reviewers?

Gordon Crelinsten: This discussion refers to two sets of reviews. One set is concerned with how the performance of individual REBs is measured. By what standards? How is an REB able to establish that they are doing a good job? Secondly, what group of people should oversee the ability of REBs to set standards for themselves and measure their compliance with those standards?

Jack Corman: I would support the second group. It could be composed of stakeholders from a broad range of people who are affected by research. Is it likely that this group will be established or will the review process go in another direction?

Gordon Crelinsten: The whole question of monitoring is difficult to answer. No one organization has taken the responsibility of accepting that particular task. The NCBHR has assumed that task at medical faculties and may be prepared to be an active participant in developing a monitoring program for REBs across Canada. Whether government, granting agencies, institutions, industry, or the public will ask for representation in that particular group is open to discussion