The Value of Random Audits

Random audits of research protocols originally were intended primarily to help monitor compliance with federal regulations to protect research participants. But research administrators say the audits are even more valuable as educational and quality improvement tools, especially when researchers and institutional review board (IRB) staff view the process as a two-way street.

At the Cleveland Clinic, for example, “We prefer to use the term ‘inspection’ instead of ‘audit’ and have developed a program that combines inspection and education to help us evaluate and continually improve human subjects protection,” says Daniel Beyer, MS, MHA, CIP, Executive Director, IRB and Human Research Protections.

“Over the years, we’ve learned that what’s most important is that it’s a two-way street. We are helping the researchers and their team, and they are helping us,” he adds. “Together, we have developed a culture of compliance that extends beyond a traditional auditing program.”

AAHRPP requires continuing review of research studies but allows accredited organizations to set their own policies and procedures for post-approval monitoring.

Department of Veterans Affairs (VA) facilities, for example, must audit each protocol at least once every three years. Other organizations have more flexibility. Some choose to audit studies that present the greatest risk to participants. Others target a particular area—such as conflict of interest—to gauge researchers’ understanding of changing federal regulations.

At the Mayo Clinic, the IRB and the quality management team develop an annual plan to guide their self-monitoring. The plan, which they update quarterly, takes into account a variety of factors, including perceived risk, data on non-conforming events and participant complaints, inquiries from researchers, and the information supplied to AAHRPP in status reports. Random audits then focus on the areas selected for self-monitoring.

According to Gary Cseko, M.S.P.H., M.B.A., Administrator, Research Operations, the emphasis is not on catching mistakes but on preventing them. “Everyone recognizes that this office is constantly looking for opportunities for improvement,” he says. “If we discover a problem, we find the root cause, change the procedure, and educate to the new procedure to prevent the problem in the future.”

At the University of Pennsylvania (UPenn), about 70 percent of research is biomedical. Random audits tend to target the more risky protocols and to mimic what researchers would expect from an external audit. “It’s an opportunity for improvement: a way to make sure everything is in order in the event of an external audit,” says Tracy Ziolek, M.S., C.I.P., Director, Human Research Protections.

UPenn is about to introduce random audits for nonclinical research and will start by making at least three site visits per month. “Our approach will be more supportive than audit driven,” Ms. Ziolek says. “We call it ‘verification,’ and we view it as directed education—identifying where we may have knowledge gaps and filling them in.”

At the Charlie Norwood VA Medical Center, the research compliance officer (RCO) audits at least one-third of all protocols every year, plus 100 percent of all informed consent documents. These are scanned and uploaded to the patient’s medical record and reviewed immediately.

“Everybody wants the same thing,” she continues: “to make sure it doesn’t happen again. We’re constantly checking ourselves to ensure that we’re in compliance and that our research is of the highest quality. We owe that to our veterans.”

“We can see them as soon as they’re completed, so if there’s an issue, we can pick up on it right away,” says Marie Andrews, Administrative Officer for Research and Integrated Ethics Program Officer. “I don’t think anyone other than the VA does that.”

The Charlie Norwood Center relies on metrics to track and address the most common findings. “Issues identified during audits direct the focus of the training we provide,” Research Compliance Officer Tiffye Rocke, R.N., says.  “We’re constantly checking ourselves to ensure that we’re in compliance and that our research is of the highest quality. We owe that to our veterans.”

What Organizations Find Most Valuable about the Accreditation Process

Accredited organizations consistently cite two key benefits of undergoing the accreditation process:

1. Validation of their existing efforts
2. Identification of ways to further strengthen their human research protection programs (HRPPs).   

In keeping with its commitment to quality improvement, AAHRPP routinely solicits feedback on site visits and the accreditation process. Organizations that complete the process are asked 11 questions, including what they found most valuable. By far the most common response is that accreditation lends credence to their efforts and verifies that they are doing the right thing.

“We got confirmation that what we are doing is good, correct, and in line with the regulations,” says William Schmechel, M.A., Director of Research at the University of California, Riverside. “We were told that all of our people are engaged and committed to the HRPP. Hearing that makes all of them want to continue that commitment.”

Organizations also say they appreciate the assessment of their strengths and weaknesses and AAHRPP’s recommendations on ways to improve policies and practices. Instead of resenting AAHRPP’s findings, organizations welcome the opportunities for improvement. In fact, organizations often use AAHRPP’s findings to leverage support for changes and to allocate resources more effectively.

One example is Southeast Louisiana Veterans Health Care System. According to Richard Mirabelli, M.P.H., C.I.M., Southeast’s Administrative Officer for Research, information gleaned during accreditation “allowed us to focus our resources and education where they were needed.”

Allocating Resources in Lean Times

Organizational officials are quick to acknowledge the strong connection between adequate resources and quality research programs. Even so, when money is tight, these officials have no choice but to deny requests for increases and, in some cases, to require cuts.

AAHRPP recently asked organizational officials how they decide which areas must make cuts and which will receive increases. More than two-thirds of respondents (66.6 percent) say they rely on objective data.

Several advise organizations to keep metrics on the performance of specific components of the human research protection program (HRPP) to support future funding requests. Others recommend bolstering requests for increases with objective information that shows how greater funding will meet “an identifiable need.”

Overall, AAHRPP posed 14 questions to organizational officials about the state of funding for their HRPPs. Following are some highlights:

• Ninety-three percent of respondents “strongly agree” or “agree” that “a properly resourced HRPP goes hand in hand with a high-quality research program.”
  
• Most set the bar for quality higher than the federal regulations. Fifty-seven percent say they “disagree” or “strongly disagree” with the following statement: “The HRPP should be supported only at a level that lets it conform to the regulations.”
  
• One-third were able to increase their budgets last year, and one-third kept funding levels the same as the prior year. The rest made reductions.
  
• Of those who made budget cuts, 60 percent acknowledge that the cuts “reduced the functionality” of the area or department that was affected. The remaining 40 percent say the functionality stayed the same.
  
• Slightly more than half (53.3 percent) say their HRPP has appropriate funding; the rest say they are underfunded. None of the respondents consider their HRPP to be overfunded.
  
• The vast majority (66.6 percent) rely on data to help make decisions about resource allocation, including increases and decreases to specific areas.

Organizations Choose Varying Approaches to Education

When it comes to education on human research protections, organizations tend to combine online and in-person training. Some education programs include one-on-one mentoring; others encourage attendance at conferences and seminars. But by far the most popular education resource—for IRB members, IRB staff, researchers, and organizational officials—is the Collaborative Institutional Training Initiative (CITI).

Those are among the findings when AAHRPP recently asked research organizations to describe their initial and continuing education requirements. Two-thirds of responding organizations rely to some extent on CITI modules. Some require CITI training; others offer it as one of several options.

Other key findings include:

• Nearly all respondents have formal requirements for initial and continuing education for organizational officials. Most use CITI modules or the Office for Human Research Protections module for organizational officials.
• All those involved in reviewing, conducting, or overseeing research (IRB staff, IRB members, researchers, research staff, and organizational officials) must take part in continuing education every two to three years.
• Sixty-seven percent of respondents allow internal programs to count toward fulfilling education requirements; 81 percent permit external programs to count.
• Training for IRB members and IRB staff typically includes CITI modules plus in-person education: conducting education sessions during IRB meetings, mentoring, and pairing new committee members with those who have more experience, for example.
• For researchers and research staff, respondents usually require CITI modules for initial and continuing education. Areas such as event reporting and research coordination might call for additional training.

Karen Hansen, CITI co-founder, attributes the reliance on CITI modules—especially among researchers and research staff―to the combination of convenience and controls. “CITI provides a uniform platform for offering basic education and training on regulations and ethical principles that apply to research,” Ms. Hansen says. “It also is an excellent alternative for clinicians whose schedules make it difficult to attend training lectures.”

Fred Hutchinson Cancer Research Center, where Ms. Hansen serves as Director, Institutional Review Office, mandates CITI training for new IRB staff members and offers the training as an option to new researchers and research staff.

AAHRPP interprets the responses to mean that organizations both take education seriously and appreciate the flexibility that is built into the AAHRPP standards.

“We require organizations to have an education program, but we don’t dictate what that program should include or how the content should be delivered,” says Peter Vasilenko, Ph.D., Chief Education and Evaluation Officer. “These results confirm that there are many different ways to fulfill the AAHRPP education requirement—and that each organization is best equipped to develop the program that meets its needs.”

IRB Review Times as a Quality Indicator

In 2009, AAHRPP began collecting IRB review times as part of the application process. The initial response from many was “we don’t collect that,” or “we can’t collect that,” usually reflecting the fact that their electronic systems were not set up to calculate review times. Three years later, the tide has turned, now collecting HRPP metrics is standard practice and people want even more information about review times.

AAHRPP looks at five parameters related to review time: time from submission of the protocol to review and time from submission to approval by the convened IRB, time from submission to review using the expedited procedure, time from submission to approval using the expedited procedure, and time from submission of a protocol to a determination of whether the research is exempt. AAHRPP uses calendar days to describe review time.

One question that arises is how to define the starting and stopping points for review times. Should the clock start when the protocol application is submitted initially—complete or not—or after all required information is received? At the other end of the review time are IRBs that cannot give final approval of a protocol until other committees send their approval, even though these IRBs might have completed their review and approved the protocol. AAHRPP’s definition is a simple one: from initial submission to the time the researcher receives approval to begin the research. Recognizing that review times are complex, involving different components and steps, AAHRPP encourages organizations to build into their analysis of review times all the elements that influence the work flow. Review times have at least three main components: time the IRB staff devotes to handling the application, time with the reviewer, and time for the researcher to respond to questions and requests for more information and to make changes. These times are not simple to calculate because protocols go back and forth between the IRB and the researcher.

Electronic systems can be configured to collect these component times. If any component appears problematic, the factors related to the workflow in that that component can be addressed, such as increase IRB staff, require reviewers to get reviews in on time, or encourage researchers to respond promptly.

Whether using the time from submission to approval, or conducting a more complex analysis of the workflow components, review time is an important quality indicator. Faster does not necessarily mean higher-quality reviews, but inefficient, sluggish systems are not beneficial to the research enterprise or research participants. Complete and comprehensive reviews conducted in an efficient manner are the ultimate goal.

Knowing, analyzing, and understanding your IRB review times, and acting accordingly to implement changes is the way to meet that goal. Organizations should track review times and formally analyze them to determine if, and where, improvement are needed including allocation of resources and personnel, education of IRB staff, IRB members, or researchers and research staff, or enhancement of electronic systems. And don’t just look at means or medians; analyzing outliers can produce valuable information, and are sometimes the situation on which the reputation of your IRB is based. Long IRB review time is the most common complaint of researchers. Being knowledgeable about your review times and able to justify or improve your metrics is an important part of quality improvement for your HRPP.

How Long Does It Take to Submit an Application?

That’s one of the most common questions we hear from organizations, whether they are considering accreditation for the first time or are preparing to apply for reaccreditation. To get an answer, AAHRPP recently invited accredited organizations to provide information on their experience with the self-assessment at the heart of the accreditation process.

Following are some highlights:

• Organizations applying for accreditation for the first time spent an average of 16.5 months completing the self-assessment, compared with 5.7 months for those seeking reaccreditation.
• Nearly 60% of responding organizations did not assign a staff member to pursue accreditation full-time. In most cases, accreditation was a team effort, involving two or more individuals working part-time.
• Organizations with 2,500 or more active protocols were more likely to devote full-time staff to work on accreditation.
• The size of the organization (number of active protocols) did not influence the average time needed to conduct the initial or reaccreditation self-assessments.

Accredited organizations offered the following tips:

• Start early (by far the most common recommendation).
• Prepare but don’t overdo it. Keep policies and information simple. Streamline whenever possible.
• Give yourself time to review all policies, procedures, and practices.
• Educate yourself on the AAHRPP standards and use AAHRPP as a resource.
• Develop a plan and a timeline and adhere to them.

Two organizations shared their different, successful approaches.

• From Vanderbilt University, which completed its self-assessment for reaccreditation in two to three months: “Each team of two took one domain and completed the self-assessment by spending one to two days a week at off-site retreats.”
• From Maine Medical Center, which completed its initial accreditation assessment in 18 months: “We posted the duties on sticky notes along the outside of my office. Every time a task was completed, we removed the sticky note. This simple process motivated everyone. The visual evidence of progress got everyone involved jazzed up to complete this monumental project.”

Make Changes Permanently, Not Just for AAHRPP Site Visits

The Step 2 process for accreditation calls for a site visit to evaluate HRPP function and ensure that the organization’s policies and procedures have been implemented effectively. A site visit occurs during initial accreditation, three years later at reaccreditation, and again at subsequent reaccreditations. The site visit covers extensive review of records and interviews of people ranging from IRB chairs, members, and staff to senior administrators, researchers, and research staff.

Sometimes organizations prepare interviewees for the site visits. Some educate IRB members and researchers and research staff prior to the visit. Education of anyone involved in human participants research and protection is always good. Ideally, the education provided in preparation of the site visit is part of an ongoing and comprehensive education program.

Some organizations scour the records from the list that AAHRPP provides the organization five days prior to the site visit and try to find and correct deficiencies. Altering records can be problematic. For example, IRB meeting minutes should not be revised to add missing determinations that the IRB did not make. Edits or changes can be made that accurately reflect IRB decisions and determinations, but the deficiency of not documenting the IRB’s determinations should be recognized as such and corrected in all records, not just those selected for the site visit. Changing records for the site visit is counterproductive if the deficiency in practice is not corrected permanently and comprehensively.

After the completion of the Step 1 review of application materials (review of written policies and procedures) the organization has about 90 days prior to the site visit to implement any changes identified during the Step 1 application process. Changes required as a result of the entire accreditation process (Step 1 and Step 2) should be implemented permanently upon obtaining accreditation. For example, documenting protocol-specific findings to justify certain IRB determinations should begin when the Step 1 application process is completed and certainly by the time of the site visit, not two or three years later in preparation for the next site visit. In fact, AAHRPP looks at records that span all of the years of the accreditation period to make sure that the organization has put functioning policies and procedures in place since the most recent accreditation was obtained.

This discussion leads to one key point: Make changes permanently, not just for AAHRPP site visits. The purpose of accreditation is to identify problems and implement changes so the organization can become a high-functioning, quality HRPP. The organization should perform at this high level continuously, not just for or during AAHRPP site visits. Another reason for making the changes permanent is that inspectors from the Office of Human Research Protection, the Food and Drug Administration, or the Office of Research Oversight of the Department of Veterans Affairs may visit most organizations at any time.

As Mr. Bradley Olsen, administrative officer at the Durham Veterans Affairs Medical Center, put it, his VA facility “wants to be ready for a site visit by anyone every day. We want to be able to succeed every day, not just ramp up every five years.” That is the hallmark of an excellent HRPP, and should be every organization’s goal.

Top Five Most Frequent Required Revisions to Accreditation Elements in Step 1 Review of Application Materials

After analyzing all Step 1 application reviews completed from October 2009 to June 2011 using the revised standards, AAHRPP has developed a list of the most frequently identified deficiencies found in Step 1 Review of Application Materials, which evaluates an organization’s written policies and procedures. Here are five areas to focus on when preparing your application for accreditation:

New Board Member Announcement

The Association for the Accreditation of Human Research Protection Programs, Inc., (AAHRPP) is pleased to announce the election of four new members to the Board of Directors. They are:

Karen Schweers Cook, M.A., Ph.D.
Vice-Provost for Faculty Development and Diversity
Leland Stanford Jr. University
Stanford, CA

Lynn D. Fleisher, Ph.D., J.D.
Counsel
Sidley Austin, L.L.P.
Chicago, IL

Victoria Brandy Kalabokes
President and Chief Executive Officer
National Alopecia Areata Foundation (NAAF)
San Rafael, CA

Marsha D. Rappley, M.D.
Dean, College of Human Medicine
Professor, Department of Pediatrics & Human Development
Michigan State University
East Lansing, MI

For a complete list of the Board of Directors, click here.

The Long and Short of Checklists

Checklists have become a popular mechanism for ensuring that IRBs or IRB members cover all necessary issues in review of research. Contrary to what some believe, AAHRPP does not require the use of checklists, but it does recognize checklists as one of many ways of fulfilling regulatory responsibilities. Should an organization opt to use checklists, AAHRPP requires the checklists to be correct and complete.

AAHRPP Accreditation Benefits Researchers

Researchers sometimes harbor misperceptions that AAHRPP accreditation leads to more requirements and to increased work for them. In reality, accreditation has benefited researchers by making human research protection programs more efficient, flexible, standardized, and responsive.

AAHRPP Achieves a New Milestone

International expansion and diverse organizations setting the example

Washington, D.C., June 16, 2011 – The Association for the Accreditation of Human Research Protection Programs (AAHRPP) announced today that it has awarded five organizations with Full Accreditation. Setting new milestones, in its tenth year, AAHRPP has accredited its first organization in China, first psychiatric hospital, and first federal central IRB. The list also includes two top medical colleges and teaching hospital.

Advance Newsletter Transitions Online

The Association for the Accreditation of Human Research Protection Programs (AAHRPP) newsletter, Advance, is now available in electronic form at the AAHRPP site (www.aahrpp.org). Marking the transition, this winter edition will be Advance’s final print issue.

Delivering a Better Product to Improve Protections for Research Participants

Helping organizations provide protections for research participants through accreditation is AAHRPP’s main goal. As we enter into our tenth year of accreditation, we are taking steps that directly support this goal. We are making the Advance accessible to all online. This transition will leverage the Advance’s ability to provide reliable and relevant information to a much larger community of those involved in protecting research participants and promoting ethically sound research. We hope that by closing the print editions we will open an avenue to listening, where we will be able to turn your feedback into actionable change that matters most to research participants and organizations.

From the President and CEO: Shift from Helsinki to ICH-GCP Leads to Single World Standard

The Food and Drug Administration (FDA) now allows foreign research sites and the sponsors of the studies conducted in those sites to follow good clinical practice (GCP):

“Accept as support for an IND (Investigational New Drug) or application for marketing approval a well-designed, well-conducted, non-IND foreign clinical study, if it was conducted in accordance with GCP and we are able to validate the data from the study through an onsite inspection, if necessary.”1