Consider your future practice as a counselor, keeping in mind a population and/or setting for your work. The Final Course Project is a synthesis of literature and it is aimed to help you determine the
A Pilot Study Examining Trainee TreatmentSession Fidelity when Parent–Child
Interaction Therapy (PCIT) Is Implemented
in Community Settings
Jamie K. Travis, MS
Elizabeth Brestan-Knight, PhD
Abstract
Research supports the impact of empirically based treatments, such as Parent–Child Interaction
Therapy (PCIT), on producing positive treatment outcomes for clients. However, achieving
outcomes in community settings that are similar to those found in research settings can be
challenging, and little research has been conducted on how to best train community providers to
implement PCIT withfidelity. This study assessed trainee implementationfidelity to the PCIT
protocol in community settings. Sessionfidelity was reviewed for trainees using pre-established
session integrity checklists and post hoc video review of key sessions. Analyses revealed that
trainees maintained strong overall sessionfidelity, althoughfidelity percentages varied by session
type and treatment phase. Results also highlight those session content items that are frequently left
out by trainees during implementation. Implications of thesefindings, limitations, and future
directions for research and PCIT workshops and consultation are discussed.
Introduction
In a 2005 policy statement, the American Psychological Association (APA) endorsed the use of
evidence-based practice in psychology (EBPP). 1One of the main facets involved in implementing
EBPP concerns a therapist’s ability to effectively use interventions that show therapeutic change in
controlled trials, known as evidence-based treatments (EBTs), with his or her clients.
2The President’s
New Freedom Commission on Mental Health (2003) Subcommittee on Children and Families, which
analyzed the strengths and weaknesses of the current mental health service system, described the
importance of informing children and families about EBPP and providing families access to EBTs,
which have been found to be more effective than usual care.
3–6
Address correspondence to Jamie K. Travis, MS, Auburn University, 226 Thach Hall, Auburn, AL 36849, USA. Phone:
+1-334-8444889; Fax: +1-334-8446671; Email: [email protected].
Elizabeth Brestan-Knight, PhD, Auburn University, 226 Thach Hall, Auburn, AL, USA. Phone: +1-334-8446486;
Fax: +1-334-8444447; Email: [email protected]
Journal of Behavioral Health Services & Research, 2013. 342–354. c
) 2013 National Council for Community Behavioral
Healthcare. DOI 10.1007/s11414-013-9326-2
342The Journal of Behavioral Health Services & Research 40:3 July 2013 The reported efficacy of EBTs and the call to action from the APA, as well as the President’s
New Freedom Commission Subcommittee on Children and Families, has created a demand for
services that are based on science and best clinical practice, such as EBTs. This demand has led to
EBPP dissemination into community settings, such as community mental health agencies and
general medical settings, where real-world providers, rather than research therapists, provide
services.
7However, one of the challenges with disseminating EBTs into community settings is that
these services are primarily developed, researched, and implemented in highly controlled settings,
such as universities, making it difficult to determine the treatment’s ecological validity and
applicability to community-based clients, who are generally more diverse and have multiple
presenting problems and co-occurring conditions.
8,9 Therefore, it is important to investigate the
transportability of EBTs into community settings by training community-based mental health
providers in implementing EBTs withfidelity.
Evidence-based parent training programs are becoming the standard treatment for childhood
conduct problems, especially in young children.
10,11 Research shows that parents of children with
conduct problems lack, or infrequently and inconsistently use, fundamental parenting skills and a
variety of behaviorally based parent-training programs focus on teaching parents to effectively
manage child behavior.
12 Parent-training treatment programs have been found to be efficacious for
addressing conduct problems and disruptive behavior in young children, producing significant
improvements in children’s behaviors, significant change in parental behavior, and reducing
negative outcomes for these children during adolescence and adulthood.
10,13–16
Parent–Child Interaction Therapy (PCIT) is one evidence-based parent-training program used to
treat young children with conduct problems and the focus of this study. 17,18 PCIT is a manualized
EBT that aims to improve the parent–child relationship and correct maladaptive parent and child
behavior patterns. To accomplish this goal, parents arefirst taught relationship enhancement skills
during the Child-Directed Interaction (CDI) phase and are then taught effective discipline skills to
use with their child during the Parent-Directed Interaction (PDI) phase.
Like other parent-training programs, PCIT utilizes a didactic format, called“teach”sessions, in
order to teach parents the basic parenting skills used in treatment, and has a strong basis in
behavioral principles and developmental psychology. The teach sessions are primarily structured as
psychoeducation sessions with the therapist teaching the parents new skills, providing a rationale
for their use, and helping parents to problem-solve when and how to use these skills. Parents meet
with the therapist to complete these didactic teach sessions, once at the beginning of the CDI phase
in order to learn relationship enhancement skills, and then again at the beginning of the PDI phase
in order to learn how to give effective commands, and implement an effective discipline strategy.
There are a few key differences between PCIT and other parent-training programs.
19In PCIT, the
parenting skills originally learned by the parents during the teach session are then expanded upon
during the following sessions, called“coach”sessions, during which in vivo instruction is used by
the therapist to coach the parents in real time, by providing them with immediate feedback on the
use of their skills. To do this, parents take turns playing with their child in a therapy room while the
therapist observes the parent–child interaction from behind a one-way mirror and talks to the parent
about his or her use of the parenting skills as the interaction occurs using a bug-in-the-ear device.
This skills coaching and immediate feedback had been shown to improve parents’skill acquisition
and to be an important component of effective parent-training programs.
20,21 Furthermore, PCIT is
structured so that treatment is performance-based rather than time-limited. This means that parents
are assessed on their use of the parenting skills with their child during their weekly coach sessions,
and are required to demonstrate a certain level of skill mastery before they are able to progress
from CDI to PDI, or graduate from treatment. The therapist also uses the coach sessions to check in
with parents about their homework completion and any changes in child behavior throughout
treatment in order to track progress. The unique structure of PCIT, including in vivo coaching and
assessment-based skill mastery, has implications for training community mental health providers in
PCIT Implemented in Community Settings TRAVIS, BRESTAN-KNIGHT343 PCIT. In particular, PCIT may require a longer training time and more interactive instructional
methods for providers to learn relative to more traditional therapy formats.
Similar to many other EBTs, PCIT has historically been disseminated in university settings
following a mentor–mentee model. However, in line with the call to disseminate EBTs into
community settings, PCIT has been disseminated to providers in the community. In order to
facilitate the dissemination of PCIT to community settings, PCIT Training Guidelines were created
by PCIT International in 2009.
22 These guidelines present the key components that should be
included in all PCIT trainings; however, little research has been conducted on how to best train
PCIT therapists, and the few studies evaluating PCIT dissemination to date have not examined
traineefidelity to the PCIT protocol.
23–25
Current research has shown that simply providing community mental health providers with access to
training in EBTs is not sufficient to ensure that these programs effectively benefitconsumers.
Furthermore, passively relying on providers to read treatment manuals and attend training workshops
does not lead to the successful implementation of EBTs.
23,26 In order for families to benefit from EBTs,
community providers need to not only be informed about and trained in EBTs, but they need to evaluate
the manner in which they implement EBTs with their clients.
27Therefore, the implementation process
needs to be active and should include extended training formats, interactive training techniques
including modeling, role-play, case discussion, and coaching, and the effective monitoring of staff
performance regarding implementationfidelity.
28,29 Research demonstrates that if EBTs are
implemented inconsistently, and EBT implementation is not monitored to assurefidelity, program
effectiveness may be compromised.
30,31 Consequently, in order to increase the effectiveness of EBTs
when used in community settings, and to eventually predict stronger treatment outcomes for
consumers, maintaining highfidelity to treatment protocols is of the utmost importance.
26,32,33
There are a variety of methods available to measure treatmentfidelity. When possible, it is
best to usefidelity assessment instruments that are designed by the program developers to
provide feedback to the program implementers.
30 When choosing an appropriate measure of
treatmentfidelity, it is also important to remember that the measure mustfirst and foremost
be both ecologically valid and feasible; otherwise, taking the time to assess treatmentfidelity
may lose its value or become overly burdensome.
34 Current assessments of treatmentfidelity
in PCIT often examine how closely providers adhere to the PCIT protocol based on PCIT
integrity checklists.
35
In randomized controlled trials (RCTs) of PCIT, adherence levels of 97 % have been
reported. 36,37 However, in a recent study addressing PCIT effectiveness in a community
setting, treatment adherence to the PCIT protocol was 91 %. 38 This difference in adherence
level suggests that implementing PCIT withfidelity may be more difficult in community
settings than it is in the controlled-study environments. Additionally, the providers
implementing the PCIT protocol in these studies learned PCIT through the traditional
mentor–mentee model of EBT training. This is an important distinction because adherence
levels may be impacted when community providers, who are typically trained during a one-
week workshop in accordance with the PCIT Training Guidelines, implement the PCIT
protocol. It is also important to note that in all of these studies, assessing treatmentfidelity in
the implementation of PCIT was a secondary goal to determining treatment outcomes. This
emphasizes a significant gap in the literature related to assessing PCIT treatmentfidelity as a
means of informing future trainings of community providers in PCIT.
Study goals
The current pilot study aims to assess trainee treatment sessionfidelity to the PCIT protocol for
community providers who were trained following the training guidelines recently established by
PCIT International. Due to the unique and complex nature of the PCIT coach sessions, it was
344The Journal of Behavioral Health Services & Research 40:3 July 2013 hypothesized that trainees would obtain higherfidelity scores on teach sessions than on coach
sessions. It was also hypothesized that trainees would obtain higherfidelity scores on CDI sessions
than on PDI sessions because CDI sessions were conducted closer in time to the trainees’
completion of the 40-h PCIT training workshop.
Methods
PCIT training procedure
At individual or agency request, trainees attended a live, 40-h face-to-face PCIT training
workshop presented by a lead trainer, who was vetted by PCIT International as a PCIT Master
Trainer and four doctoral-level clinical psychology graduate students who served as co-trainers.
The structure of all training workshops followed the PCIT Training Guidelines (2009) and included
didactic presentations, modeling, and the role-playing of skills.
The curriculum for the 40-h training workshop included didactic presentations, which provided
trainees with information on the history and theoretical framework of PCIT, the empirical research
related to PCIT, and the material that is typically taught to parents during the different phases of
PCIT including the specific relationship enhancement skills employed during the CDI phase, the
effective discipline procedure utilized during the PDI phase, and the generalization of these skills
to the home environment and public places. Trainees were also taught how to administer, score,
and interpret the assessment measures used during PCIT in order to monitor client treatment
outcomes. Role-plays of parent–child interactions were interspersed with didactic lectures to
provide trainees with the opportunity to demonstrate their ability to code parent behaviors using the
Dyadic Parent–Child Interaction Coding System (DPICS), 3rd Edition.
39These role-played parent–
child observations also allowed the trainees to demonstrate mastery of the CDI and PDI skills as
outlined by the PCIT Training Guidelines. Trainees also had the opportunity to observe the training
team conduct selected PCIT sessions with a demonstration family. Specifically, the trainees were
able to observe the trainers implement the CDI Teach, CDI Coach 1, PDI Teach, and PDI Coach 1
sessions conducted with volunteer families from the community, while following along with the
integrity checklists from the PCIT treatment protocol.
Following the completion of the 40-h training workshop, trainees returned to their agencies and
began implementing PCIT with clients between the ages of 3 and 7 years who were seeking treatment
for conduct problems or disruptive behavior. As stipulated in the 2009 PCIT Training Guidelines,
trainees then participated in follow-up consultation until they completed the PCIT protocol with two
families. Follow-up consultation included group consultation calls, post hoc video reviews of trainee
treatment and assessment sessions, and attendance at a 2-day advanced training workshop. The
frequency of consultation calls ranged from weekly to monthly and allowed trainees the opportunity to
provide updates on their ongoing cases, trouble-shoot any perceived difficulties in their caseload, and
discuss the appropriateness of potential new cases. Video reviews included the submission of a total of
seven sessions from the PCIT protocol including the baseline DPICS observation assessment session
and six key treatment sessions (CDI Teach, CDI Coach 1, PDI Teach, PDI Coach 1, House Rules, and
Public Behavior). Trainees based in the United States (71.4 %) also attended an advanced training
workshop, which occurred 5 to 7 months following the trainees’participation in the initial PCIT
training workshop. These advanced trainings provided trainees with the opportunity to refine their
coaching skills, practice DPICS coding, and conduct co-therapy with the PCIT Master Trainer.
Participants
University IRB approval was obtained for this study. Training data were collected during the
PCIT training process as described above, entered into an archival database, and de-identified.
PCIT Implemented in Community Settings TRAVIS, BRESTAN-KNIGHT345 Thus, participants in this study, the PCIT trainees, were mental health professionals from
various agencies who took part in the PCIT training process. A total of 31 trainees attended
training workshops across four training cohorts, consisting of threetraining cohorts from
various urban and rural regions of the United States and one cohort from an urban area in
Asia. However, not all of the trainees were able to continue PCIT training following the
initial training workshop. In fact, 10 (32.3 %) of the 31 trainees who attended the training
workshop were unable to participate in the follow-up consultation phase of training.
Therefore, the participants included in this study represent only 21 trainees who participated
in the follow-up consultation phase of PCIT training, as indicatedby the submission of at
least one video-recorded treatment session for post hoc review.
Of the 21 trainees who participated in the PCIT consultation phase, the mean age was 38.9 years
(SD09.3). On average, trainees worked 42.2 h per week (SD06.7) with an average of 18.1 h per
week spent in direct contact with clients (SD09.8). Trainees reported having worked with children
and families an average of 12.8 years (SD08.2), with a range from 3 to 31 years. The average
number of parent training courses completed by trainees prior to the PCIT training workshop was 1
(SD01.9), with a range from 0 to 8. Of note, participating trainees did not differ significantly from
those trainees who did not participate in the follow-up consultation portion of training on any of
the above variables. See Table1for information about the trainees’gender, race/ethnicity, level of
education, and place of employment.
Measures
Analyses for this study were conducted using archival data from the treatment session videos
submitted for review by the participating sampleof 21 trainees. As previously mentioned, six key
treatment sessions from the PCIT protocol were video-recorded by trainees participating in the follow-up
consultation phase of training and then sent to the training team for review. The sessions sent for review
Table 1
Participant information
Number Percentage
Gender
Female 20 95.2
Male 1 4.8
Race
Caucasian/White 15 71.0
Asian 6 29.0
Education
Bachelor’s 2 9.5
Master’s 6 28.6.
MSW 8 38.1
Doctorate 4 19.0
Missing 1 4.8
Place of employment
General medical center 9 42.9
For-profit CMHC 3 14.2
Non-profit CMHC 9 42.9
CMHCCommunity Mental Health Clinic
346The Journal of Behavioral Health Services & Research 40:3 July 2013 included the CDI Teach, CDI Coach 1, PDI Teach, PDI Coach 1, House Rules, and Public Behavior
sessions. Sessionfidelity was reviewed for trainees using integrity checklists and post hoc video review.
Fidelity was measured using the pre-established session integrity checklists provided by the
program developer in the PCIT protocol for the CDI Teach, CDI Coach 1, PDI Teach, and PDI
Coach 1 sessions.
29,38 The integrity checklists were utilized to determine whether the trainee
completed all parts of the session that are described in the PCIT protocol. Each checklist content
item was coded such that 1 indicated the presence of a checklist content item and 0 indicated the
absence of a checklist content item. Therefore, if a trainee discussed a particular content item with
the family during the treatment session it was coded as one, but if the trainee forgot to discuss a
particular content item during the session it was coded as zero.
The number of content items and specific content of the checklists vary based on the
treatment session being evaluated. For example, the integrity checklists for the teach sessions
contain more content items, with 29 in theCDI teach session and 44 in the PDI Teach
session, compared with the coach sessions, with 10 and 11 content items for CDI and PDI,
respectively. The content of the teach integrity checklists focus on discussing the structure of
PCIT and the structure of weekly sessions, teaching parents specific parenting skills,
reviewing and role-playing skills with parents, and assigning and planning for the completion
of homework. Meanwhile, the content of the coach integrity checklists involve discussing
parental concerns, reviewing and assigning homework, orienting the child to the structure of
the coach session, coding and coaching parents, and providing parents with feedback on their
progress toward mastery and their ratings of their child’s behavior. Due to the variation in
the number of content items included on each integrity checklist, each sessionfidelity score
was calculated as a continuous variable by dividing the number of completed items on the
integrity checklist by the total number of items listed on the checklist.
The 2009 Training Guidelines indicate that trainee implementation of House Rules and Public
Behavior (two components discussed towards the end of the PCIT protocol to help generalize
treatment outcomes to other settings) can be evaluated during a role-play or demonstrated during
actual client sessions. For this study, the House Rules and Public Behavior components were
observed from video-recorded sessions with families in treatment (i.e., not role plays) and were
approximately 15 min in length, in contrast to the CDI and PDI teach and coach sessions, which
were typically 50 min in length. Because the specific House Rules and Public Behavior segments
were reviewed rather than the entire PDI coach sessions that included these segments, we
developed integrity checklists that were more detailed than the session outlines included in the
PCIT protocol for these segments.
38
Proceduralfidelity was also examined for the various content items within each treatment session, in
order to determine areas to improve during future training workshops, and to examine trends infidelity
based on individual session criterion. Averages were calculated for each content item on the various
integrity checklists in order to determine what percentage of trainees covered the required content items
during the recorded treatment session. Session content items that were not discussed by at least 80 %
of trainees during a given session were categorized as having lowfidelity.
Data preparation and analysis
Trainees submitted a total of 114 video-recordedtreatment sessions, or work samples, for post hoc
review. However, of these 114 work samples, sevencould not be reviewed due to errors that occurred
during the video-recording process leaving a total of 107 work samples eligible for review. Of note, some
trainees submitted more than one work sample to be reviewed for each of the key sessions. In order to
best assess the trainees’ability to maintainfidelity to the treatment protocol and to maintain independence
of data, only thefirst work sample of the key treatment sessionssubmitted by each trainee was considered
for this study. This reduced the total number of videos eligible for review to 71 (see Table2).
PCIT Implemented in Community Settings TRAVIS, BRESTAN-KNIGHT347 The 71 work samples that were submitted by trainees for post hoc review were reviewed by
PCIT-trained graduate students and then checked and given afidelity score by the PCIT Master
Trainer. Additionally, to evaluate the inter-rater reliability (IRR) of the treatment session integrity
checklists, a trained undergraduate research assistant reviewed slightly over 50 % (n038) of the
trainee work samples. Items were coded such that 1 indicated agreement on a checklist content
item and 0 indicated disagreement on a checklist content item. Percent agreement was calculated
for the 38 work samples evaluated for IRR.
Results
Preliminary analyses
Regarding IRR across allfirst work samples, the overall mean percent agreement across the six
sessions was 92.9 (SD09.3).
Descriptive statistics
Descriptive analyses were conducted in order to determine overall sessionfidelity by calculating the
averagefidelity percentage for all six submitted work samples including the teach sessions, coach
sessions, and generalization sessions (house rules and public behavior segments). The averagefidelity
percentage across all six sessions was 87.2 (SD016.8, Range037.5 to 100). Averagefidelity
percentages were also calculated for each submitted session individually (see Table2).
In order to evaluate any potential differences in implementation for the two session types (teach
versus coach), a paired samplesttest was conducted for those trainees that submitted at least one session
video for the two session types and treatment phases (n012). The averagefidelity score for teach
sessions was 91.3 (SD09.7), and the averagefidelity score for coach sessions was 79.9 (SD017.8);
(t(11)03.17,p00.009); (d01.91). Furthermore, a Pearson product–moment correlation coefficient
was computed to assess the relationship between the length of treatment sessions andfidelity scores.
Results showed that there was no correlation between teach sessionfidelity scores and the length of the
teach sessions (r00.096,n024,p00.327), or between coach sessionfidelity scores and the length of
the coach sessions (r0 0.134,n022,p00.275).
A paired samplesttests was conducted for those trainees that submitted at least one session
video for the two session types and treatment phases (n012) to evaluate any potential differences
in implementation for the two treatment phases (CDI versus PDI). The averagefidelity score for
CDI sessions was 85.8 (SD016.0), and the average
fidelity score for PDI sessions was 87.3
(SD012.2); (t(11)0 0.340,p00.740); (d00.21). There was no correlation between CDI session
Table 2
Overall sessionfidelity by session
Session Number Meanfidelity score Standard deviation
CDI teach 16 90.52 13.12
CDI coach 1 15 80.17 20.01
PDI teach 13 92.31 9.49
PDI coach 1 11 82.15 17.94
House rules 9 91.11 17.64
Public behavior 7 87.76 22.48
Total 71 87.16 16.77
CDIchild-directed interaction,PDIparent-directed interaction
348The Journal of Behavioral Health Services & Research 40:3 July 2013 fidelity scores and the length of the CDI sessions (r00.326,n023,p00.064) using a Pearson product–
moment correlation, but that there was a significant negative correlation between PDI sessionfidelity
scores and the length of the PDI sessions (r0 0.417,n023,p00.024).
Regarding within session proceduralfidelity, the average percent of trainees who covered
particular session content items ranged from 43.6 % to 100 %. Four of the 29 (13.8 %) content
items on the CDI Teach integrity checklist demonstrated lowfidelity while 11 of the 44 (25 %)
content items on the PDI Teach integrity checklist demonstrated lowfidelity. Additionally, three of
the eight (37.5 %) content items on the CDI Coach 1 integrity checklist demonstrated lowfidelity
while four of the ten (40 %) content items on the PDI Coach 1 integrity checklist demonstrated low
fidelity. Finally, none of thefive content items on the House Rules integrity checklist demonstrated
lowfidelity while two of the seven (28.6 %) content items on the Public Behavior integrity
checklist demonstrated lowfidelity.
Discussion
Findings in this study indicate that community-based providers were able to maintain strong
overall treatmentfidelity, with their average treatmentfidelity score reaching 87.2 % andfidelity
scores across individual sessions remaining consistently above 80 %. Thisfinding is slightly lower
than the treatment adherence rates found in RCTs of PCIT, but relatively consistent with the
treatment adherence rate of 91 % that was reported by Lyon and Budd in their examination of the
effectiveness of PCIT when delivered in community settings.
36–38 However, the current study
differs from RCTs assessing the efficacy of PCIT in controlled settings and the Lyon and Budd 38
study in an important way. As previously noted, the providers in the current study were community
providers who have been trained in PCIT following the PCIT Training Guidelines, while the
providers in the Lyon and Budd study (2010) and previous RCTs were graduate students and
licensed clinical psychologists who had been trained in PCIT using the traditional mentor–mentee
model. Therefore, the current study represents thefirst examination of treatmentfidelity when off-
site, community-based providers implement the PCIT protocol in community settings. The overall
fidelity scores found in this pilot study demonstrate that the trainees in our sample implemented the
PCIT protocol with a high degree offidelity. Thesefindings are especially encouraging considering
that the trainees in this study work with populations that are more heterogeneous than the
populations represented in RCTs and in settings that are less controlled. Specifically, therapists in
community-based settings often need to see a number of clients at a relatively quick pace, often
without the benefit of a co-therapist.
Although the sample size in this pilot study was limited, as suggested previously,findings from
this study also indicate that the complex nature of the PCIT protocol may hinder community
providers from implementing PCIT in community settings with the same level offidelity as
providers who implement PCIT within the context of an RCT, or have been trained following the
traditional mentor–mentee model. Thesefindings suggest that the current PCIT Training
Guidelines, which include a 40-h face-to-face workshop and extended follow-up consultation,
are effective in teaching trainees how to implement PCIT withfidelity, but also indicate that the
PCIT Training Guidelines may benefit from some specific improvements. While workshop and
consultation content most likely varies across trainers and training sites, this study provides a
starting point for generating some ideas of how to best structure the PCIT training process and
what should be included during the workshop and consultation phase.
Looking more specifically at traineefidelity across session type, results support the initial
hypothesis, indicating that the trainees maintained higherfidelity to the treatment protocol during
teach sessions relative to coach sessions. One explanation for thisfinding is that the more
straightforward nature of the teach sessions make them easier to implement. Teach sessions are
more clear-cut than coach sessions because the child is not present, no technology is required, and
PCIT Implemented in Community Settings TRAVIS, BRESTAN-KNIGHT349 the provider has a detailed outline that designates the material that he or she needs to cover with the
parent. Additionally, the format of the teach sessions resemble traditional psychoeducation, which
the majority of providers have probably utilized with previous clients. On the other hand, coach
sessions are more dynamic in nature because they include both the parent and the child in session
and require the provider to code client behavior using the DPICS while simultaneously coaching
parents on the use of the PCIT skills using audio and video technology.
Given that the sample size was small in this study, the difference between trainees’ability to
implement the teach and coach sessions cannot be generalized to other samples. However, the
results of this study do have some implications for PCIT trainers to consider given the dearth of
information available regarding the implementation of PCIT in community settings. The
discrepancy betweenfidelity scores for the PCIT coach and teach sessions suggests that trainers
may need to spend more time during the training workshop preparing trainees to manage the
dynamics of a coaching session including transitioning the parent from check-in and review of
homework to the coaching component and allowing sufficient time to review the parents’skills and
assign homework before ending the session. These results also suggest that co-therapists may be an
important component for training cases, as it is easier to handle the more complex dynamics of a
coaching session when two therapists are present.
Contrary to our hypothesis, there was no statistical difference betweenfidelity scores for the two
treatment phases. We predicted that trainees would have higherfidelity scores on CDI sessions than
on PDI sessions because trainees would be implementing the CDI sessions soon after their
completion of the 40-h training workshop. However, it is likely that other factors may have been
present to motivate providers to maintain or improvefidelity during the second part of the
treatment protocol, lessening the impact of the proximity of the initial training workshop on session
fidelity. For example, it is possible that trainees may have been motivated to maintain or improve
theirfidelity scores during PDI sessions because they had already received feedback from the
trainers on their initial CDI sessions during the follow-up consultation calls or via the post hoc
video review of their session or because they may have recently received additional training and
feedback during the on-site advanced training workshop. Another possibility is that trainees may
perceive PDI sessions as being more difficult to conduct than CDI sessions, due to the complex
nature of the time out procedure, and therefore may be prone to follow the protocol more closely,
resulting in higherfidelity scores. The complexity of the PDI sessions is highlighted, however, by
the fact the trainees’seemed to have more difficulty maintainingfidelity on PDI sessions the longer
the session went on. Finally, the trainees who complete the PDI portion of treatment with their
clients may simply be more committed to the PCIT training process and/or may have been better at
maintainingfidelity during the earlier PCIT sessions than other trainees, leading them to maintain
higherfidelity to the treatment protocol.
The results of the within session proceduralfidelity check demonstrated a wide range in the
presence and absence of checklist content items. Because PCIT is a relatively complex protocol to
implement and the nature of community-based services emphasizes efficiency and productivity,
this variability may indicate that it is difficult for trainees to cover all protocol items in a 50-min
session. Thus, leading them to cover the protocol items that they view as being key aspects of
treatment, while leaving out those items that they view as being expendable. Importantly, the
results indicate that trainees consistently completed most of the essential items in the session
protocols such as describing the relationship enhancement skills taught during CDI, discussing the
steps to giving a proper command, and coding and coaching parents. Although thesefindings
cannot necessarily be generalized to other samples and training sites, these results suggest that
treatment implementation was not compromised in the transition of PCIT from academic to
community-based settings. That is, trainees were consistently providing parents with the information
they need to learn the new parenting skills, and giving parents the opportunity to implement these
skills during in vivo coaching sessions. The regular use of the in vivo coaching component is
350The Journal of Behavioral Health Services & Research 40:3 July 2013 particularly important because the skills coaching and immediate feedback available through live
coaching have been shown to improve parents’skill acquisition and to be an important component of
effective parent-training programs.
20,21 Additionally, trainees consistently took the time to code
parent behaviors in order to assess their skill level and determine mastery of the skills. This form of
progress monitoring is a core component of PCIT, as stipulated in the PCIT Training Guidelines, and,
when discussed with parents, provides them with the opportunity to improve their skills.
However, the within session check of content items also indicated that trainees seem to struggle
with items related to monitoring the families’treatment gains, including reviewing parents’
strengths and areas of improvement following coaching, discussing parent ratings of child
disruptive behaviors, and collecting and discussing client homework. These shortcomings suggest
that trainees did not fully utilize opportunities to provide feedback to families, which may have
delayed the time it takes families to meet mastery criteria and subsequently advance through the
treatment protocol. Ultimately, this delay has the potential to lead to client dropout due to
frustration with the rate of progress in treatment. Additionally, the“use of standardized assessment
instruments to guide treatment,”is a core component of PCIT that trainees appear to be
overlooking. Therefore, results from this pilot study indicate that it may be beneficial for trainers to
emphasize the importance of monitoring treatment gains to trainees during both the training
workshop and the follow-up consultation calls. One way to do this is for trainers to ask trainees
about their clients’use of the relationship enhancement skills, the clients’ratings of their child’s
disruptive behaviors, and the frequency with which clients’complete homework during every
consultation call. Frequent monitoring by trainers of trainees’knowledge of their clients’skill
levels and treatment gains may serve to ensure that trainees will take the time to monitor these
important components of treatment.
Limitations
This pilot study attempted to address thefidelity of trainees’first work sample as indicated by
session date; however, in order to allow for potential technical difficulties, trainees were not
required to send in theirfirst work sample for each session. The primary reason for this
accommodation was that some trainees simply forgot to record the required session thefirst time
they conducted it while others were unable to send theirfirst work sample due to technical
difficulties during the recording process. Therefore, one limitation for this study is that it is difficult
to determine if practice effects may have impacted traineefidelity percentages.
Additionally, in order to allow for client attrition, trainees were permitted to send work samples from
different clients, rather than sending in all sessions from one family. Because some clients may be more
difficult to work with than others, the variety of work samples may have influenced the trainees’fidelity
scores overall and across session type and treatment phase.
37 Furthermore, there is a potential for
traineefidelity scores to be inflated if trainees picked which sessions to send in for review based on their
perception of how they performed during the session. Unfortunately, client dropout is a reality of
implementing EBPP in community settings, with 40 % to 60 % of clients dropping out of treatment
before termination is recommended.
40,41 Therefore, it will likely be difficult to examine trainee
implementationfidelity across treatment sessions with one family.
Another limitation of this study is the small sample size, which limits the generalizability of
these results. Because the training process requires extensive time and resources on the part of both
the trainees and trainers, one training team can only train a limited number of providers at any
given time. The time required to conduct a training workshop and the consultation phase of
training, including video reviews and frequent consultation calls, results in a time commitment
from the trainers of approximately 100 h per training cohort. The PCIT consultation phase, which
follows trainees through two completed PCIT cases, can range from 1 to 3 years. This time
commitment is in line with reports from Fixsen et al.
29 who stated that the successful
PCIT Implemented in Community Settings TRAVIS, BRESTAN-KNIGHT351 implementation of an EBPP will likely take 2 to 4 years to complete. Therefore, the time
commitment required from trainers makes it difficult to obtain a large sample of trainees for
research. Unfortunately, the presence of technical difficulties in the recording process and both
client and trainee attrition also led to the deletion of data for this study, which further limited this
study’s sample size and may limit statistical power.
Future directions
Given the absence of published studies available on the implementation of PCIT in community
settings there are multiple possible areas of interest for future research to consider. First, future
research in this area should consider conducting similar analyses on a larger sample of trainees to
increase power. Ideally, this replication would utilize the recently updated version of the PCIT
protocol, which includes minor changes to the session outlines and treatment integrity checklists.
42
Having a larger sample of trainees would make it possible to meaningfully examine any differences
in traineefidelity across trainee demographic variables. In particular, it would be helpful to
examine whether sessionfidelity varies across trainee characteristics such as trainee education
level, training site, and hours of direct client contact per week and to assess sessionfidelity while
controlling for these and other trainee characteristics.
Additionally, because previous research with other protocols suggests that client outcomes vary as a
result of providerfidelity to the treatment protocol, it would be beneficial to track trainee sessionfidelity
along with client treatment outcomes. Because PCIT is assessment driven, it would be relatively easy
for trainers to examine the benefits of maintaining treatmentfidelity in PCIT on client outcomes.
30,31
For example, trainers could utilize the parent report measure of child disruptive behavior that clients
complete each week in order to track child treatment gains, or the behavior observations of parent–child
interactions that occur frequently during the coach sessions of PCIT, in order to evaluate the parent’s
progress towards mastery and changes in the child’sbehavior.
Implications for Behavioral Health
This pilot study has some preliminaryfindings that could help to refine the methods currently
being used in the dissemination and implementation of complex evidence-based parent-training
programs such as PCIT into community settings. In particular, this study has identified the specific
challenges present in the implementation process for both trainers and trainees and provided certain
strategies to address these challenges. For example, it is clear that trainers need to emphasize the
unique components of PCIT to ensure that trainees are implementing the core components of PCIT
and that training needs to extend the information taught during the initial training workshop into
the consultation phase.
The challenges related to staff turnover in community agencies are especially relevant to
providing training and consultation in EBTs, such as PCIT. Training staff is expensive, and most
agencies have few resources to provide continuing education for staff. As mentioned previously,
32 % of our original training cohort was unable to participate in the follow-up consultation phase
of training, which is a high percentage of“lost”training dollars. Administrators and trainers should
be conscious of provider turnover rates in agencies seeking PCIT training and future research
should target the best methods to select potential trainees (e.g., skill proficiency, interpersonal
attributes, educational background, clinical experience, access to optimal training cases). Turnover
rates have also been found to negatively predictfidelity scores for EBT implementation,
43 which
ultimately has an impact on families seeking much-needed services. Future research would benefit
from the inclusion of an evaluation of organizational readiness in order to assess individual
therapist clinical competence, family and client engagement, and organizational support systems
prior to beginning the training process with providers.
44
352The Journal of Behavioral Health Services & Research 40:3 July 2013 Finally,findings from this study illuminate the need for EBTs being disseminated into
community settings to follow evidence-based training techniques. With regard to PCIT training,
ourfindings suggest that these training strategies should include well-developed training guidelines
and materials that are standardized between trainers, specific in their measurements of trainee
fidelity, and clear with regard to the remediation efforts required to improve low trainee
implementationfidelity. Given the potential to bridge the science-to-practice gap, the dissemination
and implementation of EBTs into the community settings is time-consuming, but extremely
important work. The highfidelity scores found in this study are especially encouraging considering
that the trainees in this study work with populations that are more heterogeneous than the
populations represented in RCTs, and in settings that are less controlled. Taken together, the results
of this pilot study suggest that there is a bright future for collaborations between PCIT training
teams and agencies wanting to adapt the PCIT model to address the needs of community families.
Acknowledgments
This work was supported by funds from the Auburn University Psychology Department. The authors
wish to thank Shalonda Brooks, Carisa Wilsie, Timothy Thornberry, and Kaitlin Baker for the integral
role they played in helping to train these participants in Parent–Child Interaction Therapy (PCIT) and
for the time they spent reviewing trainee session videos and completingfidelity checklists. The authors
also wish to acknowledge the work of Kathryn Smeraglia, who dedicated countless hours to reviewing
session videos to assess their length and content. Any opinions expressed are only the authors’and do
not necessarily represent the views of any affiliated institution.
Conflict of interest statementThis article is not under consideration elsewhere, and the authors have no
significant conflicts of interest to report.
References
1. American Psychological Association.Policy statement on evidence-based practice. American Psychological Association. Available
online athttp://www.apa.org/practice/resources/evidence-based-statement.pdf. Accessed June 6, 2011.
2. Kazdin AE. Evidence-based treatment and practice: New opportunities to bridge clinical research and practice, enhance the knowledge
base, and improve patient care.American Psychologist. 2008;63(3):146–159.
3. President’s New Freedom Commission on Mental Health. Achieving the promise: Transforming mental health care in America. Final
report (U.S. DHHS Pub. No. SMA 03–3832). Rockville, MD: U.S. Department of Health & Human Services, 2003.
4. Spielmans GI, Gatlin ET, McFall JP. The efficacy of evidence-based psychotherapies versus usual care for youths: Controlling confounds
in a meta-reanalysis.Psychotherapy Research. 2010;20(2):234–246.
5. Weisz JR, Jensen-Doss A, Hawley KM. Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct
comparisons.American Psychologist. 2006;61(7):671–689.
6. Hoagwood K, Burns BJ, Kiser L, et al. Evidence-based practice in child and adolescent mental health services.Psychiatric Services.
2001;52(9):1179–1189.
7. Hoagwood K, Hibbs E, Brent D, et al. Introduction to the special section: Efficacy and effectiveness in studies of child and adolescent
psychotherapy.Journal of Consulting and Clinical Psychology. 1995;63(5):683–687.
8. Southam-Gerow MA, Marder AM, Austin AA. Dissemination of evidence-based manualized treatments for children and families in
practice settings. In: Steele RG, Elkin TD, Roberts MC, eds.Handbook of evidence-based therapies for children and adolescents.
Springer Science + Business Media, LLC; 2008:447–469.
9. Weisz JR, Doss AJ, Hawley KM. Youth psychotherapy outcome research: A review and critique of the evidence base.Annual Review of
Psychology. 2005;56:337–363.
10. Eyberg SM, Nelson MM, Boggs SR. Evidence-based psychosocial treatments for children and adolescents with disruptive behavior.
Journal of Clinical Child & Adolescent Psychology 2008;37(1):215–237.
11. Serketich WJ, Dumas JE. The effectiveness of behavioral parent training to modify antisocial behavior in children: A meta-analysis.
Behavior Therapy. 1996;27(2):171–186.
12. Patterson G.Coercive family process. Eugene, OR: Castalia; 1982.
13. Webster-Stratton C. Predictors of treatment outcome in parent training for conduct disordered children.Behavior Therapy.
1985;16(2):223–243.
14. Brestan EV, Eyberg SM. Effective psychosocial treatments of conduct-disordered children and adolescents: 29 years, 82 studies, and
5,272 kids.Journal of Clinical Child & Adolescent Psychology. 1998;27(2):180–189.
PCIT Implemented in Community Settings TRAVIS, BRESTAN-KNIGHT353 15. Lundahl B, Risser HJ, Lovejoy MC. A meta-analysis of parent training: Moderators and follow-up effects.Clinical Psychology Review.
2006;26(1):86–104.
16. Piquero A, Farrington DP, Welsh B, et al. Effects of early family/parent training programs on antisocial behavior and delinquency.
Journal of Experimental Criminology. 2009;5(2):83–120.
17. McNeil CB, Hembree-Kigin TL.Parent–child interaction therapy. 2nd ed. New York, NY: Springer; 2010.
18. Eyberg SM, Boggs SR. Parent training for oppositional-defiant preschoolers. In: Schaefer CE, Briesmeister JM, eds.Handbook of parent
training:Parents as co-therapists for children's behavior problems. Oxford, England: Wiley; 1989:105–132.
19. Kazdin AE.Parent management training:Treatment for oppositional,aggressive,and antisocial behavior in children and adolescents.
Oxford, England; Oxford University Press, Inc.; 2005.
20. Shanley JR, Niec LN. Coaching parents to change: The impact of in vivo feedback on parents' acquisition of skills. Journal of Clinical
Child & Adolescent Psychology. 2010;39(2):282–287.
21. Kaminski JW, Valle L, Filene J, et al. A meta-analytic review of components associated with parent training program effectiveness.
Journal of Abnormal Child Psychology. 2008;36(4):567–589.
22. Eyberg SM, Funderburk B, McNeil, C, et al.Training guidelines for parent–child interaction therapy. PCIT International; 2009.
23. Herschell AD, McNeil CB, Urquiza AJ, et al. Evaluation of a treatment manual and workshops for disseminating, parent–child interaction
therapy.Administration and Policy in Mental Health and Mental Health Services Research. 2009;36(1):63–81.
24. Funderburk BW, Ware LM, Altshuler E, et al. Use and feasibility of telemedicine technology in the dissemination of parent–child
interaction therapy.Child Maltreatment. 2008;13(4):377–382.
25. Pearl E, Thieken L, Olafson E, et al. Effectiveness of community dissemination of parent–child interaction therapy. Psychological
Trauma: Theory, Research, Practice, and Policy 2011;4(2):204–213.
26. Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, et al. We don't train in vain: A dissemination trial of three strategies of training
clinicians in cognitive-behavioral therapy.Journal of Consulting and Clinical Psychology. 2005;73(1):106–115.
27. Durlak JA, DuPre EP. Implementation matters: A review of research on the influence of implementation on program outcomes and the
factors affecting implementation.American Journal of Community Psychology. 2008;41(3):327–350.
28. Fixsen DL, Blase KA, Naoom SF, et al. Core implementation components.Research on Social Work Practice. 2009;19(5):531–540.
29. Moncher FJ, Prinz RJ. Treatmentfidelity in outcome studies.Clinical Psychology Review. 1991;11(3):247–
266.
30. Mihalic S. The importance of implementationfidelity. Emotional & Behavioral Disorders in Youth. 2004;4:83–86.
31. Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: Are implementation effects out of control?
Clinical Psychology Review. 1998;18(1):23–45.
32. Schoenwald SK, Sheidow AJ, Chapman JE. Clinical supervision in treatment transport: Effects on adherence and outcomes.Journal of
Consulting and Clinical Psychology. 2009;77(3):410–421.
33. Henggeler SW, Schoenwald SK, Liao JG, et al. Transporting efficacious treatments tofield settings: The link between supervisory
practices and therapistfidelity in MST programs.Journal of Clinical Child Psychology. 2002;31(2):155–167.
34. Hayes S. Market-driven treatment development.Behavior Therapist. 1998;21:32–33.
35. Eyberg SM.Parent–child interaction therapy:Integrity checklists and session materials(version 2.10). Gainesville, FL: University of
Florida; 2010.
36. Bagner DM, Eyberg SM. Parent–child interaction therapy for disruptive behavior in children with mental retardation: A randomized
controlled trial.Journal of Clinical Child and Adolescent Psychology. 2007;36(3):418–429.
37. Werba BE, Eyberg SM, Boggs SR, et al. Predicting outcome in parent–child interaction therapy.Behavior Modification. 2006;30(5):618–
646.
38. Lyon AR, Budd KS. A community mental health implementation of parent–child interaction therapy (PCIT).Journal of Child and Family
Studies. 2010;19:654–668.
39. Eyberg SM, McDiarmid MD, Duke M, et al.Manual for the dyadic parent–child interaction coding system:Third edition. Gainesville,
FL: University of Florida; 2004.
40. Baekeland F, Lundwall L. Dropping out of treatment: A critical review.Psychological Bulletin. 1975;82(5):738–783.
41. Wierzbicki M, Pekarik G. A meta-analysis of psychotherapy dropout. Professional Psychology: Research and Practice 1993;24(2):190–
195.
42. Eyberg SM, Funderburk B. Parent–child interaction therapy protocol PCIT International; 2011.
43. Woltmann E, Whitley R, McHugo G, et al. The role of staff turnover in the implementation of evidence-based practices in mental health
care.Psychiatric Services. 2008;59(7):732–737.
44. Niec LN, Eyberg SM, Chase RM. Parent–child interaction therapy: Implementing and sustaining a treatment program for families of
young children with disruptive behavior disorders. In: Rubin A, ed.Programs and Interventions for Maltreated Children and Families at
Risk. New Jersey: Wiley; 2011:61–69.
354The Journal of Behavioral Health Services & Research 40:3 July 2013 Copyright
ofJournal ofBehavioral HealthServices &Research isthe property ofSpringer
Science
&Business MediaB.V.anditscontent maynotbecopied oremailed tomultiple sites
or
posted toalistserv without thecopyright holder'sexpresswrittenpermission. However,
users
mayprint, download, oremail articles forindividual use.
