The Complexities of E-TherapyPrior to beginning work on this discussion, please watch the Virtual Clinic (Links to an external site.)Links to an external site. video and review the Guidelines for the

Best Practices for Remote Psychological Assessment via Telehealth Technologies David D. Luxton National Center for Telehealth & Technology, Tacoma, Washington and University of Washington School of Medicine Larry D. Pruitt and Janyce E. Osenbach National Center for Telehealth & Technology, Tacoma, Washington The use and capabilities of telehealth technologies to conduct psychological assessments remotely are expanding. Clinical practitioners and researchers need to be aware of what influences the psychometric properties of telehealth-based assessments to assure optimal and competent assessments. The purpose of this review is to discuss the specific factors that influence the validity and reliability of remote psychological assessments and to provide best practices recommendations. Specific factors discussed include the lack of physical presence, technological issues, patient and provider acceptance of and comfort with technology, and procedural issues. Psychometric data regarding telehealth-based psycho- logical assessment and limitations to these data, as well as cultural, ethical, and safety considerations are discussed. The information presented is applicable to all mental health professionals who conduct psychological assessment with telehealth technologies.

Keywords:psychological assessment, telehealth, telemental health, video-conferencing, mobile devices The use of telehealth technologies in psychological practice has steadily increased over the last decade and their use is expected to grow substantially in the years ahead (American Psychological Association, 2010a;Maheu, Pulier, McMenamin, & Posen, 2012). Psychological assessment is an integral com- ponent of telemental health (TMH) practice and is necessary for diagnostics, screening, symptom monitoring, and evaluations oftreatment progress and outcomes. The use of telehealth tech- nologies to conduct psychological assessments from afar can provide convenience, reduce costs (e.g., travel avoidance), and enable access to assessment services when they are otherwise unavailable.

There are many technologies available to clinicians who are engaged in TMH practice. These include traditional telephones and video-teleconferencing (VTC) equipment for synchronous (real-time) communication as well as asynchronous (store-and- forward) technologies such as fax or email to send and receive assessment materials. The Internet is also used to administer psychological tests and measures remotely on web pages.

Internet-based testing and assessments can make use of technology-enhancements such as the use ofmultimedia content (i.e., pictures, videos, sounds, etc.), computer adaptive testing techniques, and automatic scoring and interpretation algorithms (seeBarak & Buchanan, 2004). The Internet can also be used for VTC by using personal computers (PCs) and off-the-shelf web- cams, which may be an affordable and highly accessible option for home-based TMH (Luxton, 2013a). More recently, smart mobile devices (i.e., smartphones and tablets) have emerged as a way to conduct psychological assessments. Assessment measures can now be in the form of an application or “app” on mobile devices or accessed via Internet connectivity. Traditional interview tech- niques conducted via VTC using webcams on PCs or mobile devices themselves may be augmented with electronic measures completed on the device with data uploaded to clinicians for review (Luxton, McCann, Bush, Mishkind, & Reger, 2011).

Given the numerous available telehealth technologies and their increasing use, practitioners who use them need to be cognizant of the factors that influence the psychometric properties of psycho- logical assessments when administered via those technologies.

Practitioners also need to know whether a given measure or DAVID D. L UXTON is a licensed clinical psychologist who received his PhD in clinical psychology from the University of Kansas. He is a Research Psychologist and Program Manager at National Center for Telehealth & Technology (T2) and an Affiliate Associate Professor of Psychiatry and Behavioral Sciences at the University of the Washington School of Med- icine in Seattle. His research and writing are focused in the areas of military psychological health, telehealth, and technology-based treatments.

L ARRY D. P RUITT is a licensed clinical psychologist who received his PhD from the University of Nevada, Reno. He is a Research Psychologist with the National Center for Telehealth & Technology (T2). His research and writings are concentrated in the areas of telemental health and the devel- opment of treatments for anxiety and mood related conditions.

J ANYCE E. O SENBACH received her PhD in Psychometrics from Fordham University. She is currently a Research Psychologist with the National Center for Telehealth & Technology. Her previous research publications examined the risk factor profiles of posttraumatic stress disorder (PTSD) and how those with PTSD interact with the law. Her current interest is on the influence of mobile technology to induce behavioral change.

T HE VIEWS EXPRESSED are those of the authors and do not reflect the official policy or position of the Department of Defense of the U.S. Government.

C ORRESPONDENCE CONCERNING THIS ARTICLE should be addressed to David D. Luxton, National Center for Telehealth & Technology (T2), Defense Centers of Excellence (DCoE) for Psychological Health & Traumatic Brain Injury, 9933 West Hayes Street, Joint Base Lewis-McChord, WA 98431.

E-mail:[email protected]@mail.mil Professional Psychology: Research and PracticeIn the public domain 2014, Vol. 45, No. 1, 27–35DOI:10.1037/a0034547 27 assessment technique is appropriate for use and they need to be familiar with the proper administration procedures in order to assure competent and ethical practice. Our purpose with this article is therefore to review the specific issues that influence the validity and reliability of telehealth-based assessments and to provide best practice recommendations for practitioners. Although we focus primarily on psychological assessment and evaluation during treat- ment services (i.e., diagnostic and symptom assessment), the prin- ciples and procedures that we discuss are pertinent to other remote assessment and testing applications including neuropsychological/ cognitive testing, forensic risk assessment, and occupational test- ing. Reliability and Validity Considerations and Recommendations Remote Physical Presence and Setting The primary and most obvious difference between telehealth and in-person assessment is the fact that the patient is not in the same room as the clinician. The lack of in-person presence may influence how information is assessed as well as what can be assessed. Nonverbal information is useful for determining the patient’s emotional state and, in some cases, risk behaviors. For example, olfactory sensory information can provide clinically rel- evant information regarding hygiene as well as the use of alcohol or other substances. Body posture, facial expressions, body lan- guage (e.g., foot tapping, hand wringing), as well as nonverbal emotional responses such as facial flushing, tearing up, and direc- tion of eye-gaze, also provide important information. The obser- vation of psychomotor and other medical symptoms are also important to observe during psychological assessments. Further, the observation of how an examinee approaches a test or measure may be critical for making an accurate assessment. The lack of physical presence, however, may limit the range of information available or how it can be observed. VTC assessments may be influenced by camera angle, screen size, room characteristics, or other technical factors (e.g., network bandwidth issues) that pro- hibit the observation of all behaviors. Further, the lack of physical presence in itself may influence a patient’s clinical presentation.

For example, patients who are socially anxious may underreport symptom severity when they are assessed remotely because the fear-evoking stimulus (i.e., the presence of the assessor) is phys- ically distant (Grady & Melcer, 2005). Also, in the case of home- based assessments, symptoms of panic disorder, agoraphobia, or the hyper-arousal symptom cluster of PTSD may be less salient because the patient is able to avoid situations that may be per- ceived as threatening, such as driving to a clinic or being around strangers in a busy waiting room.

To help assure the validity and reliability of remote assessment, it is first necessary to make sure that the environmental conditions at the remote location are conducive to the assessment procedures.

The location of the room for the assessment session should assure comfort and privacy. The assessment space should be large enough for the patient to feel comfortable in and assessments that involve groups and family interviews will require a space that is large enough to accommodate multiple people and, for some applica- tions, may require a table and other supplies (Kramer, Ayers,Mishkind, & Norem, 2011). In the case of home-based assessment, the presence of roommates, family members, pets, unexpected phone calls, or other distractions may disrupt the assessment process. It is therefore important for the practitioner to work with the patient to plan for and schedule sessions during a time that is free of potential disruptions. These considerations are particularly important for home-based assessments because the practitioner will have less control of the environment than they may have in an office setting.

Given the potential limits of what and how information can be collected during remote assessments, it may be appropriate to modify typical in-person assessment procedures. However, careful review of the instructions or administration manuals for measures and tests should be conducted to assure that procedures or envi- ronmental conditions for standardized administration are not al- tered in a way that threatens the reliability and validity of the assessment. In the case of VTC-based assessment, it may be necessary to ask a patient to hold a paper-and-pencil assessment (e.g., self-report measures or therapy homework) up to the video camera for viewing or to use larger handwriting because of small screen size or poor image quality. In addition, it may be helpful to ask the patient to read their responses out loud in scenarios where synchronous video is not used or when the connection quality is inadequate. When nonverbal information is useful but is unavail- able or limited, it may also be necessary to ask additional questions to improve the accuracy of the assessment. For example, if admin- istering the Hamilton Rating Scale for Depression (Hamilton, 1967), it may be appropriate to ask the patient to self-report symptoms of psychomotor retardation and agitation with specific follow-up questions such as “Do you have problems sitting still for more than a minute or two” or “Do you move more slowly than your coworkers?”.

It is important to note that the procedures for some assessments may not lend themselves to remote administration without physical presence. For example, the Wechsler Adult Intelligence Scale (WAIS-IV;Wechsler, 2008) involves hands-on interaction, such as administration of the Block Design, Matrix Reasoning, and Visual Puzzles subtests from the Perceptual Reasoning Index, which would be inappropriate and impractical to administer via VTC. In some cases, however, it may be feasible to administer assessments remotely, such as cognitive function testing (seeCul- lum, Weiner, Gehrmann, & Hynan, 2006), whereby an on-site staff member administers the assessment and then shares the results with a remote clinician who scores and interprets them. Also, some assessment instruments, such as the Minnesota Multiphasic Per- sonality Inventory (MMPI)-2 or WAIS-IV, should be physically safeguarded (not made openly available to the public) to assure the validity of future administrations. It is therefore important for the practitioner to consider whether remote administration of assess- ment materials presents a risk to the integrity of the instrument (e.g., by patients being able to print items at home or share them via the Internet, etc.). Practitioners should also consider whether there is an increased risk for dishonest responses (e.g., responses obtained from the Internet or someone else taking the assessment) because control over the testing environment is reduced (Bu- chanan, Johnson & Goldberg, 2005;Reips, 2000).

28 LUXTON, PRUITT, AND OSENBACH Technology Issues There are several technical issues associated with the use of tele- health technologies that may influence the quality of telehealth-based assessments. Eye gaze angle is the anglebetween the eye and the camera and the eye and the center of the display (Tam, Cafazzo, Seto, Salenieks, & Rossos, 2007). A potential problem when using VTC technology is that users often make eye contact with the image of the person on the screen rather than with the camera (Chen, 2002)—a phenomenon that gives the appearance that one person is looking down or away from the other person. Eye contact between a patient and a clinician is important because it provides visual cues to which the participants can respond (Grayson & Monk, 2003;Tam et al., 2007). Eye contact is also a source of clinical information that is useful for determining the presence of psychological states or particular disorders (e.g., autistic disorder).

Interpretation of facial expressions and affect may be difficult when eye contact is misleading, and eye gaze angle may also influence satisfaction with using VTC (Tam et al., 2007). Cameras should be positioned in a way that allows the images of both parties to appear straight-on and centered in their respective mon- itors so that both appear to speak eye-to-eye with each other (Kramer et al., 2011).Tam et al. (2007)pointed out that improved eye contact can be realized by increasing the horizontal distance of participants from the videoconferencing unit. Sometimes, how- ever, patients may shift position during a session, or the camera may be accidently be shifted from the optimal angle. It may therefore be necessary to ask the patient to make adjustments. It is also recommended to check-in with the patient to make sure they can see and hear clearly. The “picture in picture” function avail- able on many VTC devices can be used to ensure that the provider is clearly in frame as well.

Network connection quality is another important factor that can influence assessment capabilities and quality. Connection prob- lems can be caused by a variety of factors such as low quality equipment, an overloaded computer (e.g., too many programs running at one time), inadequate bandwidth, and user inexperience with VTC (Hyler, Gangure, & Batchelder, 2005;Jones, Johnston, Reboussin, & McCall, 2001;Luxton, Mishkind, Crumpton, Ayers, & Mysliwiec, 2012).Jones et al. (2001)found that inadequate audio quality can influence the ability to accurately gather infor- mation from the patient. For example, the observation of vocal properties (e.g., shakiness, inflection, and tone), as well as whether an individual may be crying, can be an important source of information regarding emotional states, and a low-quality audio connection may inhibit observation of this information. It is also important to consider that technological issues (e.g., bandwidth limitations, signal drop-outs, etc.) may influence how well the patient understands the clinician (not just how well the clinician understands the patient). It is therefore a best practice to test the quality of the connection at the beginning of the assessment session and to check in with the patient from time to time to make sure the connection quality is still adequate. Although there is some guidance available regarding minimum recommended band- width requirements for TMH (seeAmerican Telemedicine Asso- ciation, 2009), what is or is not adequate in any given application will depend on a variety of factors, including requirements for the type of assessment, environmental conditions, and the technology itself.Potential distractions caused by use of technology (e.g., web cam, personal computer, microphone, mobile device, etc.) may also introduce threats to the validity and reliability of remote psychological assessments. For example, when conducting a clin- ical assessment interview over a web cam, the patient may become distracted by inconsistent connections, error messages, or other technical anomalies (Germain, Marchand, Bouchard, Drouin, & Guay, 2009;Yoshino et al., 2001). Furthermore, technical mal- functions during telehealth sessions may become a source of frustration for patients (Luxton, Mishkind, et al., 2012;Luxton, Sirotin, & Mishkind., 2010). Persistent technical malfunctions that occur before or during remote assessment sessions may therefore influence motivation, agreeableness, and adherence to assessment procedures. It is thus important to have a plan to resolve technical malfunctions by expeditiously troubleshooting the problem, re- scheduling the session, or conducting it with an alternative me- dium (e.g., over the phone) if necessary.

It is also important to consider potential cognitive and/or sen- sory deficits that patients may have that could impair their ability to use telehealth technology. Technological aides (e.g., headsets, screen magnification devices, speech to text translation software, etc.) or the involvement of family members or other care givers that can assist may be appropriate. Possible fatigue or physical discomfort caused by technology use (e.g., eye strain when view- ing computer monitors) should also be evaluated before and during the assessment process, especially during lengthy assessment ses- sions. User Acceptance Generally, the validity of any psychological assessment is mod- ulated by the degree to which the person being assessed accepts (i.e., is willing to participate in) the context of a given assessment including the setting and manner in which the assessment is conducted (Cronbach, 1970;Elhai, Sweet, Guidotti Breting, & Kaloupek, 2012). An individual’s acceptance of a particular type of assessment is a multifaceted construct that depends on an individual’s physical and emotional state, motivation, attention, personality, and temperament. Poor acceptance has been cited as a factor that reduces compliance and the motivation to engage in mental health assessments (Rogers, 2001). Inadequate acceptance of TMH by either the patient or practitioner can therefore be expected to have a negative influence on the validity and reliability of psychological assessments.

Several reviews that discuss overall acceptance and satisfaction with TMH provide insight into the factors that may influence acceptance of telehealth-based psychological assessments. For ex- ample,Modai et al. (2006)reported that patients and providers are generally satisfied with VTC and that regular use of VTC im- proves the overall degree of satisfaction with this medium. A review byRichardson, Frueh, Grubaugh, Egede, & Elhai (2009) showed that there are high levels of user satisfaction and accep- tance with TMH across diverse clinical populations and services.

In particular, the benefits of reduced travel time, wait times, and lost work time, as well as greater sense of personal control over sessions were specifically associated with higher satisfaction among patients (seeHilty, Nesbitt, Kuenneth, Cruz, & Hales, 2007;Simpson, Bell, Knox, & Britton, 2005). These benefits may be especially important when considering the need for multiple 29 ASSESSMENT visits for some psychological assessments (i.e., initial interview, assessment battery administration, and feedback/treatment plan- ning).

Technological issues may also play an important part in the acceptability of telehealth-based assessments as well as rapport between the patient and practitioner (Glueck, 2013). A review by Backhaus et al. (2012)found that patient acceptability of VTC is generally on par with the acceptability of face-to-face contact, although the most common areas of dissatisfaction were associated with technical difficulties that interrupted sessions. In particular, problems with establishing a connection, connection speed, sound echo/feedback, and inability to transmit written material (e.g., a thought journal or activity log) in a way that allowed both the patient and the therapist to review it together were noted (Cowain, 2001;Folen, James, Earles, & Andrasik, 2001).Hyler, Gangure, and Batchelder (2005)found that both patients and providers preferred in-person assessment when compared to low-bandwidth VTC assessment, especially when detailed observation of patients was necessary; however, when high-bandwidth VTC was avail- able, this method was preferred over in-person assessments. Also, the loss or distortion of nonverbal behavior and other patient characteristics may also negatively impact a clinician’s acceptabil- ity of telehealth-based assessments (Grady & Melcer, 2005). So- lutions to these technology-based issues are available (e.g., faster Internet speeds, head sets, adjunct technology such as fax/docu- ment scanner) but come with financial costs that may limit feasi- bility. Overall, it is important for practitioners to consider that the factors that may influence the acceptability of telehealth-based psychological assessments may not be consistent across all assess- ment sessions or settings.

Cultural Considerations As with in-person psychological assessment and testing, prac- titioners and researchers who make use of telehealth technologies must attend to a broad range of cultural factors, including the patient’s age, technological familiarity, and culture-specific norms to assure valid and reliable assessments. For example, the remote physical presence inherent in TMH may create a barrier that reduces a patient’s engagement in the assessment process, espe- cially among members of cultures or groups that emphasize inter- personal connectedness or that rely heavily on nonverbal interac- tions (Nieves & Stack, 2007;Savin, Glueck, Chardavoyne, Yager, & Novins, 2011). An essential component of this is the provider’s ability to make use of whatever nonverbal communication is available. This is a skill that is essential when working with groups where the symptoms of mental illness may be minimized or stigmatized (e.g., Asian and Asian American populations, military populations;Yeung, Hails, Chang, Trinh, & Fava, 2011). Also, patients that are less comfortable or have less experience with technology, such as elderly or severely impoverished populations, may display a more drastic discrepancy between in-person and VTC assessments (Rohland, Saleh, Rohrer, & Romitti, 2000).

When working with specific populations, the provision of TMH services, including psychological assessment, should be tailored to the needs, resources, and technological infrastructure of the local community (Brooks, Spargo, Yellowlees, O’Neill, & Shore, 2013).

Preliminary work suggests that the customization of TMH to the needs and features of the group that is being served has thepotential to enhance access to psychological services within tradi- tionally underserved populations (Dwight-Johnson et al., 2011).

Shore et al. (2008)have also demonstrated that structured assess- ments provided via VTC were as acceptable as in-person assess- ments among an American Indian population and that the VTC use did not influence ratings of the perceived usability of the assess- ment, patient/provider interaction, or overall satisfaction. Also, individual backgrounds may present a strong contextualinfluence on whether and how technology is used (Brooks, Spargo,Yellowlees, O’Neill, & Shore, 2013). It is therefore important for practitioners to be sensitive to the capabilities and preferences of patients during TMH assessments and also recognize that telehealth-based assess- ments may not be appropriate for all individuals. The provision of a brief questionnaire or interview survey to evaluate previous experiences and preferences regarding technology may be helpful. Ethical, Privacy, and Safety Considerations Attention to general ethical principles, such as those specified in theEthical Principles of Psychologists and Code of Conduct (American Psychological Association, 2010b), is necessary during psychological practice whether it is conducted in-person or re- motely. Standard 9, Assessment, specifically addresses standards for psychological assessment and these too apply to all forms and mediums of psychological assessment. There are, however, aspects of telehealth-based assessments that require additional thought in order to assure ethical practice and optimal assessments. For example, it would be inappropriate practice to select, develop, or modify assessment instruments or alter procedures for remote administration without evidence of sufficient scientific validation or the appropriate disclosure of limitations. It is therefore neces- sary for practitioners to be familiar with what measures or tech- niques are supported by the scientific literature before using them.

Moreover, the assurance of patient confidentiality is an example of ethical practice that may influence the validity and reliability of psychological assessments. If a patient does not feel that their privacy is respected and valued by practitioners, the patient may be less willing to disclose information (Rogers, 2001).

Both physical and electronic safeguards should be used to assure confidentiality during remote psychological assessments. For ex- ample, people may speak louder when using telehealth technology than when in-person and electronic speakers may amplify sound significantly. Thus, audio should only be loud enough at each end so that both the patient and practitioner can be easily heard but not so loud that the TMH session can be overheard by people outside the room (Kramer et al., 2011). In the case of home-based TMH assessments, practitioners should assess whether the patient has any extra concerns about their privacy (i.e., whether family mem- bers or others may overhear the assessment session).

Practitioners conducting psychological assessments with tele- health technology also need to be cognizant of the applicability of the Health Insurance Portability and Accountability Act, the Health Information Technology for Economic and Clinical Health Act, applicable state law, and local privacy and security require- ments. The American Telemedicine Association provides specific practice standards and guidelines regarding this topic (American Telemedicine Association, 2009,2013). Appropriate disclosure of safeguards and potential risks associated with privacy and elec- tronic data should be addressed during the informed consent pro- 30 LUXTON, PRUITT, AND OSENBACH cess. As pointed out byMaheu and McMenamin (2013)however, the informed consent process or an agreement with patients may not be adequate in all situations, in all states, or in foreign coun- tries. Moreover, the diversity in the types of technologies, network infrastructures, and procedures for their use requires careful review of data security risks and requirements (seeKramer, Mishkind, Luxton, & Shore, 2013;Luxton, Kayl, & Mishkind, 2012).

Whether or not a particular technology platform or application meets standards for security and privacy of data can be compli- cated by complex issues such as whether and how digital data is stored on commercial servers, manufacturer agreements regarding ownership of transmitted data, and other potential technical risks to data security and privacy. Consultation with applicable legal or regulatory offices, information technology system administrators, equipment and software manufacturers, and other experienced experts in the field may be necessary when selecting telehealth platforms or when uncertainties regarding particular applications exist.

It is also important to consider the safety of the patient during TMH assessment sessions and to have a safety plan in place (Luxton, O’Brien, McCann, & Mishkind, 2012). A principal con- cern involves what to do if a patient becomes distressed or has a medical emergency during a remote assessment session. Safety plans should include procedures for contacting emergency services in the patient’s locale, alternate contact methods in case the syn- chronous telehealth connection is lost (e.g., backup phone contact), and plans for resolving technical problems (American Telemedi- cine Association, 2013;Luxton, O’Brien et al., 2012). The iden- tification and involvement of a local collaborator, such as a family member or friend of the patient that can assist with on-site tech- nical problems or provide support to a patient during emergency situations should be considered (American Telemedicine Associ- ation, 2013;Gros, Veronee, Strachan, Ruggerio, & Acierno, 2011; Luxton, O’Brien et al., 2012). The use of a collateral person, as well as overall telehealth assessment procedures, risks, and bene- fits should be addressed during the informed consent process.

Patients with a history of adverse reactions during treatment (e.g., severe panic attacks), or those who are at high risk of harm to self or others (e.g., family members in the case of home-based TMH), may not be appropriate candidates for telehealth services provided to clinically unsupervised settings (Luxton, O’Brien et al., 2012). These issues should also be considered when conduct- ing remote assessments, especially when providing assessment results. As noted byPope (1992), the form of assessment results and the process of presenting them may influence how patients interpret their meaning. Given these concerns, it is important to consider whether providing assessment results remotely via tele- health technologies is appropriate for any given patient. Prior to engaging in remote assessments, review of the patient’s history and potential risks, assessment of available technologies and the patient’s familiarity with them, as well as discussion of preferences regarding engaging in remote assessments are recommended. Al- ternative options (e.g., in-person) may be necessary if patients are not appropriate candidates for telehealth-based services due to safety concerns, clinical contraindications, technological barriers, or personal preferences (Luxton, O’Brien, et al., 2012;Luxton, Sirotin, & Mishkind, 2010). Selecting Assessment Measures: Psychometric Considerations It is important for practitioners to consider that even if an assessment tool has been shown to be valid and reliable in the original paper form or in one particular modality (i.e., in-person interview or computer-based administration) it does not necessar- ily mean that the measure or tool will be valid or reliable when conducted remotely via telehealth technologies. Moreover, even if there is empirical support for the use of a particular measure in one telehealth technology medium, such as on an Internet web page, it does not necessarily mean that it will be valid or reliable when transferred to another medium, such as a mobile device. The differences in the physical format of these mediums and proce- dures for use may influence the psychometric properties of mea- sures administered through them. As mentioned previously in this article, it is necessary for practitioners to be familiar with the available scientific literature regarding a measure or technique’s appropriateness for use.

Several published reviews provide useful information regarding the validity and reliability of remote psychological assessment via various telehealth technologies. For example,Hyler et al. (2005) conducted a review and meta-analysis that included 14 studies that compared telepsychiatry to in-person psychiatric assessments. Five studies used objective assessment measures, two studies used subjective measures (i.e., satisfaction measures), and seven studies used a combination of objective assessment measures and subjec- tive measures. The Brief Psychiatric Rating Scale (Overall & Gorham, 1962) was the most common assessment instrument. The meta-analysis results indicated that objective telepsychiatry assess- ments were similar to in-person assessments in regard to diagnosis or symptoms assessment. The Telemental Health Standards and Guidelines Working Group (Grady et al., 2011) also conducted a review of published data regarding the psychometric properties of remote telehealth-based psychological assessment. They noted several studies that have examined psychological assessment via clinical interviews or psychiatric interviews based on the Struc- tured Clinical Interview for DSM Disorders (SCID;Spitzer, Wil- liams, Gibbon, & First, 1990). Two studies (Ruskin et al., 1998; Shore, Savin, Orton, Beals, & Manson, 2007) demonstrated high reliability in the administration of the SCID. Comparability be- tween face-to-face and VTC is also demonstrated for the Hamilton Depression Rating Scale for depression (Kobak, 2004;Kobak, Williams, & Engelhardt, 2008).Backhaus and colleagues (2012) conducted a systematic review of the research on psychotherapy using VTC and reported that 69% of the 42 studies that they reviewed used a well-accepted psychometrically validated (in- person) standardized measure for treatment outcomes. Of these, the most common assessment (24% of the 42 studies) was the Beck Depression Inventory (BDI-II;Beck, Steer, & Brown, 1996).

Grady et al. also noted that there are not any psychometric data available regarding projective testing over VTC.

Grady et al. also noted evidence that demonstrates the feasibility of remote neuropsychological assessment (Hildebrand, Chow, Williams, Nelson, & Wass, 2004;Saligari et al., 2002) and com- parability of scores between remote and in-person assessment (Cullum et al., 2006;Loh, Donaldson, Flicker, Maher, & Gold- swain, 2007), as well as some research that has demonstrated differences on test scores (Ball, Tyrrel, & Long, 1999;Loh et al., 31 ASSESSMENT 2004;Montani et al., 1996). Cognitive assessments that have been examined and validated include the cognitive section of the Cam- bridge Mental Disorders of the Elderly Examination (Ball & Puffett, 1998), the Mini-Mental State Examination (Grob, Wein- trau, Sayles, Raskin, & Ruskin, 2001), the National Adult Reading Test, and the Adult Memory and Information Processing Battery (Tschirch, Walker, & Calvacca, 2006). The development of new norms has been recommended so that the thresholds used for impairment are valid when compared with face-to-face adminis- tration (Grady et al., 2011;Kirkwood, Peck, & Bennie, 2000).

Evidence of equivalence as well as differences between Internet-based questionnaire assessments and paper-and-pencil ad- ministrations of standardized measures has also been reported in the literature (Barak & English, 2002;Barak, Hen, Boniel-Nissim, & Shapira, 2008;Buchanan, 2002;Naglieri et al., 2004). For example, evaluation of online versions of BDI-II have shown that psychometric properties differ across testing modalities of psycho- metrically validated assessments, even when comparisons are made between equivalent samples, and that online BDI-II scores tended to be higher (Glaze & Cox, 1991;Peterson, Johannsson, & Carlsson, 1996;Schulenberg & Yutrzenka, 1999). Also, some evaluations of personality inventories administered online have shown differences in item loadings compared to paper-and-pencil versions (Buchanan, 2001;Buchanan, Johnson, & Goldberg, 2005). These differences may be due to the lessened impact of social desirability, an increase of self-disclosure in online assess- ment, or as the result of the technology-based presentation itself (Buchanan, 2003).

In sum, the literature base regarding the psychometric properties of telehealth-based assessments is growing; however, there are gaps in the literature that practitioners should consider when selecting particular assessment instruments and mediums. In par- ticular, the vast majority of available measures and assessment tools are based on norms that were established by employing traditional in-person procedures. The reevaluation of these tools with diverse populations, clinical presentations, and telehealth mediums is necessary to assure the validity of assessments con- ducted via telehealth technologies. It is critical for practitioners to be cognizant of assessment measure limitations and to appropri- ately disclose and document them in their practice. Keeping up with the scientific literature as well as publications by organiza- tions such as the APA and American Telemedicine Association is recommended. Discussion The validity and reliability of psychological assessments con- ducted via telehealth technology is influenced by factors that are both common to in-person assessment and unique to telehealth- based assessments. Although the psychometric characteristics and standardized procedures of traditional in-person psychological as- sessments provide useful information about how they may trans- late to other mediums, practitioners would be remiss to simply assume equivalency between in-person and remote administration of psychological assessments. It is therefore important for practi- tioners who already use or are considering the use of telehealth technologies to be familiar with these factors and to use appropri- ate administration techniques when conducting remote psycholog- ical assessments. It is feasible to assure reliable and valid remotepsychological assessments with appropriate knowledge, prepara- tion, and practice.

The use of telehealth technologies for remote psychological assessment has several benefits for both patients and practitio- ners. For example, telehealth-based assessments allow practi- tioners to conveniently monitor symptoms and other health variables betweenin-person or telehealth treatment sessions. Fur- ther, telehealth-based psychological assessment may improve care satisfaction and overall health outcomes by providing services that are specialized for the patient’s needs. In particular, telehealth technologies may provide access to clinical specialty assessments (e.g., neuropsychological assessments) that are not available in the patient’s locale. Telehealth-based psychological assessment may also increase access to services among patients who speak differ- ent languages. For example, a non-English speaking patient could engage in mental health assessment and treatment with a clinician who speaks their native language, regardless of physical location, which may minimize potential misunderstanding and misdiagnosis of a patient’s symptom report (Yeung et al., 2011). This would also be true for the use of American Sign Language and would remove the need for a third-party translator who may unintentionally change the meaning of a communication during the translation process. The use of telehealth technologies also allows patients to connect with providers that are trained in specialized assessments and who also have experience working with particular cultural groups (e.g., military culture, the elderly, specific ethnic groups, etc.).

Current and emerging technologies not only allow remote ad- ministration of traditional assessments but may also offer new or improved capabilities and methods of assessment. In particular, the growing field of mobile device apps has created opportunities for self-care assessments and symptom screening that were not pos- sible just a decade ago (Luxton, McCann et al., 2011). Assessment apps on smartphones and tablet PCs can also be useful for mea- suring the dynamic characteristics of a person. For instance, sub- jective mood or anxiety levels can be tracked in real-time and data from bio-feedback equipment can be tracked and analyzed re- motely. The small size and touch screen features of smartphones and tablet PC devices are factors that may influence the psycho- metric characteristics of assessments provided on these devices.

Preliminary data, however, have suggested that these devices may be a feasible platform for assessments that is comparable to paper- and-pencil and computer-based assessments (Bush, Skopp, Smo- lenski, Crumpton & Fairall, in press). More research is needed, however, regarding the psychometric properties of psychological assessments administered via mobile devices.

Assessments conducted with computer-simulated virtual reality environments are another emerging capability (Holloway & Reger, 2012;Parsons, Silva, Pair, & Rizzo, 2008;Riva, Wiederhold, & Molinari, 1998). Assessments can be built into the virtual envi- ronment so that measures appear virtually while a patient is in the virtual environment or to simulate real-world conditions that are useful for the assessment of particular variables. For example, virtual environments have been tested as a way to create environ- mental or social cues for assessment of emotional and behavioral responses among patients being treated for addiction or anxiety related behaviors (seeBordnick, Carter, & Traylor, 2011). The application of artificial intelligence technologies to conduct clini- cal interviews, psychological assessments, and evaluations is also 32 LUXTON, PRUITT, AND OSENBACH a promising area (Luxton, 2013b). Virtual intelligent agents capa- ble of human-like social interaction can be designed to conduct clinical interviews, analyze results, and provide feedback to pa- tients. These types of systems have already been developed for clinical training and some treatment services (DeAngelis, 2012; Parsons, Kenny, et al., 2008). Artificial intelligence-enabled tech- nologies that use advanced sensing and language processing ca- pabilities are also being developed to assess physiological and psychological variables. These advances in technology have the potential to increase the reliability and validity of psychological assessment, improve clinical care, and reduce costs for both pa- tients and practitioners.

In conclusion, the use of telehealth technologies provide an opportunity for psychologists and other health care professionals to expand the capabilities of their practice, provide quality ser- vices, and meet the health care needs of care seekers. The in- creased user demand for technology as well as the continued growth of TMH services will push the need for telehealth-based psychological assessments. The adherence to best practices and competencies for psychological assessment via telehealth technol- ogies is the responsibility of psychologists and others who provide such services. Practitioners must remain familiar with available research and guidelines before engaging in remote assessments.

Moreover, practitioners must consider the applicability to specific populations and appropriateness of any assessment measure or technique on a case-by-case basis. Ultimately, the decision to conduct psychological assessments from afar should depend on both the practitioner’s and patient’s comfort level with the process. References American Psychological Association. (2010a).Telepsychology is on the rise. Monitor on Psychology, 41,11.

American Psychological Association. (2010b).American Psychological Association ethical principles of psychologists and code of conduct.

Retrieved fromhttp://www.apa.org/ethics/code American Telemedicine Association. (2009).Practice guidelines for videoconferencing-based telemental health. Retrieved fromhttp://www .americantelemed.org/practice/standards/ata-standards-guidelines American Telemedicine Association. (2013).Practice guidelines for videoconferencing-based online mental health services. Retrieved from http://www.americantelemed.org/practice/standards/ata-standards- guidelines Backhaus, A., Agha, Z., Maglione, M. L., Repp, A., Ross, B., Zuest, D., & Thorp, S. R. (2012). Videoconferencing psychotherapy: A systematic review.Psychological Services, 9,111–131.doi:10.1037/a0027924 Ball, C., & Puffett, A. (1998). The assessment of cognitive function in the elderly using videoconferencing.Journal of Telemedicine and Telecare, 4,36 –38.doi:10.1258/1357633981931362 Ball, C., Tyrrell, J., & Long, C. (1999). Scoring written material from the Mini-Mental State Examination: A comparison of face-to-face, fax and video-linked scoring.Journal of Telemedicine and Telecare, 5,253–256.

doi:10.1258/1357633991933819 Barak, A., & Buchanan, T. (2004). Internet-based psychological testing and assessment. In R. Kraus, G. Stricker, & C. Speyer (Eds.),Online counseling: A handbook for mental health professionals(pp. 4217– 4239). San Diego, CA: Elsevier Academic Press.

Barak, A., & English, N. (2002). Prospects and limitation of psychological testing on the internet.Journal of Technology in Human Services, 19, 65– 89.doi:10.1300/J017v19n02_06 Barak, A., Hen, L., Boniel-Nissim, M., & Shapira, N. (2008). A compre- hensive review and a meta-analysis of the effectiveness of internet-basedpsychotherapeutic interventions.Journal of Technology in Human Ser- vices, 26,109 –160.doi:10.1080/15228830802094429 Beck, A. T., Steer, R. A., & Brown, G. K. (1996).Manual for the Beck Depression Inventory–II. San Antonio, TX:Psychological Corporation.

Bordnick, P. S., Carter, B. L., & Traylor, A. C. (2011). What virtual reality research in addictions can tell us about the future of obesity assessment and treatment.Journal of Diabetes Science and Technology, 5,265–271.

Brooks, E., Spargo, G., Yellowlees, P., O’Neill, P., & Shore, J. H. (2013).

Integrating culturally appropriate care into telemental health practice. In K. Myers & C. L. Turvey (Eds.),Telemental Health: Clinical, Technical, and Administrative Foundations for Evidenced-Based Practice(pp. 63– 79). Waltham, MA: Elsevier.

Buchanan, T. (2001). Online personality assessment. In U.-D. Reips & M.

Bosnjak (Eds.),Dimensions of Internet Science(pp. 57–74). Lengerich, DE: Pabst Science Publishers.

Buchanan, T. (2002). Online assessment: Desirable or dangerous?Profes- sional Psychology: Research and Practice, 33,148 –154.doi:10.1037/ 0735-7028.33.2.148 Buchanan, T. (2003). Internet-based questionnaire assessment: Appropriate use in clinical contexts.Cognitive Behaviour Therapy, 32,100 –109.

doi:10.1080/16506070310000957 Buchanan, T., Johnson, J. A., & Goldberg, L. R. (2005). Implementing a five-factor personality inventory for use on the internet.European Jour- nal of Psychological Assessment, 21,115–127.doi:10.1027/1015-5759 .21.2.115 Bush, N. E., Skopp, N. A., Smolenski, D., Crumpton, R., & Fairall, J. (in press).Behavioral screening measures delivered with a smartphone “app”: Psychometric properties and user preference.Journal of Nervous and Mental Disease.

Chen, M. (2002). Leveraging the asymmetric sensitivity of eye contact for videoconferencing. In L. Terveen (Ed.),Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 2002(pp. 49 –56).

Minneapolis, MN: ACM Press.

Cowain, T. (2001). Cognitive-behavioural therapy via videoconferencing to a rural area.Australian and New Zealand Journal of Psychiatry, 35, 62– 64.doi:10.1046/j.1440-1614.2001.00853.x Cronbach, L. J. (1970).Essentials of psychological testing(3rd ed.). New York, NY: Harper & Row.

Cullum, C. M., Weiner, M. F., Gehrmann, H. R., & Hynan, L. S. (2006).

Feasibility of telecognitive assessment in dementia.Assessment, 13, 385–390.doi:10.1177/1073191106289065 DeAngelis, T. (2012, March). A second life for practice?Monitor on Psychology, 43,48. Retrieved fromhttp://www.apa.org/monitor/2012/ 03/avatars.aspx Dwight-Johnson, M., Aisenberg, E., Golinelli, D., Hong, S., O’Brien, M., & Ludman, E. (2011). Telephone-based cognitive-behavioral therapy for Latino patients living in rural areas: A randomized pilot study.Psychi- atric Services, 62,936 –942.doi:10.1176/appi.ps.62.8.936 Elhai, J. D., Sweet, J. J., Guidotti Breting, L. M., & Kaloupek, D. (2012).

In J. J. Vasterling, R. A. Bryant, & T. M. Keane (Eds.),PTSD and mild traumatic brain injury(pp. 174 –198). New York, NY: The Guilford Press.

Folen, R. A., James, L. C., Earles, J. E., & Andrasik, F. (2001). Biofeed- back via telehealth: A new frontier for applied psychophysiology.Ap- plied Psychophysiology and Biofeedback, 26,195–204.doi:10.1023/A:

1011346103638 Germain, V., Marchand, A., Bouchard, S. Drouin, M. S., & Guay, S.

(2009). Effectiveness of cognitive behavioural therapy administered by videoconference for posttraumatic stress disorder.Cognitive Behaviour Therapy, 38,42–53.doi:10.1080/16506070802473494 Glaze, R., & Cox, J. L. (1991). Validation of a computerized version of the 10-item (self-rating) Edinburgh postnatal depression scale.Journal of Affective Disorders, 22,73–77.doi:10.1016/0165-0327(91)90086-8 Glueck, D. (2013). Establishing therapeutic rapport in telemental health. In 33 ASSESSMENT K. Myers, & C. L. Turvey (Eds.),Telemental health: Clinical, technical and administrative foundations for evidence-based practice(pp. 29 – 46). Waltham, MA: Elsevier.

Grady, B. J., & Melcer, T. (2005). A retrospective evaluation of telemental healthcare services for remote military populations.Telemedicine and e-Health, 11,551–558.doi:10.1089/tmj.2005.11.551 Grady, B., Myers, K. M., Nelson, E-L., Belz, N., Bennett, L., Carnahan, L., & Voyles, D. (2011). Evidence-based practice for telemental health.

Telemedicine and e-Health, 17,131–148.doi:10.1089/tmj.2010.0158 Grayson, D. M., & Monk, A. F. (2003). Are you looking at me? Eye contact and desktop video conferencing.ACM Transactions on Computer-Human Interaction, 10,221–243.doi:10.1145/937549 .937552 Grob, P., Weintraub, D., Sayles, D., Raskin, A., & Ruskin, P. (2001).

Psychiatric assessment of a nursing home population using audiovisual telecommunication.Journal of Geriatric Psychiatry and Neurology, 14, 63– 65.doi:10.1177/089198870101400203 Gros, D. F., Veronee, K., Strachan, M., Ruggiero, K. J., & Acierno, R.

(2011). Managing suicidality in home-based telehealth.Journal of Tele- medicine and Telecare, 17,332–335.doi:10.1258/jtt.2011.101207 Hamilton, M. (1967). Development of a rating scale for primary depressive illness.British Journal of Social & Clinical Psychology, 6,278 –296.

doi:10.1111/j.2044-8260.1967.tb00530.x Hildebrand, R., Chow, H., Williams, C., Nelson, M., & Wass, P. (2004).

Feasibility of neuropsychological testing of older adults via videocon- ference: Implications for assessing the capacity for independent living.

Journal of Telemedicine and Telecare, 10,130 –134.doi:10.1258/ 135763304323070751 Hilty, D. M., Nesbitt, T. S., Kuenneth, C. A., Cruz, G. M., Hales, R. E.

(2007). Rural versus suburban primary care needs, utilization, and sat- isfaction with telepsychiatric consultation.The Journal of Rural Health, 23,163–165.doi:10.1111/j.1748-0361.2007.00084.x Holloway, K. M., & Reger, G. M. (2012). T2 virtual PTSD experience: A virtual worlds environment to educate Service Members and Veterans about combat-related post-traumatic stress disorder.International Jour- nal of Human-Computer Interaction. Advance online publication.doi:

10.1080/10447318.2012.735186 Hyler, S. E., Gangure, D. P., & Batchelder, S. T. (2005). Can telepsychiatry replace in-person psychiatric assessments? A review and meta-analysis of comparison studies.CNS Spectrums, 10,403– 413.

Jones, B. N., Johnston, D., Reboussin, B., & McCall, W. V. (2001).

Reliability of telepsychiatry assessments: Subjective versus observa- tional ratings.Journal of Geriatric Psychiatry and Neurology, 14,66 – 71.doi:10.1177/089198870101400204 Kirkwood, K. T., Peck, D. F., & Bennie, L. (2000). The consistency of neuropsychological assessments performed via telecommunication and face to face.Journal of Telemedicine and Telecare, 6,147–151.doi:

10.1258/1357633001935239 Kobak, K. A. (2004). Acomparison of face-to-face and videoconference administration of the Hamilton Depression Rating Scale.Journal of Tele- medicine and Telecare, 10,231–235.doi:10.1258/1357633041424368 Kobak, K. A., Williams, J. B., & Engelhardt, N. (2008). A comparison of face-to-face and remote assessment of inter-rater reliability on the Ham- ilton Depression Rating Scale via videoconferencing.Psychiatry Re- search, 158,99 –103.doi:10.1016/j.psychres.2007.06.025 Kramer, G. K., Ayers, T., Mishkind, M., & Norem, A. (2011).DoD Telemental Health Guidebook. Tacoma, WA: National Center for Tele- health and Technology. Retrieved fromhttp://t2health.org/sites/default/ files/cth/guidebook/tmh-guidebook_06-11.pdf Kramer, G. M., Mishkind, M. C., Luxton, D. D., & Shore, J. H. (2013).

Managing risk and protecting privacy in telemental health: An overview of legal, regulatory, and risk management issues. In K. Myers & C. L.

Turvey (Eds.)Telemental health: Clinical, technical and administrativefoundations for evidence-based practice(pp. 83–107). Waltham, MA:

Elsevier.

Loh, P. K., Donaldson, M., Flicker, L., Maher, S., & Goldswain, P. (2007).

Development of a telemedicine protocol for the diagnosis of Alzhei- mer’s disease.Journal of Telemedicine and Telecare, 13,90 –94.doi:

10.1258/135763307780096159 Loh, P. K., Ramesh, P., Maher, S., Saligari, J., Flicker, L., & Goldswain, P. (2004). Can patients with dementia be assessed at a distance? The use of telehealth and standardised assessments.Internal Medicine Journal, 34,239 –242.doi:10.1111/j.1444-0903.2004.00531.x Luxton, D. D. (2013a). Considerations for planning and evaluating eco- nomic analyses of telemental health.Psychological Services, 10,276 – 282.doi:10.1037/a0030658 Luxton, D. D. (2013b). Artificial intelligence in psychological practice:

Current and future applications and implications.Professional Psychol- ogy: Research and Practice.doi:10.1037/a0034559 Luxton, D. D., Kayl, R. A., & Mishkind, M. C. (2012). mHealth data security: The need for HIPAA-compliant standardization.Telemedicine and e-Health, 18,284 –288.

Luxton, D. D., McCann, R. A., Bush, N. E., Mishkind, M. C., & Reger, G. M. (2011). mHealth for mental health: Integrating smartphone tech- nology in behavioral healthcare.Professional Psychology: Research and Practice, 42,505–512.doi:10.1037/a0024485 Luxton, D. D., Mishkind, M. C., Crumpton, R. M., Ayers, T. D., & Mysliwiec, V. (2012). Usability and feasibility of smartphone video capabilities for telehealth care in the U.S. Military.Telemedicine and e-Health, 18,409 – 412.doi:10.1089/tmj.2011.0219 Luxton, D. D., O’Brien, K., McCann, R. A., & Mishkind, M. C. (2012).

Home-based telemental healthcare safety planning: What you need to know.Telemedicine and e-Health, 18,629 – 633.doi:10.1089/tmj.2012 .0004 Luxton, D. D., Sirotin, A. P., & Mishkind, M. C. (2010). Safety of telemental healthcare delivered to clinically unsupervised settings: A systematic review.Telemedicine and e-Health, 16,705–711.doi:

10.1089/tmj.2009.0179 Maheu, M. M., & Mcmenamin, J. (2013, March 28). Telepsychiatry: The perils of using Skype.Psychiatric Times. Retrieved fromhttp://www .psychiatrictimes.com/blog/telepsychiatry-perils-using-skype Maheu, M., Pulier, M., McMenamin, J., & Posen, L. (2012). The future of telepsychology, telehealth, and various technologies in psychological research.Professional Psychology: Research and Practice, 43,613– 621.doi:10.1037/a0029458 Modai, I., Jabarin, M., Kurs, R., Barak, P., Hanan, I., & Kitain, L. (2006).

Cost effectiveness, safety, and satisfaction with video telepsychiatry versus face-to-face care in ambulatory settings.Telemedicine and e-Health, 12,515–520.doi:10.1089/tmj.2006.12.515 Montani, C., Billaud, N., Couturier, P., Fluchaire, I., Lemaire, R., Malterre, C., Franco, A. (1996). “Telepsychometry”: A remote psychometry con- sultation in clinical gerontology: Preliminary study.Telemedicine Jour- nal, 2,145–150.doi:10.1089/tmj.1.1996.2.145 Naglieri, J. A., Drasgow, F., Schmit, M., Handler, L., Prifitera, A., & Margolis, A., & Velasquez, R. (2004). Psychological testing on the Internet: New problems, old issues.American Psychologist, 59,150 – 162.doi:10.1037/0003-066X.59.3.150 Nieves, J. E., & Stack, K. M. (2007). Hispanics and telepsychiatry.Psy- chiatric Services, 58,877.doi:10.1176/appi.ps.58.6.877 Overall, J. E., & Gorham, D. R. (1962). The brief psychiatric rating scale.

Psychological Reports, 10,799 – 812.doi:10.2466/pr0.1962.10.3.799 Parsons, T. D., Kenny, P., Ntuen, C. A., Pataki, C. S., Pato, M. T., Rizzo, A.A....(2008). Objective structured clinical interview training using a virtual human patient.Studies in Health Technology and Informatics, 132,357–362.

Parsons, T. D., Silva, T. M., Pair, J., & Rizzo, A. A. (2008). Virtual environment for assessment of neurocognitive functioning: Virtual re- 34 LUXTON, PRUITT, AND OSENBACH ality cognitive performance assessment test.Studies in Health Technol- ogy and Informatics, 132,351–356.

Peterson, L., Johannsson, V., & Carlsson, S. G. (1996). Computerized testing in a hospital setting: Psychometric and psychological effects.

Computers in Human Behavior, 12,339 –350.doi:10.1016/0747- 5632(96)00012-X Pope, K. S. (1992). Responsibilities in providing psychological test feed- back to clients.Psychological Assessment, 4,268 –271.doi:10.1037/ 1040-3590.4.3.268 Reips, U-D. (2000). The web experiment method: Advantages, disadvan- tages, and solutions. In M. H. Birnbaum (Ed.),Psychochological exper- iments on the internet(pp. 89 –117). San Diego, CA: Academic Press.

doi:10.1016/B978-012099980-4/50005-8 Richardson, L. K., Frueh, B. C., Grubaugh, A. L., Egede, L., & Elhai, J. D.

(2009). Current directions in videoconferencing tele-mental health re- search.Clinical Psychology: Science and Practice, 16,323–338.doi:

10.1111/j.1468-2850.2009.01170.x Riva, G., Wiederhold, B., & Molinari, E. (1998).Virtual environments in clinical psychology and neuroscience: Methods and techniques in ad- vanced patient-therapist interaction. (Vol.58). Amsterdam, NL: IOS Press.

Rogers, R. (2001).Handbook of diagnostic and structured interviewing.

New York, NY: The Guilford Press.

Rohland, B. M., Saleh, S. S., Rohrer, J. E., & Romitti, P. A. (2000).

Acceptability of telepsychiatry to a rural population.Psychiatric Ser- vices, 51,672– 674.doi:10.1176/appi.ps.51.5.672 Ruskin, P. E., Reed, S., Kumar, R., Kling, M. A., Siegel, E., Rosen, M., & Hauser, P. (1998). Reliability and acceptability of psychiatric diagnosis via telecommunication and audiovisual technology.Psychiatric Ser- vices, 49,1086 –1088.

Saligari, J., Flicker, L., Loh, P. K., Maher, S., Ramesh, P., & Goldswain, P. (2002). The clinical achievements of a geriatric telehealth project in its first year.Journal of Telemedicine and Telecare, 8(Suppl 3), 53–55.

doi:10.1258/13576330260440862 Savin, D., Glueck, D. A., Chardavoyne, J., Yager, J., & Novins, D. K.

(2011). Bridging cultures: Child psychiatry via videoconferencing.Child and Adolescent Psychiatric Clinics of North America, 20,125–134.

doi:10.1016/j.chc.2010.09.002Schulenberg, S. E., & Yutrzenka, B. A. (1999). The equivalence of com- puterized and paper-and-pencil psychological instruments: Implications for measures of negative affect.Behavior Research Methods, Instru- ments & Computers, 31,315–321.doi:10.3758/BF03207726 Shore, J., Brooks, E., Savin, D., Orton, H., Grigsby, J., & Manson, S.

(2008). Acceptability of telepsychiatry in American Indians.Telemedi- cine and e-Health, 14,461– 466.doi:10.1089/tmj.2007.0077 Shore, J. H., Savin, D., Orton, H., Beals, J., & Manson, S. M. (2007).

Diagnostic reliability of telepsychiatry in American Indian veterans.The American Journal of Psychiatry, 164,115–118.doi:10.1176/appi.ajp .164.1.115 Simpson, S., Bell, L., Knox, J., & Britton, P. (2005). Therapy via video- conferencing: A route to client empowerment?Clinical Psychology & Psychotherapy, 12,156 –165.doi:10.1002/cpp.436 Spitzer, R. L., Williams, J. B. W., Gibbon, M., & First, M. B. (1990).

Structured Clinical Interview for DSM–III–R: Patient Edition/Non- Patient Edition (SCID-P/SCID-NP). Washington, DC: American Psy- chiatric Press.

Tam, T., Cafazzo, J. A., Seto, E., Salenieks, M. E., & Rossos, P. G. (2007).

Perception of eye contact in video teleconsultation.Journal of Telemedi- cine and Telecare, 13,35–39.

Tschirch, P., Walker, G., & Calvacca, L. T. (2006). Nursing in tele-mental health.Journal of Psychosocial Nursing and Mental Health Services, 44, 20 –27.doi:10.1089/tmj.2010.0158 Wechsler, D. (2008).Wechsler Adult Intelligence Scale–Fourth Edition (WAIS-IV). San Antonio, TX: Pearson.

Yeung, A., Hails, K., Chang, T., Trinh, N., & Fava, M. (2011). A study of the effectiveness of telepsychiatry-based culturally sensitive collabora- tive treatment of depressed Chinese Americans.BMC Psychiatry, 11, 1– 8.doi:10.1186/1471-244X-11-154 Yoshino, A., Shigemura, J., Kobayashi, Y., Nomura, S., Shishikura, K., Den, R., & Ashida, H. (2001). Telepsychiatry: Assessment of televideo psychiatric interview reliability with present- and next-generation inter- net infrastructures.Acta Psychiatrica Scandinavica, 104,223–226.doi:

10.1034/j.1600-0447.2001.00236.x Received December 5, 2012 Revision received June 18, 2013 Accepted June 20, 2013 E-Mail Notification of Your Latest Issue Online! Would you like to know when the next issue of your favorite APA journal will be available online? This service is now available to you. Sign up at http://notify.apa.org/ and you will be notified by e-mail when issues of interest to you become available! 35 ASSESSMENT