Statistics-theory & research

We're the ideal place for homework help. If you are looking for affordable, custom-written, high-quality and non-plagiarized papers, your student life just became easier with us. Click either of the buttons below to place your order.


Order a Similar Paper Order a Different Paper

This  assignment will require you to take the PICO, 5 research articles and place them in a table for analysis.  You will critique each article and identify the most important parts of the research, analysis, and findings.

Then  you will summarize the articles by grading the research, and identifying gaps in the literature as well as possible interventions (see grading rubric and examples).

Instructions:

1. Identify your PICO or research question of interest

2. Gather 5 research articles on your topic: be sure to save them and submit them along with the matrix;  please make sure they are PDF documents.

3. Do not use clinical guidelines or Cochran Reviews, abstracts, future research reports or  poster presentations.  You can search for research only by indicating “research” when you do an advanced search.  If you can’t answer a lot of the questions, it is probably not a research article.  

4. We do not recommend that you use more than one  qualitative research article or systematic review or meta analysis.  These are harder to evaluate because they have so much more information in them.  Remember it is not individuals in these studies- it is the articles.  Use the reference list in the systematic reviews or meta analyses to find individual studies that may be easier to understand and use.

5. Review sample matrices and summaries

6. Use matrix table- one for each article and critique the parts of the article using the rubric

7.  Be sure to identify the evaluation tool used to grade the evidence such as (See below for grading the evidence tools)

8. Identify where there are issues with the articles and what gaps were not addressed with the research; be prepared this may change the way you look at your topic or may result in a slightly different direction for your area of interest. This is ok- that is what you want to accomplish with this assignment. It will really assist you as you move forward with your project.

9. For this assignment you will turn in the matrix tables, summary, references and pdf copies of your 5 articles. 

10. Key definitions:

1. Level of evidence:  the process used to evaluate the level of evidence of your articles- such as Jones Hopkins,  Cincinnati Children’s evaluation etc,

2. Evaluation tool: use the method and describe how you arrived at the scoring or knowing that the article included all content it needed to- such as CASP; 

3. Instrument: What type of instrument or tool was used in the article?  This could be  a depression screening tool, Nurse satisfaction tool etc.  Describe the instrument- how many questions, reliability- consistency with test-retest, Cronbach Alpha, inter-rater reliablity; validity with content validity, face validity 

Links to critical appraisal tools to evaluate research quality:

Joanna Briggs Institute (joannabriggs.org) https://jbi.global/critical-appraisal-tools

CASP checklists https://casp-uk.net/casp-tools-checklists/

Mixed Method appraisal checklist McGill:  http://mixedmethodsappraisaltoolpublic.pbworks.com/w/page/24607821/FrontPage

Cincinnati Children’s Hospital Medical Center Legend tools- very helpful: https://www.cincinnatichildrens.org/research/divisions/j/anderson-center/evidence-based-care/legend

Johns Hopkins EBP Models and Tools:  https://www.hopkinsmedicine.org/evidence-based-practice/ijhn_2017_ebp.html

My PICOT Question: “In the adolescent population with mental health issues seen in primary care settings, how does utilization of Ask Suicide-Screening Questions (ASQ) toolkit impact patient referrals to the psychiatric department over 3 months

Population: Adolescents asked screening questions in primary care setting

I: Ask Suicide Screening questions toolkit

C: those not screened with the toolkit

O: prevention of suicide attempts or worsening of depression

T: 3 months

ARTICLES

Quantitative Articles MUST be Used. Kindly help verify that the attached articles are Quantitative

I found the following articles, you can use any 5 of them: however; I don’t know which ones are quantitative, I found a bunch of these: the articles are attached.

Characteristics of cancer patients who died by suicide: A quantitative study of 15‐year coronial records 

Vera Y. Men1 | Clifton R. Emery1 | Paul S. F. Yip1

Validation of the ask suicide-screening questions (ASQ) with youth in outpatient specialty and primary care clinics 

Laika D. Aguinaldo, Shayla Sullivant, Elizabeth C. Lanzillo, Abigail Ross, Jian-Ping He, Andrea Bradley-Ewing, Jeffrey A. Bridge, Lisa M. Horowitz, Elizabeth A. Wharff 

Development of the Uni Virtual Clinic: an online programme for improving the mental health of university students 

Louise M. Farrer , Amelia Gulliver, Natasha Katruss, Kylie Bennett, Anthony Bennett, Kathina Ali† and Kathleen M. Griffiths

The Feasibility and Impact of a Suicide Risk Screening Program in Rural Adult Primary Care: A Pilot Test of the Ask Suicide-Screening Questions Toolkit 

Mary A. LeCloux, Ph.D., Mathew Weimer, M.D., Stacey L. Culp, Ph.D., Karissa Bjorkgren, B.S., Samantha Service, M.S., John V. Campo, M.D.

Improving Suicidal Ideation Screening and Suicide Prevention Strategies on Adult Nonbehavioral Health Units

Anne C. Lindstrom, DNP, APRN Rush University, Northwestern Medicine Central DuPage Hospital, Winfield, IL Melinda Earle, DNP, RN

Contact between patients with suicidal ideation and nurses in mental health wards: Development and psychometric evaluation of a questionnaire

Joeri Vandewalle, Veerle Duprez,1 Dimitri Beeckman, Ann Van Hecke,and Sofie Verhaeghe

Patient Opinions About Screening for Suicide Risk in the Adult Medical Inpatient Unit 

Deborah J. Snyder, Elizabeth D. Ballard, Ian H. Stanley, Erica Ludi, Julie Kohn-Godbout, Maryland Pao, Lisa M. Horowitz.

Screening Youth for Suicide Risk in Medical Settings (Time to Ask Questions) 

Lisa M. Horowitz, PhD, MPH, Jeffrey A. Bridge, PhD, Maryland Pao, MD, Edwin D. Boudreaux

DNP 618 Article Matrix and Analysis

Student Name __________________________________

PICO Question ___________________________________________________________________

Search process :

Search terms:

Data bases:

Total number of articles obtained from search results: N=

Number of articles initially excluded based on abstract reading: N=

Number of articles reviewed: N=

Number of articles excluded based on criteria: N=

Inclusion Criteria:

Exclusion Criteria:

Number of systematic reviews or meta analyses used in Matrix- N =

Repeat this table – one for each article you are review. DO NOT double space in the table

The matrix and analysis assignment to submit consists of : 1). introduction describing the search process for this topic, 2). the review table (1 for each article = 5), 3). summary analysis, 4). reference page and 5). PDF copies of the articles – list by 1st author name as attachments

Author, year; Credentials Article #1


If credentials not identified- just state here, or identify place of employment

Article Focus/Title

Research Design/Intervention (describe intervention)

Level of Evidence and model used to grade evidence

Evaluation Tool (CASP or others- identify tool used)

Sample/# of subjects, how recruited, power analysis?

Data Collection procedure

Instruments and

Reliability/validity

of instruments

Data Analysis- id statistics, LOM, findings

Results

Discussion/

Significance of findings

Reliability and Validity of study, limitations

Helpful/Reliable

Compared to other articles

Author, year; Credentials Article #1


If credentials not identified- just state here, or identify place of employment

Article Focus/Title

Research Design/Intervention (describe intervention)

Level of Evidence and model used to grade evidence

Evaluation Tool (CASP or others- identify tool used)

Sample/# of subjects, how recruited, power analysis?

Data Collection procedure

Instruments and

Reliability/validity

of instruments

Data Analysis- id statistics, LOM, findings

Results

Discussion/

Significance of findings

Reliability and Validity of study, limitations

Helpful/Reliable

Compared to other articles

Author, year; Credentials Article #1

If credentials not identified- just state here, or identify place of employment

Article Focus/Title

Research Design/Intervention (describe intervention)

Level of Evidence and model used to grade evidence

Evaluation Tool (CASP or others- identify tool used)

Sample/# of subjects, how recruited, power analysis?

Data Collection procedure

Instruments and

Reliability/validity

of instruments

Data Analysis- id statistics, LOM, findings

Results

Discussion/

Significance of findings

Reliability and Validity of study, limitations

Helpful/Reliable

Compared to other articles

Author, year; Credentials Article #1

If credentials not identified- just state here, or identify place of employment

Article Focus/Title

Research Design/Intervention (describe intervention)

Level of Evidence and model used to grade evidence

Evaluation Tool (CASP or others- identify tool used)

Sample/# of subjects, how recruited, power analysis?

Data Collection procedure

Instruments and

Reliability/validity

of instruments

Data Analysis- id statistics, LOM, findings

Results

Discussion/

Significance of findings

Reliability and Validity of study, limitations

Helpful/Reliable

Compared to other articles

Author, year; Credentials Article #1

If credentials not identified- just state here, or identify place of employment

Article Focus/Title

Research Design/Intervention (describe intervention)

Level of Evidence and model used to grade evidence

Evaluation Tool (CASP or others- identify tool used)

Sample/# of subjects, how recruited, power analysis?

Data Collection procedure

Instruments and

Reliability/validity

of instruments

Data Analysis- id statistics, LOM, findings

Results

Discussion/

Significance of findings

Reliability and Validity of study, limitations

Helpful/Reliable

Compared to other articles

Summary section:

Summary and synthesis of all 5 research articles; include level of evidence of all 5 articles, summary and comparison of outcomes of articles, include statements about the quality of the articles, instruments, what research still needs to be done on your topic, identifies gaps in care, addresses health promotion pertinent for area, analyzes interventions for populations. Did the article change your thinking about your research topic?

Reference Page – List articles used in the matrix as well as any other references you might have used in the summary.
Start this on a new page.

Fall 10.2020

1

Article Matrix and Analysis

Used with Permission

College of Health and Human Services-School of Nursing, Northern Kentucky University

DNP 816: Analysis and Application of Health Data for APRN Practice

September 20, 2020


Article Matrix and Analysis

Student Name: XXXXXX

PICO Question: In patients with hypertension (P), what is the effect of education about hypertension and medications (I) in comparison to no education (C), on compliance with following medication regimens (O) within three months of the first prescription (T).

Search process: Data base search was done using Northern Kentucky University library. Boolean phrases were used. Articles were assessed based on abstract and criteria listed below until five articles were left.

Search terms: hypertension OR high blood pressure AND education OR educational (must be in title) AND medication adherence OR medication compliance

Data bases: CINAHL Complete, Gale Academic OneFile

Total number of articles obtained from search results: N=29/ N=18 N= 33

Number of articles initially excluded based on abstract reading: N=1/N=4

Number of articles reviewed: N= 3/N=6

Number of articles excluded based on criteria: N=20/N=8 N=20

Inclusion Criteria: Full text only, English, Article within the past five years, Peer reviewed

Exclusion Criteria: Article published before 2015, No full text link, Language other than English, Not peer reviewed, Duplicate article, Non-research article

Studies included in systematic review or meta-analysis- N = 0

Author, year; Credentials Article #1

Aghakhani, N. (PhD), Parizad, N. (PhD), Soltani, B. (MSN), Khademvatan, K. (MD), & Rahimi, Z. (MSN) (2019)

Article Focus/Title

The effect of the blended education program on treatment concordance of patients with hypertension: A single-blinded randomized, control trial

Research Design/Intervention

Randomized control trial, single-blind. The intervention was education that was blended in style between face-to-face education and online education.

Level of Evidence

Level I (Dearholt et al., 2012)

Sample/# of subjects

Patients age 20-65, hypertension diagnosis, blood pressure greater than 140/90 but less than 180/110, agree to participate in the study, able to read and write, able to receive emails and texts, no severe underlying disease (ex. Kidney disease or heart problems) that could impact the study. Total number of participants – 60

Evaluation Tool

CASP for randomized control trial Score 11/11

Data Collection procedure

Questionnaires/tests were administered and scored; control and intervention group received respective education; two weeks after education programs were finished, the same questionnaire/test was administered and scored

Type of Instrument

Questionnaire/test that included demographic information and questions addressing treatment in three dimensions; dietary adherence, compliance to medication regimen, and physical activity plan, the questionnaire had thirty four, twelve, and fourteen items per category respectively.

Reliability and Validity

The questionnaires were used in another study that confirmed validity and reliability. The face and content validity was inspected and in this other study Cronbach’s alpha was used along with intraclass correlation coefficient to confirm reliability. Standard data collection was performed. Approved by IRB and informed consent was obtained prior to the beginning of the study. Adequate statistical data is presented and appropriate statistical methods were used.

Data Analysis

Data was analyzed by a researcher who was blind to the data. Descriptive statistics including mean and standard deviation were utilized and inferential statistics including
t-tests (independent and paired), Chi-squared test, and Fischer exact tests were utilized.
P-values were two tailed, and the significance level was less than 0.05.

Results/Themes

There was a statistical difference in the results of the post-test between the control and the intervention groups for dietary adherence (
P=<.001), medication adherence (
P=<.001), and physical activity (
P=.04). Medication adherence scores increased the most in the intervention group from pre-test to post-test.

Themes: blended educational program; improving medication adherence

Discussion/

Significance/

Limitations

Is an additional study that strengthens the evidence on the use of blended education programs. The use of blended education programs could promote better treatment/recommendation adherence in patients by increasing the effectiveness and desirability of the educational experience.

Limitations: The sample was a convenience sample from one geographical area. The intervention period was relatively short.

Helpful/Reliable/Compared to other art

Helpful, reliable, results were statistically significant. Clear statement of findings.

Compared to other articles:
Consistent findings in regard to education programs and increased medication adherence. Similar to Ozoemena et al. where multiple types of education were present (2019).

Author, year; Credentials, Article #2

Delavar, F., Pashaeypoor, S., & Negarandeh, R. (2020). Published in the Journal of Patient Education and Counseling. All authors were associated with Tehran University of Medical Sciences.

Article Focus/Title

The effects of self-management education tailored to health literacy on medication adherence and blood pressure control among elderly people with primary hypertension: A randomized controlled trial

Research Design/Intervention

Randomized control trial. The intervention was education based on the participants health literacy for two weeks in the form of two, thirty to forty-five minute face-to-face sessions. After the two weeks of face-to-face education, there were two weeks of fifteen minute educational sessions held via telephone call, two each week. Educational topics included hypertension, risk factors, complications, medications, side effects, management of side effects, importance of medication adherence, and doctors appointments.

Level of Evidence

Level I (Dearholt et al., 2012)

Sample/# of subjects

Elderly patients who had uncontrolled hypertension and inadequate health literacy who had been referred to the cardiovascular clinic at Fayyazbakhsh hospital in Iran. Inclusion criteria: over the age of sixty, diagnosed uncontrolled hypertension (BP over 140/90 or more), prescription for antihypertensives, can speak Persian, does not have cognitive or psychiatric disorders, not addicted to drugs, and inadequate health literacy defined by the Health Literacy for Iranian Adults (HELIA) scale (score of less than sixty six percent). Exclusion criteria: unwilling to stay in the study, development of serious health conditions that led to hospitalization or death, failure to attend one of the face to face sessions of the intervention, or failure to answer two or more of the telephone calls. Intervention group n=54, control group n=58, total sample size N=112

Evaluation Tool

CASP for randomized control trial Score 10/11

Data Collection procedure

Data was collected/measured by demographic questionnaire, eight-item Morisky Medication Adherence Scale (MMAS), and blood pressure measurement at the beginning of the study; the control and intervention groups received their respective education; then the same tools were used to collect/measure data six weeks after the intervention was completed.

Type of Instrument

Questionnaire: ten questions related to the demographic information of the participants

Eight-item MMAS: has seven yes/no style questions and one Likert scale question that is scored the same as a yes/no question. The scores can vary between zero and eight (eight indicating good medication adherence)

Blood pressure measurements: instrument used was a mercury sphygmomanometer

Reliability and Validity

The eight-item MMAS was translated to Persian and was reported to be validated by two other studies. The reliability of the eight-item MMAS was shown by an intraclass correlation coefficient (ICC) of 0.71.

The blood pressures were collected using standardized procedures. The measurement device’s validity was confirmed based on measurement precision and authenticity of manufacturer. Reliability was confirmed through repeated measurements on twenty patients, the ICC of the systolic and diastolic blood pressures were 0.81 and 0.60 respectively.

Standardized data collection procedures were used. Ethical considerations were made. Approved by the Ethics Committee of Tehran University of Medical Sciences. Written, informed consent was obtained from participants.

Data Analysis

SPSS was used to analyze data. Fischer’s exact tests and Chi-squared tests were used to assess between group comparisons for categorical variable. Independent
t-tests and Mann-Whitney U tests were used to assess between group comparisons for numerical variables. Paired
t-tests were used to assess within group differences. An analysis of covariance (ANCOVA) was performed. The significance level was less than 0.05.

Results/Themes

The systolic and diastolic blood pressures of the intervention group were significantly lower in the post-test (
p=<0.001 for both pressures). The medication adherence scores of the intervention group was significantly better than the scores of the control group in the post-test (
p=0.002). The rates of controlled/uncontrolled blood pressures after the intervention in the between group analysis was nonsignificant (
p=>0.05).

Themes: Patient education based on health literacy; improving medication adherence; uncontrolled hypertension; elderly population

Discussion/

Significance/

Limitations

Shows improvement in medication adherence after an educational program about hypertension/medications compared to basic education given when prescription is given. Identifies the benefit of using education tailored to the health literacy level of the patient.

Limitations: Limited validity because author credentials are not expressed in the article, only the Universities they are affiliated with. Risk of type II error because fifty nine participants were needed per group and that was not achieved. Researchers discuss limitations related to short intervention time and possibility of unrealistic responses from participants about adherence.

Helpful/Reliable/Compared to other art

Helpful but cautious, small sample size and no credentials of the authors contributes to lack of validity, clear statement of findings.

Compared to other articles: Consistent with Aghakhani et al. (2019); both establish that education beyond what is traditionally provided with the start of a prescription improved medication adherence.

Author, year; Credentials, Article #3

Farazian, F. (MSN), Moghadam, Z. E. (Assistant Professor), Nabavi, F. H. (Assoictate Professor), & Vashani, H. B. (Instructor) (2019). Published in the Journal of Evidence-Based Care. All are faculty at Mashhad University of Medical Sciences

Article Focus/Title

Effect of self-care education designed based on bandura’s self-efficacy model on patients with hypertension: A randomized clinical trial.

Research Design/Intervention

Randomized control trial. The intervention was one forty-five minute educational session each week for four weeks. The sessions were designed based on Bandura’s social learning theory and the content was based on Bandura’s self-efficacy theory

Level of Evidence

Level I (Dearholt et al., 2012)

Sample/# of subjects

Sample was collected using a two-stage random sampling technique. Population was the residents of Mashhad, Iran who had hypertension. Inclusion criteria: age of 40-70, living with family, resident of Mashhad, had been diagnosed with hypertension for more than one year, diagnosis confirmed by a cardiologist, active medical file at the health center, no cognitive impairment, no participation in self-care programs within the last six months. Exclusion criteria: participation in another educational program, missing more than one session, failure to participate in the post-test. Sample size n=60.

Evaluation Tool

CASP for randomized control trial Score 10/11

Data Collection procedure

All participants completed the demographic form and the questionnaire before the intervention started. Then the intervention group received their education and the control group received routine education. The questionnaire was administered again immediately after the intervention was complete and then again one month later.

Type of Instrument

Questionnaire with twenty nine questions addressing self-care for patients with high blood pressure. Topics that were a part of the questionnaire include; medication adherence, adherence to a low-salt diet, physical activity, smoking, alcohol and weight management.

Reliability and Validity

Research approved by the research ethics committee at the Mashhad University of Medical Sciences, Consent obtained, Standard data collection methods. Potential bias is evident – the researcher was the educator and also exchanged phone numbers with participants to provide additional encouragement or counseling if needed. Validity was tested by assessment of the questionnaire by ten experts according to word choice, grammar, and readability, potential for bias because all ten experts were also faculty of the same University that the researchers were affiliated with. Reliability was confirmed by measuring internal consistency, the Cronbach’s alpha was measured (
a=0.84).

Data Analysis

Data was analyzed using SSPS (version 16). CI of 95%, significance level was less than 0.05. A repeated measures ANOVA was used to test the hypotheses, Kolmogorov-Smirnov test was used to confirm the normal distribution of the quantitative data, and Shapiro-Wilk test was used to confirm the normal distribution of the qualitative data.

Results/Themes

Significant positive effect overall on self-care and self-management of hypertension. Statistically significant results in categories for medication adherence, physical activity, and weight management. Statistically non-significant results in categories for low salt diet and non-smoking. Repeated measures ANOVA was used and in regard to medication adherence showed that the effect of the group (
p=0.006) and time (
p=0.04) were both significant and their mutual effect was significant (
p=0.01). Th

Themes: self-care in patients with hypertension, educational interventions

Discussion/

Significance/

Limitations

Highlighted the use of an educational program to improve self-care in patients with hypertension. Shows that educational programs are superior to routine education that is provided at practices. Shows medication adherence and some of the other aspects of self-care are still improved one month after the intervention is finished.

Limitations: Lack of knowledge about patient’s baseline knowledge about hypertension management and the degree to which they could be influenced by other sources of information. Short time frame for follow up. Potential for bias is evident in the intervention itself and in the assessment of the validity of the questionnaire.

Helpful/Reliable/Compared to other art

Helpful but cautious, the potential bias negatively impacts the validity and the reliability of the study. Clear statement of findings. Importance of the topic was established. Rigorous data analysis.

Compared to other articles: Consistent findings in regard to education programs and increased medication adherence.

Author, year; Credentials, Article #4

Ozoemena, E. L., Iweama, C. N., Agbaje, O. S., Umoke, P. C. I., Ene, O. C., Ofili, P. C., Agu, B. N., Orisa, C. U., Agu, M., & Anthony, E. (2019), Published in the Archives of Public Health

All authors are associated with the Department of Human Kinetics and Health Education and are faculty at the University of Nigeria except B. Agu and C. Orisa. Agu is associated with the Department of Public Health and is faculty in health sciences at Madonna University Elele. Orisa is associated with the Department of Human Kinetics, Health, and Safety Education and Ignatius Ajuru University of Education.

Article Focus/Title

Effects of a health education intervention on hypertension-related knowledge, prevention and self-care practices in Nigerian retirees: A quasi-experimental study.

Research Design/Intervention

Quasi-experimental study. Intervention was a twelve-week health education course about hypertension and was based on the information-motivation-behavioral skills model. There were twelve sessions each one lasting around sixty minutes.

Level of Evidence

Level II (Dearholt et al., 2012)

Sample/# of subjects

Two-stage sampling procedure. Purposeful selection of three specific zones to enhance representativeness, then convenience sampling to acquire the 400 participants. Participants were randomly assigned to the control or intervention groups. Inclusion criteria: over or equal to the age of sixty, classified as a retiree, gave informed consent, located in the geographical zone selected. Exclusion criteria: those who were ill prior to the study, those who declined participation. Total participants = 400, each group containing 200 participants.

Evaluation Tool

JBI checklist for quasi-experimental studies score 8/9

Data Collection procedure

Pre-test post-test was used. Demographic information was collected at baseline as well. The intervention lasted twelve weeks and there was a post-test at the sixteen week mark and a follow-up one month after that. The tests were administered/facilitated by health education experts and nurses but were supervised by the principal researchers. Height and weight were taken to calculate BMI and blood pressure was taken from each participant. It is not stated exactly when these measures were taken,

Type of Instrument

Questionnaire. Included questions about demographics, hypertension knowledge, practices on prevention and self-care related to hypertension, physical activity (IPAQ-SF), the quality and pattern of sleep (PSQI), substance (smoking and alcohol) use, maintaining a healthy diet, medication adherence (MMAS-4), and home blood pressure monitoring. Face validity, content validity, and construct validity were all confirmed. Kuder-Richardson-20 coefficient was used to confirm reliability (0.72).

Reliability and Validity

Written script was used to ensure standardization of material provided to participants. Standardized data collection procedures were used. No apparent bias. Approved by the Ethical Committee on Research Projects at Enugu State Ministry of Health. Informed consent obtained. Repeated measures were performed. Appropriate statistical methods were used.

Data Analysis

Data analysis was done using SPSS (version 20). Demographic data was analyzed using independent sample
t-tests and Chi-squared test for continuous and categorical variables respectively. One-way repeated measures ANOVA was used to analyze the effect of the intervention on hypertension knowledge, self-care, and prevention. Statistical significance is
p<0.05. Data is presented in the text as well as in tables in the article. Some
P values are presented at 0.000 in the text but are clarified in the tables as 0.0001.

Results/Themes

All participants showed improved hypertension knowledge from the pre-test to post-test. The intervention group had significant (
p<0.0001) improvement from the control group. There was significant improvement between pre- and post-tests in the intervention group in the factors of self-care and prevention compared to the control group. For medication adherence
p=<0.0001.

Themes: Education program to improve hypertension knowledge, Self-care, Follow-up testing, Educating older adults

Discussion/

Significance/

Limitations

Highlighted that adequate knowledge about hypertension can help improve medication adherence and lifestyle changes. Provides an example of an effective education program that utilizes educational booklets, videos, charts, and discussion.

Limitations: Credentials of authors are not stated, only their associations/Universities. There was baseline differences in the gender and BMI categories that were statistically significant. Self-report measures were used with participants which introduce risk for bias. Potential for poor generalizability because the population was only retirees from Enugu state.

Helpful/Reliable/Compared to other art

Helpful, reliable, valid. Clear statement of findings. Importance to topic established in thorough literature review.

Compared to other articles: Consistent findings in regard to education programs and increased medication adherence.

Author, year; Credentials, Article #5

Yazdanpanah, Y. (MSC of Geriatric Nursing), Saleh Moghadam, A. R.(Assistant Professor), Mazlom, S. R. (Instructor), Ali Beigloo, R. H.( MSC of Medical Surgical Nursing), & Mohajer, S. (PhD) (2019), Published in Journal of Evidence-Based Care

Article Focus/Title

Effect of an educational program based on health belief model on medication adherence in elderly patients with hypertension.

Research Design/Intervention

Randomized control trial. The intervention was eight educational sessions about hypertension based on the Health Belief Model (HBM). Sessions were held twice a week for four weeks and lasted about an hour in length.

Level of Evidence

Level I (Dearholt et al., 2012)

Sample/# of subjects

Population was elderly patients with hypertension at health centers in the western region of Mashhad, Iran. Inclusion criteria: aged sixty years or older, able to read and write, history of office visit in the past six months, history of hypertension (self-report and health record), history of use of hypertension medication daily for at least six months, and no psychological or cognitive disorders. Exclusion criteria: declined participation, failure to attend in two or more sessions, and suffering from acute illness or hospitalization. It is not stated what method is used for collecting the sample. The separating into intervention and control groups was randomized but it is not clear as to if there was any blinding involved. Total number of participants was sixty, with thirty in each group (intervention and control).

Evaluation Tool

CASP for randomized control trial Score 10/11

Data Collection procedure

Pre-test post-test method was used. The pre-test was administered prior to the intervention, then the intervention group received their education sessions on hypertension and the control group received sessions typical to the centers the sessions were provided at. The post-test was administered after the last session.

Type of Instrument

Questionnaire including demographic information and the MMAS-8. Content validity was confirmed. Reliability was confirmed with internal consistency using Cronbach’s coefficient alpha (0.69). Reliability was also confirmed by test-retest method. The Pearson correlation coefficient was r=0.86.

Reliability and Validity

Sessions were administered by one of the researchers while the rest of the researchers supervised. Data collection was standardized. Approved by the Ethics Committee of Mashhad University of Medical Sciences. Informed consent was obtained. Ethical considerations were assessed, and the control group was provided with all of the information the intervention group received after the study was complete.

Data Analysis

Data was analyzed using SPSS (version 20). Mann-Whitney U test was used to check for significant difference between the control and intervention group (
p=0.12). Chi-squared test and Fischer’s exact test showed no significant differences between the demographics of the two groups
T-tests were utilized to assess the between-group and within-group differences. A two way ANOVA was also used but additional testing to find the specific difference was not mentioned.

Results/Themes

Overall, the education program is associated with improved the medication adherence of the intervention group. The results of the independent
t-test showed that the difference between the post-test scores of the intervention and control groups were statistically significant (
p=<0.001). The results of the paired
t-test showed that the change in mean medication adherence score of the intervention group from pre-test to post-test had significance (
p=<0.001). There was not a significant difference in the same measurement for the control group (
p=0.56). A two-way ANOVA was done and showed that there was significantly improved medication adherence according to demographic information for both groups. There was no additional explanation of follow-up testing in the article.

Themes: Education program for elderly patients with hypertension, improving medication adherence, Health-belief model

Discussion/

Significance/

Limitations

It was estimated that implementing this educational program could increase medication adherence by about fifty nine percent. This research provides additional strength to the evidence that education designed based on the HBM can help improve medication adherence.

Limitation: The method of sample collection is not clear and blinding about group assignment is not expressed. Self-report method can present the possibility for bias from participants. The sample was elderly so mental or psychological conditions that older adults experience may have had an effect on their answers.

Helpful/Reliable/Compared to other art

Helpful but cautious, the Cronbach’s coefficient alpha is 0.69 which is moderately low this indicates a lower reliability of the questionnaire and the ANOVA did not have a post hoc analysis. There was a clear statement of findings. The importance of the topic was established by a thorough literature review.

Compared to other articles: Consistent findings in regard to education programs and increased medication adherence. Similar to Farazian et al. in the way the education program is created based on a model (Bandura’s self-efficacy model and the HBM) (2019).


Summary and Analysis of Article Matrix

This review of literature provided evidence for the benefit of utilizing educational programs for patients with hypertension to improve medication adherence. Failure to adhere to a medication regimen in a patient with hypertension can increase the risk for serious health problems. Baseline (traditional) education about hypertension, medications, prevention, and self-care can be very minimal, leaving patients with a knowledge deficit that can impact their quality of life in a dramatic way such as a stroke or a heart attack.

Feedback from the studies done on using education to increase medication adherence in people with hypertensive has been helpful but there were flaws in some of the studies that could impact the results. The articles reviewed in this matrix include four level I evidence articles, and one level II evidence article. The four randomized control trials were valid according to the CASP checklist for randomized control trials and the quasi-experimental study was valid according to the JBI checklist for quasi-experimental studies. All of the studies provided a review that established importance for this topic. Three of the studies had aspects that may hurt the reliability and/or validity of the study, but their results were still consistent with other research. The studies all presented data collection that was standard, and the findings seemed to be interpreted appropriately.

The studies all are consistent in their statement that after an educational program for patients with hypertension the rate of medication adherence increases. There still needs to be more research done on the long term effects of these educational programs. Three of studies included no follow-up after the post-test. Ozoemena et al. and Farazian et al. both completed a follow-up one month after the intervention was complete (2019;2019). This was still considered a short time frame and was considered a limitation of the research (Farazian et al., 2019). Knowing the long term effects would add additional strength to the evidence that is established on the benefit of educational programs. The variety in the focus of the educational programs shows that whether the educational program is focused on a model such as the HBM in the article by Yazdanpanah et al. or if it focuses on the way education is presented, such as in the article by Aghakhani et al. who uses blended education, the results still show an improvement in medication adherence (2019; 2019). The studies reviewed utilized pre-test/post-test methods to assess medication adherence. There is a gap in care related to this topic because more emphasis is not being placed on education to improve medication adherence. These are only a few of the many articles that present evidence supporting additional education to hypertension patients. Physicians and their extenders may be prescribing the medications and they should provide education. However, this topic identifies an opportunity for the utilization of nurses for education (Aghakhani et al., 2019; Delavar et al., 2020). An educational program is associated with health promotion in the aspects of prevention and to guide people into taking charge of their own health. While all of these studies were done with small to moderate sample sizes, the same programs can be applied to larger populations. This indicates the need to research what the most beneficial class size would be, but since most of the research has been focused on group classes and not individual meetings, it provides an easier transition from sample sized interventions to entire populations.

In summary, all articles reviewed came to the same conclusion; that an educational program that focuses on hypertension and medications increases medication adherence in the participants. The long term effects need additional research, but overall the studies reviewed in the matrix addressed the original PICOT question. No changes to the PICOT question are indicated at this time.


References

Aghakhani, N., Parizad, N., Soltani, B., Khademvatan, K., & Rahimi, Z. (2019). The effect of the blended education program on treatment concordance of patients with hypertension: A single-blind randomized, controlled trial.
Journal of Vascular Nursing, 37(4), 250–256. https://doi.org/10.1016/j.jvn.2019.08.001

Dearholt, S., Dang, D., & Sigma Theta Tau International. (2012).
Johns hopkins nursing evidence-based practice: Models and guidelines. https://libguides.ohsu.edu/ld.php?content_id=16277844

Delavar, F., Pashaeypoor, S., & Negarandeh, R. (2020). The effects of self-management education tailored to health literacy on medication adherence and blood pressure control among elderly people with primary hypertension: A randomized controlled trial.
Patient Education and Counseling, 103(2), 336–342. https://doi.org/10.1016/j.pec.2019.08.028

Farazian, F., Moghadam, Z. E., Nabavi, F. H., & Vashani, H. B. (2019). Effect of self-care education designed based on bandura’s self-efficacy model on patients with hypertension: A randomized clinical trial.
Journal of Evidence-Based Care, 9(2), 43–52. doi: 10.22038/ebcj.2019.36466.1944

Ozoemena, E. L., Iweama, C. N., Agbaje, O. S., Umoke, P. C. I., Ene, O. C., Ofili, P. C., Agu, B. N., Orisa, C. U., Agu, M., & Anthony, E. (2019). Effects of a health education intervention on hypertension-related knowledge, prevention and self-care practices in Nigerian retirees: A quasi-experimental study.
Archives of Public Health, 77(1), N.PAG. https://doi.org/10.1186/s13690-019-0349-x

Yazdanpanah, Y., Saleh Moghadam, A. R., Mazlom, S. R., Ali Beigloo, R. H., & Mohajer, S. (2019). Effect of an educational program based on health belief model on medication adherence in elderly patients with hypertension.
Journal of Evidence-Based Care, 9(1), 52–62. https://doi.org/10.22038/ebcj.2019.35215.1895

2

Article Matrix and Analysis

Student Name: Used with Permission

College of Health and Human Services-School of Nursing, Northern Kentucky University

DNP 816: Analysis & Application of Health Data for ANP

Dr. Faculty Name

September 20, 2020


Article Matrix and Analysis

Student Name: XXXXX

PICO Question: In adult patients with diabetes (P), how does strict glucose monitoring and treatment regimens (I), when compared to decreased compliance to home treatment i.e. routine glucose monitoring, medication adherence and nutritional changes (C), affect and influence patient quality of life through improvement in treatment education and expectations (O) over one year (T)?

Search process: I utilized the NKU online library to search for articles. The data bases included CINHAL and MEDLINE PLUS. These are peer-reviewed, current and reputable search engines that provided the most accurate and reliable articles for the assignment.

Search terms: Patients with diabetes AND adherence OR compliance to treatment OR management; noncompliance OR nonadherence in diabetic treatment AND effects on quality of life; compliance to diabetic treatment AND management AND importance OR significance

Data bases: CINHAL COMPLETE and MEDLINE

Total number of articles obtained from search results: N=11,568

Number of articles initially excluded based on abstract reading: N=37 (out of first 100 relevant)

Number of articles reviewed: N=9

Number of articles excluded based on criteria: N=7,868

Inclusion Criteria: Must be in English, full text only, research or review article, published in 2013 or later, peer-reviewed articles, include adults as the subject, be discovered in a reputable database

Exclusion Criteria: Article published prior to 2013, language other than English, no full text link, obtained from a non-reputable source

Number of systematic review or meta-analysis used in Matrix: N =0

Repeat this table – one for each article you are reviewing.

The matrix and analysis assignment to submit consists of : 1). introduction describing the search process for this topic, 2). the review table (1 for each article = 5), 3). summary analysis, 4). reference page and 5). 5 PDF copies of the articles

Author, year; Credentials Article #1

Mirahmadizadeh, A. Delam, H. Seif, M. et al. (2019). All authors had a doctorate degree with one holding a masters. Published in the International Journal of Molecular Sciences.

Article Focus/Title

Factors Affecting Insulin Compliance in Patients with Type 2 Diabetes in South Iran, 2017: We Are Faced with Insulin Phobia

Research Design/Intervention

Cross-sectional study : No intervention

Level of Evidence and model used to grade evidence

Level IV :Johns Hopkins

Sample/# of subjects, how recruited

Adults aged 30 years or greater who attended one of twelve diabetic clinics in Shiraz, Iran were evaluated once written consent was obtained. Sample N= 457 patients among 8376 diabetic patients in total. The sample was obtained via convenience sampling. There was 95% confidence interval, error of 0.05 and design effect of 1.7.

Evaluation Tool (CASP or others- identify tool used)

JBI Critical Appraisal Checklist for Analytical Cross-Sectional Studies, Score 8/8

Data Collection procedure

Collected via a questionnaire that was administered by a trained nurse who completed face-to-face interviews, obtained the responses from the participants, and then completed a verify and read-back with the patients to confirm accuracy of responses.

Type of Instrument, reliability and validity of instrument

Questionnaire broken into two parts. Part one encompassed demographic information. Part two included 28 questions that assessed the patients’ reasons for noncompliance to treatment. This second component held two dimensions that focused on noncompliance related to use of insulin as well as noncompliance related to socioeconomic factors.

Reliability and Validity of study, limitations

Validity determined through a review of diabetic experts, epidemiologists, and five professors at multiple stages throughout the collection process. All experts approved the questionnaire. Reliability and internal consistency of the questionnaire was evaluated through test-retest as well as Chronbach’s alpha. The reliability was determined to be 0.91 and the Chronbach alpha was calculated to be 0.92. Study approval from the Ethics Committee of Shiraz. Written informed consent obtained. Statistical data listed in percentages and standard deviations. Appropriate statistical and standardized methods.

Data Analysis – identify LOM, Statistics, findings

T-test was implemented to analyze data. A logistic regression model was utilized to evaluate the relationship between the demographic variables and the clinical variables. Analysis of this model was completed through the Wald method. In addition, the statistical software used to analyze the data was the SPSS version 19. Statistical significance level was 0.05.

Results

N+308 (67.4% of participants) were women. Mean age= 55.16 +/- 5.79 years mean A1C level was 8.92%. 60.2% of participants were noncompliant with insulin administration. Factors such as age, gender, marital status, educational level, residence, insurance, diet, and physical activity were all variables associated with noncompliance as evidenced by statistical significance P<0.001. The duration that the participants had with diabetes was also associated with compliance with a P value of 0.025.

Discussion/

Significance/

Limitations

Findings demonstrate multiple variables and the association with decreased compliance to diabetic treatment. Additionally, the findings support the need for increased educational and financial support for diabetic patients.

Limitations: data was only obtained from public diabetic centers. There was no inclusion of private practices. Convenience sampling implemented. Limits generalizability. The cross-sectional design of the study as limits causation of results.

Helpful/Reliable/Compared to other art

Helpful. Statistically significant results. Appropriate analysis and interpretation of results. Clear findings. Addressed limitations and gaps in research for future studies. This study was Other literature makes note of noncompliance to diabetic treatment. In this present study, <40% of the participants were willing to complete insulin therapy. A study in existing literature was mentioned to state that 48% of the population was willing to start insulin therapy. Discrepancies in result could be in relation to sociocultural and financial differences among the different nations. Other studies reported determine compliances to insulin treatment and other modalities comes from experience. These findings were also similar in the present study. Fear, concerns, and adverse effects were lessened in participants who had initiated and implemented insulin therapy in the past.

Author, year; Credentials, Article #2

Sahin, G. Rizalar, S. (2018). Both authors held a doctorate level degree. Published in International Journal of Caring Sciences.

Article Focus/Title

Relationship between Nutritional Status, Treatment and Care Attitude in Diabetic Individuals

Research Design/Intervention

Descriptive and cross-sectional study- no intervention used

Level of Evidence and model used to grade evidence

Level IV:Johns Hopkins

Sample/# of subjects

Volunteer individuals with type 2 diabetes N=100 who applied to the Samsun Education Research Hospital. Participants must have had diabetes for at least one year.

Evaluation Tool (CASP or others- identify tool used)

JBI Critical Appraisal Checklist for Analytical Cross-Sectional Studies, Score 6/8

Data Collection procedure

Questionnaire prepared by researcher and Diabetes Attitude Scale developed by National Diabetes Commission. No other information included regarding collection process.

Type of Instrument reliability and validity of instruments

Questionnaire survey and Diabetes Attitude Scale. 33 questions identifying the sociodemographic characteristics, nutritional status and diabetes characteristics of participants were collected via a survey form. A Diabetes Attitude Scale was also included. This Likert-type scoring system ranged from 1-5 with a score >3 equivalent to a positive attitude and a score of <3 equivalent to a negative attitude.

Reliability and Validity of study, limitations

Validity and reliability were performed by Ozcan. Obtained ethics committee approval. Consent obtained from volunteer individuals. All results listed as percentages or standard deviations. Information and statistical methods appropriate.

Data Analysis: id statistics, LOM, findings

Analysis completed in SPSS. Descriptive statistics were shown as mean +/- standard deviation for variables that demonstrated normal distribution. One -way ANOVA for distribution of normal groups. P value results < 0.05 were deemed statistically significant.

Results

Mean age of participants was 59.54 +/- 12.04 years. Participants deemed “slightly obese” was 21% and those considered “obese” was 64%. Participants who follow the prescribed dietary lifestyle was 9% and 49% demonstrated partial adherence. The Diabetic Attitude

Scale (DAS) score was not significantly affected by the demographic variables of gender, marital status, BMI, education, occupation, and diet compliance. Statistical significance (P=0.000) was reported between compliance and income levels of the participants. Statitical significance was also noted between the attitude scale and physical exercise with a P value of 0.032.

Discussion/

Significance/

Limitations

Many socioeconomic factors, except for income, were significant to the DAS Score. Physical exercise and DAS score showed statistical significance. No significant difference was discovered between duration of the disease, fasting glucose level, A1C, and the type of treatment implemented. Diet compliance is low, and education is inadequate regarding treatment. Limitations: small sample size, no mention of sampling technique, poor description of data collection process and analyzation.

Helpful/Reliable/Compared to other art

Helpful but cautions warranted. Sampling method, effect size and power analysis are not mentioned. Small sample size. However, statistically significant results, appropriate use of tables and interpretation of results. Clear findings provided. Other literature shows similar results that concluded that education and income levels regarding diabetes, treatment, and importance of compliance are generally low.

Author, year; Credentials, Article #3

Furthauer, J. Flamm, M. and Sonnichsen, A. (2013). All authors have some relation to an Institute of General Practice. Published through Biomed Central.

Article Focus/Title

Patient and Physician Related Factors of Adherence to Evidence Based Guidelines in Diabetes Mellitus Type 2, Cardiovascular Disease and Prevention: A Cross- Sectional Study

Research Design/Intervention- describe intervention

Cross-sectional study no intervention done

Level of Evidence

Level IV

Sample/# of subjects, how recruited.

N=58 general practitioners (GP) were included in the study. Initially 124 were randomly selected, but only 58 responded. They were under the agreement to work 58 straight consecutive days from January to April 2011. All consecutive patients were included from that day as long as the inclusion criteria were met. N=501 patients were included in the study. 526 patients were eligible to participate but only 95.3% gave consent to be included. The patient sample represents a random consecutive selection.

Evaluation Tool (CASP or other tool used)

JBI Critical Appraisal Checklist for Analytical Cross-Sectional Studies, Score 7.5/8

Data Collection procedure

A structured case report form (CRF) was completed on each patient to collect demographic data regarding chronic diseases, medication and past medical history. Nine quality indicators were implemented after the CRF to determine the guideline-adherence of treatment for the chronic target diseases. Interviews of the patients and GPs occurred independently of one another, again using the structured CRF.

Type of Instrument, reliability and validity of instruments used

Interview, quality (QI) and structured case report form (CRF)

Reliability and Validity of study, limitations

Informed consent was obtained from both participant party members. The study was compliant with the Declaration of Helsinki and with the Austrian data protection legislation. Ethics approval was obtained from the ethics committee and federal state of Salzburg. Randomization utilized in sampling. Appropriate statistical methods implemented.

Data Analysis – identify statistics, LOM, findings

All data obtained from the CRFs was recorded and transferred to IBM SPSS Statistics v. 19.0 for analysis. Response rate documented in percentages.

Results

46.8% of GP responded to participate in the study. In these 58 GP surgeries, 95.3% of patients gave consent to be included corresponding to a mean of 8.6+/-5.3 standard deviation patients per surgery. 1224 total QIs were analyzed, 174 of which had type II diabetes. The remaining included other chronic disease evaluated in the study. 64.4% of patients are unsure why they are taking a particular drug as treatment for their chronic conditions, 20.0% state they do not need treatment, 3.4% state they do not want treatment, 5.4% report an adverse drug event to the treatment, and 6.8% falsely report discontinuation of treatment. From the GP perspective, nonadherence is related to contraindications, falsely discontinued, noncompliance, adverse drug even, avoidance of polypharmacy and falsely not indicated. 72.2% of all nonadherence to chronic treatment is as a result of the GPs not initiating/thoroughly explaining the therapy. A large knowledge gap between physicians and patients regarding treatment regimens and expectations was determined.

Discussion/Significance/ Limitations

More than half of participants are unaware why they are prescribed particular drugs making physicians responsible for nonadherence to treatment for diabetes and other chronic conditions alike. It was noted that physicians ignore guideline recommendations for treatment most frequent because they falsely assumed a particular prescription or treatment was not necessary or indicated. Education plays an essential role to treatment compliance. Lack of knowledge, communication and, awareness about proper treatment guidelines and expectations of the patient were noted in this study and indicative for implementation of better health education practices. Limitations: reported small sample size, 50% response rate of GPs, single visitation day for each GP and only interviewing patients on that particular day. Possible biases to result from this. Narrow set of only nine quality indicators implemented in the study to judge adherence to treatments.

Helpful/Reliable/Compared to other art

Helpful. Randomization in sample selection, valid and reliable. Results were clear and interpreted appropriated. Proper use of tables throughout the study. Rigorous analysis of data. Adds to evidence base and provided enhancement on need for further education. When compared to other literature, findings are consistent and enhances the evidence.

Author, year; Credentials, Article #4

Andradre-Dias, J. Rodriguez, R. Sales, Z. et al. (2016). All authors are nurses, many of which have obtained a PhD or are a PhD student.

Article Focus/Title

Diabetes Mellitus Clients’ Conceptions About the Treatment

Research Design/Intervention (describe intervention)

Descriptive and exploratory study with a qualitative approach – no intervention

Level of Evidence and model used to grade evidence

Level IV Johns Hopkins

Sample/# of subjects. How recruited?

Registered users in diabetes service, N=11. Obtained through convenience sampling of a project presentation held in a basic health unit.

Evaluation Tool (CASP or other tools used)

CASP Qualitative checklist, Score 7/10

Data Collection procedure

Information was obtained through a semi-structured interview and voice recorder. No mention of who collected the data or how despite brief information of tool used.

Type of Instrument, reliability and validity of instrument

Semi-structured interview. Voice recorder also utilized.

Reliability and Validity of study, limitations

Research approved by Research Ethics Committee, participants informed and aware of these, objectives, rationale, risks and benefits. All participants signed consent form. Standard and synonymous method of data collection. Appropriate literature review provided.

Data Analysis – identify statistics, LOM, findings

Data was obtained from the eleven interviews with a deeper reading occurring later. This information was coded and grouped semantically after collection from the interviews. This allowed for the identification of five categories: correct use of hypoglycemic drugs, adoption of proper diet, physical activities, use of phytotherapy and adoption of preventative attitude.

Results

Participants are aware of DM treatment, but lack substantiative knowledge and lack applicable treatment to their daily routines often leading to nonadherence. It was determined that there is a greater need for health education on the subject in order to overcome knowledge deficiencies related to proper DM treatment. Five categorical themes derived.

Discussion/Significance/ Limitations

Lack of essential knowledge needed for patients to be compliant. Limitations: difficult to generalize as sample is small, no mention of effect size. Poor mention of data collection process and analyzation of results.

Helpful/Reliable/Compared to other art

Helpful, but cautious. This study was a small sample size. Missing information regarding thorough data collection and analysis process of said data. However, thorough and clear representation of discussion section and five categorical themes identified. Interpretation of these results are appropriate. Sahin, G and Rizalar, S. (2018) support the notion that compliance to the five categorical themes are essential to diabetic management and improved quality of life.

Author, year; Credentials, Article #5

Gimenes Faria, H. T., Luchetti Rodrigues, F. F., Zanetti, M. L., et al. (2013). All authors are associated with two different universities in Brazil. It is unknown of further credentialing or roles at said universities.

Article Focus/Title

Factors Associated with Adherence to Treatment of Patients with Diabetes Mellitus

Research Design/Intervention (describe intervention)

Cross-sectional Study

Level of Evidence and model used to grade evidence

Level IV

Sample/# of subjects, how recruited

N=357 out of a population of 1,406 individuals with type 2 diabetes mellitus from 17 primary care institutions in brazil. All participants were greater than 18 years of age, used oral antidiabetic therapies, had cognitive and hearing capacities and were a patient in the institution of study.

Evaluation Tool (CASP or others -identify tool used)

JBI Critical Appraisal Checklist for Analytical Cross-Sectional Studies, Score 6/8

Data Collection procedure

Data was collected between February and December 2010 by trained researchers. Data collection was broken into several stages. Structured interviews were obtained in the participant homes and then a second stage was completed within the 17 health institutions.

Type of Instrument, reliability and validity of instruments

Four questionnaires. The first questionnaire evaluated demographic/clinical/and metabolic variables. The second was titled Measurements of Treatment Adherence (MTA) which assessed patient behaviors related to treatment drug use. A food consumption frequency questionnaire and an international physical activity questionnaire.

Reliability and Validity of study, limitations

Study was compliant with national and international standards of ethics. Consent and inclusion criteria were implemented prior to data collection. Standard data collection used as well as appropriate statistical methods.

Data Analysis – identify statistics, LOM, findings

Descriptive statistics were used to analyze data. This form of analysis assisted in the determination of the prevalence of adherence to treatment and the characterization of the first questionnaire implemented. The MTA was analyzed on a 6-point ordinal scale. Scores greater than or equal to 5 were indicative of adherent patients and scores less than 5 resembled nonadherent patients. Patients who were adherent based off of the food consumption questionnaire attended 3/6 nutritional recommendations. To analyze the final questionnaire regarding physical activity, those considered adherent were participants who were classified as moderately to very active. Nonadherent patients were classified as sedentary and inactive. Fisher’s exact test was implemented to analyze adherence with the demographic variables. Crude Odd’s ratio was calculated with a confidence interval of 95%. Statistical analysis was performed with SAS 9.0 statistical software. Values of P <0.05 are considered statistically significant.

Results

Mean age was 62.4 years +/- 11.8; 66.7% of the sample were female. 84.4% of the sample showed adherence to medication therapy, 58.6% to physical exercise and 3.1% to diet. Only 6 patients showed adherence to all three components. No association noted between adherence to treatment and gender, age, education, family income or time since diagnosis. A1C and physical exercise were statistically significant and showed association with adherence as evidenced by a p value of 0.036 and p=0.006 respectively.

Discussion/Significance/ Limitations

Findings showed no association between demographic variable and adherence to treatment. Exercise and A1C show showed statistically significant results in association with aadherence. Support groups and knowledge are important to achieving adherence to diabetic treatment. Limitations: cross-sectional design which prevents cause/effect relationships.

Helpful/Reliable/Compared to other art

Helpful. Appropriate data analysis with statistically significant results. Clear representation of findings. Rigor and validity established. Findings are consistent with other literature results which indicates that demographics are weak predictors of treatment adherence.


Summary of Article Matrix

It has been stated that by the year 2030, diabetes mellitus is estimated to be the seventh leading cause of death in the world (Mirahmadizadeh, A., Delam, H., Seif, M., 2019). This condition, for the most part, is highly manageable and in some cases preventable. For those who do have it, it is essential that patients are compliant and adhere to the prescribed treatment regimens to have the best quality of life. This was noted in all of the studies. The literature reviews from the studies were a mix of the different treatment modalities for diabetes mellitus and the evaluation of compliance to said treatments.

The articles used in this matrix assignment included five level IV articles, four of which were cross-sectional studies and one being a qualitative exploratory design. All articles utilized showed importance to the topic discussed. Several themes were noted either in the literature or in the gaps of literature. It was stated numerous times throughout the various studies the significance that proper patient and physician education has on treatment compliance, yet many of the studies mention the lack of education, knowledge and awareness that patients often have regarding their own treatment. The studies also supported the notion that lack of proper education was a major reason behind a patient’s noncompliance.

While research is quick to state the best treatment options to manage the disease, the studies had many gaps in their reviews that need to be further evaluated to determine the potential effect they may have. These gaps, apart from the lack of education, include limited information on social support and a patient’s mental frame of mind when it comes to treatment of their condition. Attitudes affect behavior (Sahin, G., and Rizalar, S. (2018). These two criteria may provide essential information when it comes to why a person is noncompliant. Further research should be completed evaluating these components.

It was difficult to find literature exemplifying exactly what I was searching for in relation to my PICO. I do think it would be in my best interest to change up my initial PICO question. I found it interesting that education, social support and mentality were not primary components when it came to compliance. I would like to adjust my PICO to reflect these areas. I am changing my PICO to: In patients with diabetes mellitus, how does a patient’s support system and attitude toward treatment compared to lack of support and poor mentality affect compliance and quality of life over one year? I feel that this change addresses the why factor when it comes to the compliance issue that floods our healthcare establishments and can provide the best insight into this prevalent frame of mind.

References

Andrade-Dias, J. A., Alves Rodrigues, R., Nogueira Sales, Z., Meira Oliveira, Z., & Gonçalves Nery, P. I. (2016). Diabetes mellitus clients’ conceptions about the treatment. Journal of Nursing UFPE / Journal of Nursing UFPE, 10(7), 2470–2479. https://doi.org/10.5205/reuol.9106-80230-1-SM1007201622

CASP checklist for qualitative research. (2020). CASP – Critical Appraisal Skills Programme. https://casp-uk.net/wp- content/uploads/2018/01/CASP-Qualitative-Checklist-2018.pdf

Checklist for analytical cross-sectional studies. (2020). Joanna Briggs Institute. https://joannabriggs.org/sites/default/files/2020- 08/Checklist_for_Analytical_Cross_Sectional_Studies.pdf

Fürthauer, J., Flamm, M., & Sönnichsen, A. (2013). Patient and physician related factors of adherence to evidence-based guidelines in diabetes mellitus type 2, cardiovascular disease and prevention: a cross sectional study. BMC Family Practice, 14, 47.
https://doi- org.northernkentuckyuniversity.idm.oclc.org/10.1186/1471-2296-14-47

Gimenes Faria, H. T., Luchetti Rodrigues, F. F., Zanetti, M. L., de Araújo, M. F. M., & Coelho Damasceno, M. M. (2013). Factors associated with adherence to treatment of patients with diabetes mellitus. Acta Paulista de Enfermagem, 26(3), 231–237.

JBI levels of evidence. (2013). Joanna Briggs Institute. 
https://joannabriggs.org/sites/default/files/2019-05/JBI-Levels-of- evidence_2014_0.pdf

Mirahmadizadeh, A., Delam, H., Seif, M., Banihashemi, S. A., & Tabatabaee, H. (2019). Factors Affecting Insulin Compliance in Patients with Type 2 Diabetes in South Iran, 2017: We Are Faced with Insulin Phobia. Iranian Journal of Medical Sciences, 44(3), 204–213.

Sahin, G., & Rizalar, S. (2018). Relationship between nutritional status, treatment and care attitude in diabetic individuals. International Journal of Caring Sciences, 11(3), 1557–1565.

Summer 2019

DNP 816 Matrix and Summary Rubric

Element

7.5 points

5 points

3.5 points

2 points

Source information and quality

(7.5 points)

All key elements are present: Author credentials listed, article is less than 5 years old, and publication is peer reviewed/

scholarly, article is based on research and relates to the chosen topic of concern; is a primary source

Two key elements are present: Author credentials listed, article is less than 5 years old, ad publication is peer reviewed/

scholarly, article is based on research and relates to the chosen topic of concern but is a secondary source

One key element is listed: Author credentials listed, article is less than 5 years old, ad publication is peer reviewed/

scholarly, the articles only partially relate to the chosen topic of concern or is a secondary source

Missing key elements: Author credentials listed, article is more than 5 years old, and publication is peer reviewed/

scholarly, the article has little or nothing to do with the topic of concern and is a secondary source. The article is not research.

Research Design and interventions described (.7.5 points)

See Polit/Beck pp. 18, 210), (pp. 17, 201 in 11th ed.)

All key elements are present: appropriate research design identified, thorough description of intervention, justification for not using a different research design, longitudinal or prospective, or causal intent. Identifies IV and DV if appropriate

Elements are covered but not in enough depth: appropriate research design identified, thorough description of intervention, justification for not using a different research design, longitudinal or prospective, or causal intent. Identifies IV and DV

Missing elements in this category- research design or intervention: appropriate research design identified, thorough description of intervention, justification for not using a different research design, longitudinal or prospective, or causal intent. Identifies IV and DV

Missing key elements: does not identify the correct research design, no description of the intervention (if present), does not identify IV or DV (if appropriate)

Level of Evidence and model used to grade evidence and Evaluation tool used (CASP or others) (7.5 points)

See Polit/Beck p. 35 (p. 36-37 in 11th ed)

Key elements addressed: What was the strength of the evidence in support of your research topic- what model was used to grade the evidence? What evaluation tool was used to assess the evidence?

Key elements are not well described but are present:

Strength of evidence, model used to grade the evidence and evaluation tool used.

Missing elements in this category: Includes some information but it missing content related to grading the evidence, model use or evaluation tool.

Does not include the level of evidence and model used to grade it. Does not use an evaluation tool to assess design

Sample/# of participants, how recruited, power analysis? Data Collection procedure (7.5 points)

See p. 263 in Polit/Beck (p. 274 in 11th Ed)

Key elements addressed: was population identified, were sample procedures described? What type sampling plan as used? How were people recruited? Was there a power analysis? Was sample size large enough? All elements thoroughly addressed.

Key elements are not well described but are present: was population identified, were sample procedures described? What type sampling plan as used? How were people recruited? Was there a power analysis? Was sample size large enough?

Missing or superficial information: was population identified, were sample procedures described? What type sampling plan as used? How were people recruited? Was there a power analysis? Was sample size large enough?

Does not include a discussion of the participants, how they were recruited, power analysis information, sample size adequacy. Limited information included.

Instruments and

Reliability/validity

of instruments (7.5 points)

See p. 325 in Polit/Beck (p. 336 in 11th Ed).

Key Elements addressed: includes a complete and thorough discussion of the instruments used, types of questions, reliability- (Cronbach alpha) and validity, LOM

Key elements are not well described but are present: includes some discussion of the instruments used, types of questions, reliability- (Cronbach alpha) and validity, LOM

Key elements are missing or are very superficial: includes a complete discussion of the instruments used, types of questions, reliability (Cronbach alpha) and validity, LOM

Does not include information concerning reliability or validity of instruments.

Data Analysis-

identify statistics,

LOM, findings,

Results (7.5

points)

See pp. 371, 399

in Polit

/Beck (pp. 381,

408 in 11th

Ed)

Key elements addressed thoroughly: Was level of measurement identified? Were inferential stats used? Were tests parametric or nonparametric- why used? Were there significant results? Was there an appropriate amount of statistics info reported? Were all important results discussed?

Key elements are not well described but are present:

Was level of measurement identified? Were inferential stats used? Were tests parametric or nonparametric- why used? Were there significant results? Was there an appropriate amount of statistics info reported? Were all important results discussed?

Key elements are missing or very superficial discussion: Was level of measurement identified? Were inferential stats used? Were tests parametric or nonparametric- why used? Were there significant results? Was there an appropriate amount of statistics info reported? Were all important results discussed?

Missing information from discussion of data analysis: does not identify statistics used, no LOM, findings or results

Discussion/

Significance of findings, Reliability and Validity of study, limitations (7.5 points)

See p. 457 in Polit/Beck (p. 465 in 11th ed)

Key elements addressed thoroughly: Was interpretation appropriate? Were limitations identified? Addressed study implications for clinical practice, did they make specific recommendations or miss important implications? Did research address clinical significance? Did they address generalizability?

Key elements are not well described but are present: interpretation, limitations, implications for clinical practice, clinical significance, generalizability

Key elements superficial : interpretation, limitations, implications for clinical practice, clinical significance, generalizability

Missing information regarding: interpretation, limitations, implications for clinical practice, clinical significance, generalizability

Analysis, Helpful/Reliable

Compared to other articles (7.5 points)

Key elements answered thoughtfully: is the information biased or objective, useful and reliable or not? How does the source compare with other reviewed articles? How is this information similar or different from other articles you have read?

Was the information helpful? How?

Key elements are not well described but are present: is the information biased or objective, useful and reliable or not? How does the source compare with other reviewed articles? How is this information similar or different from other articles you have read? Was the information helpful? How?

Key elements are present but superficial: is the information biased or objective, useful and reliable or not? How does the source compare with other reviewed articles? How is this information similar or different from other articles you have read? Was the information helpful? How?

Missing key elements: is the information biased or objective, useful and reliable or not? How does the source compare with other reviewed articles? How is this information similar or different from other articles you have read? Was the information helpful? How?

Summary

(20 points)

20 points

14 points

7.5 points

0 points

Summary

Overall Synthesis of all 5 research articles

Use Polit Box20.1, p. 457 as guide , (11th ed Box 21.1, p. 465) (20 points)

Key Elements:

Thorough and complete discussion about the quality of the articles (level of evidence), overall findings, what research still needs to be done on your topic, identifies gaps in care, addresses health promotion pertinent for area, analyzes interventions for populations. Did the article change your thinking about your research topic?

Discusses each article individually, with some evaluation of quality and needed research. Did the article change your thinking about your research topic? Addresses a gap in care of population, few health promotion or prevention issues, few interventions for population

Includes most of the articles, spotty evaluation of the articles, no research identified, limited discussion if view changed on the topic.

Limited analysis/summary with focus on gaps identified, health prevention/promotion or interventions

Does not include the summary of all 5 articles, no evaluation of quality of the articles, no needed research identified, limited discussion addressing gaps in care, health promotion/prevention or interventions. Did not address if view on topic has changed

SCHOLARLY WRITING

(20 points)

10 points (each category below is worth 10)

7.5 points

(each below are worth 7.5 pts in this category)

5 points

(each below are worth 5 pts in this category)

0 points

(each below are worth 0 pts in this category)

Writing quality (10 points)

No grammatical, spelling or punctuation errors. Succinct

Almost no grammatical, spelling or punctuation errors. Nearly succinct

A few grammatical spelling or punctuation errors.

Many grammatical, spelling or punctuation errors. Too brief or not succinct

APA Format (10 points)

Source is consistently documented in APA format

Source is accurately documented but a few minor errors noted

Multiple errors in accuracy and APA format.

Sources are neither accurately documented nor in APA Format

TOTAL Points= 100

10.2020

Received: 11 December 2020 – Revised: 14 January 2021 – Accepted: 22 January 2021

DOI: 10.1002/pon.5634

OR I G I NA L AR T I C L E

Characteristics of cancer patients who died by suicide: A
quantitative study of 15‐year coronial records

Vera Y. Men1 | Clifton R. Emery1 | Paul S. F. Yip1,2

1Department of Social Work and Social

Administration, The University of Hong Kong,

Pokfulam, Hong Kong

2Hong Kong Jockey Club Centre for Suicide

Research and Prevention, The University of

Hong Kong, Pokfulam, Hong Kong

Correspondence

Paul S. F. Yip, Centre for Suicide Research and

Prevention, The University of Hong Kong,

Pokfulam, Hong Kong.

Email: [email protected]

Funding information

Li Ka Shing Foundation, Grant/Award Number:

AR180055; Hong Kong Research Grants

Council General Research Fund (GRF), Grant/

Award Number: 17103620

Abstract

Objective: Cancer patients have elevated suicide risk compared to the general

population. However, little is known about the characteristics of cancer patients

who have died by suicide. The objectives of the study were to compare the char-

acteristics of suicide cases with, and without cancer, and determine whether age

was associated with differences in characteristics.

Methods: A total of 14,446 suicide cases between 2003 and 2017 in Hong Kong

were identified using Coroner’s Court reports. Cases were grouped by cancer sta-

tus, based on medical history in the reports. Information extracted from the reports

included sociodemographic variables and detailed descriptions of the suicide event.

Univariate analyses and overall and subgroup multiple logistic regressions were

performed to compare characteristics between the two groups.

Results: Of the 14,446 suicide cases, 1,461 (10.11%) had a cancer history.

Compared to noncancer cases, cancer patients were generally older and less likely

to live alone; more likely to use violent methods; less likely to have histories of

physical and psychiatric problems; and more likely to communicate about their

suicidal intent before death. Age was significantly associated with differences be-

tween cancer and noncancer cases.

Conclusions: Cancer suicide cases have different characteristics from noncancer

cases. Mental health screening may not be sufficient for suicide prevention among

cancer patients. Healthcare professionals and caregivers should be aware of cancer

patients’ suicide risk, even when there are no signs of psychiatric disturbance.

K E YWORD S

cancer, mental health, psycho‐oncology, suicide prevention, terminal care

1 | BACKGROUND

Cancer casts a huge disease burden around the world. From 2006 to

2016, the global incidence of cancer increased by 28%.1 As the

second leading cause of death globally, cancer claimed 9.6 million

lives in 2018.2 Besides treating the physical symptoms of cancer

patients, optimizing their mental health and quality of life is also

important. Public Health England3 suggested that a cancer diagnosis

negatively impacts patients’ quality of life, potentially resulting in

elevated suicide risk. Recent epidemiological and clinical research has

indicated that the incidence of suicide in cancer patients was signif-

icantly higher than in the general population.4‐7 Factors associated

with cancer‐related suicides included the time period after diagnosis,
age at diagnosis,8 gender,7,9 mental and physical health status,10,11

income,12 employment status, marital status, education level,4 and

religion.13

In Hong Kong, publicity about suicides by terminally ill people

has raised awareness due to increasing cancer incidence and the

Psycho‐Oncology. 2021;30:1051–1058. wileyonlinelibrary.com/journal/pon © 2021 John Wiley & Sons Ltd. – 1051

health challenges of growing older.14 Currently, medically based

suicide prevention strategies focus mainly on mental health screening

and depression management.15,16 Despite an association between

suicide and cancer, little is known about the characteristics of cancer‐
related suicide cases, and whether these differ from noncancer cases.

Research has suggested that cancer patients who killed themselves

were typically older and less likely to have mental health problems

compared to noncancer cases.17,18 If this situation is the same in

Hong Kong, then current suicide screening and prevention strategies

may be ineffective in identifying and managing cancer patients

considering suicide.

This study aimed to explore the influence of cancer status, and

age, on characteristics of people who died from suicide.

2 | METHODS

The Hong Kong Coroner’s Court Suicide reports 2003–2017 were

used for suicide case identification. In Hong Kong, when a suspicious

death case is reported, the Coroner’s Court will collect information

on the deceased including sociodemographic background, history of

physical and mental illnesses, police report description of the suicide

act, family and friend witness reports and interviews, and/or hospital

records. The case will be certified as suicide only after the Coroner’s

Court’s investigates and concludes that the death is self‐inflicted
with an intention to end his/her life beyond reasonable doubt. Ethics

approval has been obtained from the Hong Kong Coroner’s Court

(JUD CC 6‐5/1).
Study data was extracted from the reports, comprising cancer

status, sociodemographic information (age, gender, marital status,

employment status, living conditions); suicide descriptions (place and

method of the suicide); medical history (physical or psychiatric prob-

lems); family engagement (living alone or with family); and whether

the deceased had communicated with others about suicidal intent.

The suicide methods were further classified as violent (hanging, use of

firearms, jumping, deep‐cutting, car or railway crash, burning, elec-

trocution, suffocation) or nonviolent (poisoning and drug overdose).19

2.1 | Statistical methods

Independent t‐tests or chi‐square tests were used to examine differ-
ences in study variables of interest between the cancer and noncancer

groups. The number of cases varied for each analysis because of

missing information in the original Coroner’s reports. Multiple logistic

regressions were performed on the overall study sample, and within

age strata (age subgroups of 10–44 years, 45–64 years, 65+ years),

applying the significant predictor variables from univariate analyses.

Adjusted odds ratios (aORs) (95% confidence interval [95% CI]) were

reported. The significance level was set at p < 0.05 and all analyses

were performed with SAS (version 9.4; SAS Institute, Inc., Cary, NC).

3 | RESULTS

The total number of verified suicide cases within the 15‐year study
period was 14,446, of which 1,461 (10.1%) reported having cancer.

Table 1 summarizes the comparisons between cancer and noncancer

suicide cases. Cancer patients (mean age = 65.7 years, SD = 13.9)

were significantly older (mean age = 49.9 years, SD = 19.5)

(p < 0.001); and had a higher proportion of males than the noncancer

group (p < 0.05).

Differences in characteristics between cancer and noncancer

suicide cases were observed. Compared with noncancer cases,

cancer patients were more likely to be married or widowed

(p < 0.0001); less likely to be employed and more likely to be

retired (p < 0.0001); more likely to live in public housing and less

likely to live alone (p < 0.001). Among all cancer cases, jumping was

the most common method (54.3%), followed by hanging (27.5%),

and carbon monoxide (CO) poisoning (6.6%). Compared to non-

cancer cases, cancer patients were more likely to use violent

methods with high lethality (p < 0.0001); more likely to communi-

cate about their suicidal intent prior to actioning it (p < 0.0001);

less likely to suffer psychiatric illnesses (p < 0.0001); and more

likely to suffer one or more physical concerns other than cancer

(p < 0.001).

Overall, cancer cases were more likely to be older compared

to noncancer cases (OR = 1.0, 95% CI = 1.0–1.1). After holding

other variables constant, and compared with noncancer cases,

cancer patients were more likely to never have married (OR = 1.8,

95% CI = 1.0–3.1) or be divorced (OR = 2.1, 95% CI = 1.2–3.5);

use violent suicide methods (OR = 1.7, 95% CI = 1.3–2.1);

communicate their suicidal intent prior to the event (OR = 1.2,

95% CI = 1.1–1.4); less likely to be employed; live alone

(OR = 0.8, 95% CI = 0.6–1.0); suffer from psychiatric problems

(OR = 0.3, 95% CI = 0.3–0.4), and suffer from physical problems

other than cancer (see Table 2).

The differences in characteristics between cancer and non-

cancer cases varied significantly by age. Among young and middle‐
aged suicide cases, and compared with noncancer cases, cancer

patients were more likely to be older (p < 0.0001) and less likely to

live alone (p < 0.05). However, cancer status made no difference in

the elderly. In the young suicide group, cancer patients were more

likely to be female (p < 0.01), however, there was no gender dif-

ference among the middle‐aged and elderly groups. Middle‐aged
and elderly cancer patients were more likely to use violent methods

compared to noncancer cases (p < 0.05); and middle‐aged cancer

patients were more likely to communicate their suicidal intent to

others compared to noncancer cases (p < 0.01). This situation was

not found among the young and elderly. Some characteristics had

similar patterns across all age groups. For instance, compared with

noncancer cases, cancer patients were less likely to be employed

(p < 0.05), and they were less likely to have psychiatric problems

(p < 0.001).

1052 – MEN ET AL.

TAB L E 1 Comparisons of characteristics between cancer and non‐cancer suicide cases

Subject characteristics

Whether having cancer before death

Yes N (%) No N (%) The number of cases analyzed p‐Value

Gender (female) 505 (34.6) 4874 (37.5) 14,445 0.026

In lifetime marital status 13,771 <0.0001

Married 941 (65.7) 5081 (41.2)

Cohabiting 9 (0.6) 328 (3.1)

Never married 133 (9.3) 4219 (34.2)

Divorced 139 (9.7) 1338 (10.8)

Widow 196 (13.7) 1162 (9.7)

Married but separated 15 (1.1) 160 (1.3)

Employment status 12,328 <0.0001

Employed 106 (8.5) 3147 (28.4)

Homemaker 110 (8.9) 732 (6.6)

Unemployed 260 (20.9) 3669 (33.1)

Retired 730 (58.8) 2753 (24.8)

Others 36 (2.9) 785 (7.1)

Type of housing 14,079 <0.0001

Public housing 720 (50.0) 5352 (42.4)

Public subsidized sale flats 185 (12.8) 1517 (12.0)

Private permanent housing 462 (32.1) 4597 (36.4)

Aged home 47 (3.3) 237 (1.9)

Others 27 (1.9) 935 (7.4)

Living alone 219 (17.7) 2825 (24.9) 12,572 <0.0001

Suicide method 14,446 <0.0001

Hanging 402 (27.5) 2684 (20.7)

Jumping 793 (54.3) 6571 (50.6)

CO poisoning 96 (6.6) 2205 (17.0)

Other poisoning 37 (2.5) 473 (3.6)

Drowning 67 (4.6) 408 (3.1)

Others 66 (4.52) 644 (5.0)

Violent suicide method 1328 (90.9) 10307 (79.4) 14,446 <0.0001

Place of suicide 14,312 <0.0001

Inside home 829 (57.2) 7372 (57.3)

Public place 204 (14.1) 1343 (10.4)

Same building other than home 281 (19.4) 2137 (16.6)

Others 135 (9.3) 2011 (15.6)

Any psychiatric conditions 336 (23.0) 6086 (46.9) 14,446 <0.0001

Number of physical conditions (except cancer) 14,446 0.0005

0 744 (50.9) 7227 (55.7)

1 289 (19.8) 2261 (17.4)

2 207 (14.2) 1493 (11.5)

(Continues)

MEN ET AL. – 1053

4 | DISCUSSION

This paper presents new information on characteristics of cancer‐
related suicide cases in Hong Kong, how these differ from noncancer

cases, and the influence of age on the differences in characteristics.

To the best of our knowledge, this study reported on the largest

number of cases so far in this area of research, and thus provides

important information for healthcare providers who are supporting

people suffering from cancer.

Cancer patients were generally older than noncancer cases,

particularly so in the 10–44‐year and 45–64‐year age groups.

However, there were no significant age differences between cancer

and noncancer cases in the oldest (65+) age group. This may be

because the risk of having cancer increases with age, and cancer is

most common among the elderly.20 After adjusting for covariates,

cancer patients were less likely to live alone compared to noncancer

cases. Although living alone is an identified risk factor for suicide,21

our results indicated that living alone would be less of a concern for

cancer patients and they are usually supported by others due to their

physical illnesses. More importantly, our results showed that the

cancer patients were more likely to communicate their suicidal intent

with others compared to noncancer cases, although this difference

was no longer statistically significant after controlling for other

covariates in the young and the elderly subset. Previous research

indicates that barriers preventing healthcare professionals from

identifying suicidality among cancer patients include people giving no

warning and concealing their suicidality.22 Our contradictory results

may be explained by the fact that the information gathered in this

study was based on interviews with family members and friends. It is

possible that cancer patients expressed their intention to kill them-

selves to family members and friends, but were reluctant to seek

professional help from doctors, because of poor mental health liter-

acy, and the stigma that their suicidal ideation might attract within

Chinese culture.23 Hence, by raising the awareness of patient suicidal

risk among caregivers and engaging those who are living with cancer

patients to identify early suicidal signals, it may be possible to

implement prevention strategies.

In our sample, 90.9% of the cancer suicide cases and 79.4% of the

noncancer cases chose violent methods, with the most common

method for both groups being jumping. These percentages are much

higher than those reported in Cheung et al.17 (where 63.6% of

terminal cancer patients and 59.5% of nonterminal cancer patients

chose violent suicide methods). The differences may reflect regional

and cultural differences as our study was conducted in Hong Kong

while Cheung et al.’s study population was from New Zealand. A

study on international suicide method patterns supported our

conclusion that jumping from heights is the most prevalent suicide

method with high lethality in Hong Kong because of easy accessibility

to very tall buildings. However, fewer than 5% of suicide cases used

this method in the New Zealand study.24

Our study also indicated that cancer patients were more likely to

use violent methods compared to noncancer cases, especially by the

middle‐aged and elderly. One explanation is that people with cancer
who are contemplating suicide may be clear in their resolve to end

their lives, in order to end their suffering. Methods commonly used

by cancer patients in our sample, such as jumping, hanging, and

drowning, are all considered lethal with a much higher fatality rate

compared to drugs and poisoning.25 Cancer patients often experience

pain, fatigue, distress, and insomnia during treatment and at end‐of‐
life.26 Moreover, physical discomfort, emotional distress,27 psychi-

atric problems,28 unemployment,29 unmet needs for rehabilitation,30

and low quality of life31 are prevalent among cancer patients. These

factors are strong predictors of suicide.32 Experiencing concurrent

health events which compromise quality of life may drive cancer

patients to choose the most lethal method they can find to end their

pain and suffering. Further research is needed to explore the factors

that influence choice of suicide method among cancer patients who

die by suicide.

As a paradox, however, compared with noncancer cases, cancer

patients were less likely to experience physical problems other than

cancer, with this being most pronounced in the middle‐aged and

elderly groups. One explanation is based on the fact that the medical

history shown in the Coroner’s Court report is obtained from both

medical records and family interviews. It not only covers a range of

severe physical diseases such as cancer, diabetes, and heart disease,

but also includes minor physical problems such as insomnia, head-

aches, and migraines. It is possible that information bias may occur

when the family members of cancer patients do not think about minor

physical issues after reporting major physical diseases like cancer.

This could lead to under‐reporting of other physical conditions. As

people age, they are more likely to experience a range of physical

discomforts. That may explain why the differences in physical condi-

tions between cancer and noncancer suicide cases are more promi-

nent among the middle‐aged and elderly, but not among the young.
Similar to physical comorbidities, cancer patients were less likely

to have been diagnosed with psychiatric conditions compared to the

noncancer cases, with differences being consistent across all age

groups. This finding may be explained by the potential underdiagnosis

T A B L E 1 (Continued)

Subject characteristics

Whether having cancer before death

Yes N (%) No N (%) The number of cases analyzed p‐Value

3+ 221 (15.1) 2004 (15.4)

Any communication of suicide intent 654 (44.8) 4615 (35.5) 14,446 <0.0001

Note: Bold and italic indicate that the p‐value is significant.

1054 – MEN ET AL.

TAB L E 2 Overall and age‐stratified aORs with 95% CI for the association between characteristics and cancer status among suicide cases

Subject characteristics

Age groups

Overall aOR (95% CI) Age 10–44 aOR (95% CI) Age 45–64 aOR (95% CI) Age 65+ aOR (95% CI)

Age 1.0 (1.0–1.1)*** 1.1 (1.0–1.1)*** 1.1 (1.0–1.1)*** 1.0 (1.0–1.0)

Gender (female as reference) 1.0 (0.9–1.2) 0.5 (0.3–0.8)** 1.0 (0.7–1.3) 1.2 (0.9–1.5)

In lifetime marital status

Married (Reference) (Reference) (Reference) (Reference)

Cohabiting 0.6 (0.2–1.5) 0.6 (0.1–7.3) 0.4 (0.1–1.7) 2.0 (0.3–12.3)

Never married 1.8 (1.0–3.1)* 1.7 (0.2–14.4) 1.5 (0.6–4.1) 1.8 (0.8–3.8)

Divorced 2.1 (1.2–3.5)** 1.3 (0.2–10.0) 1.8 (0.7–4.7) 2.1 (1.0–4.1)*

Widow 0.9 (0.5–1.6) 1.3 (0.2–10.2) 1.0 (0.4–2.8) 1.2 (0.6–2.7)

Married but separated 1.12 (0.7–2.0) 2.6 (0.2–46.0) 1.0 (0.3–3.2) 1.6 (0.8–3.2)

Employment status

Employed (Reference) (Reference) (Reference) (Reference)

Homemaker 3.4 (2.4–4.9)*** 3.9 (1.6–9.7)** 2.0 (1.2–3.2)** 5.9 (1.9–18.4)**

Unemployed 2.6 (2.0–3.4)*** 2.4 (1.3–4.5)** 2.4 (1.7–3.3)*** 3.9 (1.3–11.8)*

Retired 3.9 (2.9–5.2)*** 17.3 (3.3–91.8)** 3.1 (2.1–4.5)*** 5.2 (1.8–15.2)**

Others 1.6 (1.0–2.4)* 0.8 (0.2–2.8) 1.6 (1.0–2.5) 2.3 (0.3–16.0)

Type of housing

Public housing (Reference) (Reference) (Reference) (Reference)

Public subsidized sale flats 0.8 (0.7–1.0) 1.2 (0.6–2.5) 0.8 (0.6–1.1) 0.9 (0.7–1.3)

Private permanent housing 0.8 (0.7–0.9)** 0.7 (0.4–1.3) 0.8 (0.6–1.0)* 0.9 (0.7–1.1)

Aged home 0.6 (0.1–2.8) NA NA 0.8 (0.2–3.8)

Others 0.4 (0.3–0.7)** 0.23 (0.0–2.0) 0.3 (0.1–0.6)** 0.7 (0.4–1.4)

Living alone 0.8 (0.6–1.0)* 0.4 (0.2–1.0)* 0.5 (0.4–0.8)*** 0.9 (0.7–1.2)

Violent suicide method 1.7 (1.3–2.1)*** 1.5 (0.8–2.8) 1.6 (1.1–2.2)** 1.6 (1.0–2.4)*

Place of suicide

Inside home (Reference) (Reference) (Reference) (Reference)

Public place 1.1 (0.9–1.4) 1.3 (0.6–3.1) 1.1 (0.8–1.6) 0.9 (0.7–1.2)

Same building other than home 1.2 (1.0–1.5) 1.0 (0.5–2.1) 1.2 (0.9–1.6) 1.2 (1.0–1.6)

Others 0.9 (0.7–1.2) 0.5 (0.2–1.2) 0.8 (0.6–1.2) 1.2 (0.8–1.9)

Any psychiatric conditions 0.3 (0.3–0.4)*** 0.3 (0.2–0.5)*** 0.3 (0.2–0.4)*** 0.3 (0.2–0.4)***

Number of physical conditions (except cancer)

0 (Reference) (Reference) (Reference) (Reference)

1 0.4 (0.4–0.5)*** 0.7 (0.3–1.5) 0.4 (0.3–0.5)*** 0.4 (0.3–0.5)***

2 0.3 (0.2–0.3)*** 1.3 (0.5–3.5) 0.4 (0.2–0.5)*** 0.2 (0.2–0.3)***

3+ 0.2 (0.1–0.2)*** 1.3 (0.4–4.6) 0.3 (0.2–0.5)*** 0.1 (0.1–0.2)***

Any communication of suicide intent 1.2 (1.1–1.4)** 1.6 (1.0–2.7) 1.4 (1.1–1.7)** 1.1 (0.9–1.4)

Note: Bold and italic indicate that the p‐value is significant.
Abbreviations: 95% CI, 95% confidence interval; aOR, adjusted odds ratio; NA, not available.

*p‐Value between 0.01 and 0.05.

**p‐Value between 0.001 and 0.01.

***p‐Value less than 0.001.

MEN ET AL. – 1055

of mental disorders among cancer patients. Weinberger et al.33

argued that the diagnosis of depression among cancer patients can be

complicated because of the overlapping symptoms of depression and

cancer. It is thus likely that the cancer patients in our sample were

underdiagnosed with depression and other psychiatric problems.

Another possible explanation is that cancer patients may suffer

emotional distress because of cancer‐related symptoms and treat-

ment, which may lead to suicide. However, this emotional distress

may not be sufficiently severe to receive a formal diagnosis of a

psychiatric disorder from a healthcare provider.

On the other hand, our findings supported Cheung et al.’s

conclusion, that terminal cancer patients who completed suicide

were significantly less likely to suffer from depression, or to have had

contacts with health system for mental health concerns compared

with noncancer cases.17 The article further explored the concept of

rational suicide, arguing that cancer patients were more likely to end

their lives for reasons that others would consider understandable.

Another study investigating the characteristics of terminal cancer

patients who killed themselves during a home palliative care program

found that many expressed significant concerns about lack of

autonomy and independence, fear of being dependent on others and

fear of suffering.34 It is possible that cancer patients may rationalize

their decision to kill themselves. Such rationales may include not

wanting to be a burden to the family or die with dignity, in contra-

distinction to being influenced by cognitive distortions, emotional

distress or psychiatric problems.

Currently, there are heated debates and ethical concerns about

both rational suicide and euthanasia/physician assisted dying. It is

beyond the scope of this study to establish possible links between

rational thinking and suicide, or to explore the decision‐making
processes among those people with cancer who died by suicide.

Future studies should investigate the mechanisms through which

cancer patients choose to take their lives, and analyze the reasons for

their actions.

Strengths of this study are that it is the first to investigate dif-

ferences in characteristics between cancer and noncancer suicide

cases in an Asian population. The findings suggest that suicide by

cancer patients is prevalent, that the characteristics of cancer suicide

cases are different from noncancer cases, and that age has significant

influence on differences in characteristics. Hence, suicidal risks for

cancer patients cannot be overlooked, and suicide prevention and

intervention strategies should be tailored to the needs of cancer and

terminally ill patients. The study interrogates 15 years of high quality

and large‐volume data collected by a rigorous and comprehensive

agency, as every suspicious case of death in Hong Kong will be

investigated by the Coroner’s Court to gather information and

compile the report. The validity of the dataset is high with a relatively

low chance of misclassification of information. The 15‐year window
allows accurate depiction of over‐time fluctuations in suicide rates in
Hong Kong that serve to reduce potential cohort effects. The long

sampling time frame also enables the inclusion of large sample size

and hence has high statistical power.

4.1 | Study limitations

There was missing information in the Coroner’s Court’s reports due

to reasons such as that the information was unknown, the family was

unable to provide it, etc. There is also a lack of detailed information

on the medical history of the cancer patients. For example, the cancer

type, stage of cancer, and the date of the diagnosis for the patients

was largely unknown. Future studies might provide more compre-

hensive findings by examining how these factors influence cancer

patients’ suicides. Consequently, we are unable to perform additional

analyses to explore suicide‐decision pathways or temporal and casual
relationships between different risk factors for suicide. Future

longitudinal cohort studies may provide a more comprehensive

understanding of the potential risk factors for suicide among cancer

patients.

4.2 | Clinical implications

Current suicide prevention strategies in Hong Kong medical settings

are mainly mental health screening and depression management. Our

findings together with previous research suggested that focusing on

mental health issues for suicide prevention may not be adequate in

detecting early signals for suicide among cancer patients. Besides

mental illness and emotional distress, healthcare professionals and

caregivers should also pay attention to potential suicide risk factors

such as physical pain and the feeling of being a burden, even for those

patients who show no signs of psychiatric or depressive symptoms.

Our results suggested that cancer patients were more likely to

communicate their suicidal intent to family members and friends, but

not necessarily to healthcare professionals. When evaluating cancer

patients’ mental health, it is important for healthcare professionals to

talk to caregivers about cancer patients’ communication about the

quality of their daily lives. Pamphlets including suicide prevention

strategies and resources such as hotline support should be designed

for cancer patients as well as family members and friends to use

during hospital visits. Multidisciplinary efforts are required in clinical

settings and the community and larger social settings to provide

holistic support that is not limited to medical, healthcare, and social

work, and to implement suicide prevention strategies efficiently and

effectively for people at risk.

5 | CONCLUSIONS

In conclusion, our study found that in Hong Kong, cancer suicide

cases have different characteristics compared to noncancer suicide

cases, and the differences in characteristics vary by age group.

Cancer patients are generally older, more likely to kill themselves

using violent methods, more likely to communicate with others about

their suicidal intent, and less likely to have physical and psychiatric

comorbidities. Future research into the motivations behind the

1056 – MEN ET AL.

suicide act will provide deeper understandings of suicide among

cancer patients with the aim of enhancing their wellbeing.

ACKNOWLEDGMENTS

The authors thank the Coroner’s Court for support in accessing the

data. The authors would also like to express our gratitude to Cheuk

Yui Yeung, M.P.H. at the School of Social Work and Social Adminis-

tration, The University of Hong Kong for helpful discussion and

comments that greatly improved the manuscript. This work was

supported by the Li Ka Shing Foundation under Grant (number

AR180055); and the Hong Kong Research Grants Council General

Research Fund (GRF) under Grant (number 17103620).

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available from the

Hong Kong Coroner’s Court. Restrictions apply to the availability of

these data, which were used under license for this study. Data are

available from the corresponding author Paul Siu Fai Yip with the

permission from the Hong Kong Coroner’s Court.

ORCID

Vera Y. Men https://orcid.org/0000-0002-5593-7146

REFERENCES

1. Fitzmaurice C, Akinyemiju TF, Al Lami FH, et al. Global, regional, and

national cancer incidence, mortality, years of life lost, years lived

with disability, and disability‐adjusted life‐years for 29 cancer

groups, 1990 to 2016 a systematic analysis for the global burden of

disease study global burden o. JAMA Oncol. 2018;4(11):1553‐1568.
https://doi.org/10.1001/jamaoncol.2018.2706.

2. Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A.

Global cancer statistics 2018: GLOBOCAN estimates of incidence

and mortality worldwide for 36 cancers in 185 countries. CA Cancer J
Clin. 2018;68(6):394‐424. https://doi.org/10.3322/caac.21492.

3. Public Health England. Cancer Patients at Increased Risk of Suicide.
Public Health England; 2018. Published. https://www.gov.uk/

government/news/cancer‐patients‐at‐increased‐risk‐of‐suicide.
4. Ahn E, Shin DW, Cho SIl, Park S, Won YJ, Yun YH. Suicide rates and

risk factors among Korean cancer patients, 1993‐2005. Cancer Epi-
demiol Biomarkers Prev. 2010;19(8):2097‐2105. https://doi.org/

10.1158/1055‐9965.EPI‐10‐0261.
5. Anguiano L, Mayer DK, Piven ML, Rosenstein D. A literature review

of suicide in cancer patients. Cancer Nurs. 2012;35(4). https://doi.
org/10.1097/NCC.0b013e31822fc76c.

6. Henson KE, Brock R, Charnock J, Wickramasinghe B, Will O, Pitman

A. Risk of suicide after cancer diagnosis in England. JAMA Psychia-
try. 2019;76(1):51‐60. https://doi.org/10.1001/jamapsychiatry.2018.
3181.

7. Lin PH, Liao SC, Chen IM, et al. Impact of universal health coverage

on suicide risk in newly diagnosed cancer patients: population‐based
cohort study from 1985 to 2007 in Taiwan. Psycho Oncol. 2017;
26(11):1852‐1859. https://doi.org/10.1002/pon.4396.

8. Choi J, Lee M, Ki M, et al. Risk factors for feelings of sadness

and suicide attempts among cancer survivors in South Korea:

findings from nationwide cross‐sectional study (KNHANES IV‐VI).
BMJ Open. 2017;7(12):1‐9. https://doi.org/10.1136/bmjopen‐2017‐
016130.

9. Shin DW, Ahn E, Kim H, Park S, Kim YA, Yun YH. Non‐cancer
mortality among long‐term survivors of adult cancer in Korea:

National Cancer Registry Study. Cancer Causes Control. 2010;21(6):
919‐929. https://doi.org/10.1007/s10552‐010‐9521‐x.

10. Joshi P, Song HB, Lee SA. Association of chronic disease prevalence

and quality of life with suicide‐related ideation and suicide attempt

among Korean adults. Indian J Psychiatry. 2017;59(3):352‐358.
https://doi.org/10.4103/psychiatry.IndianJPsychiatry_282_16.

11. Zhong BL, Li SH, Lv SY, et al. Suicidal ideation among Chinese cancer

inpatients of general hospitals: prevalence and correlates.

Oncotarget. 2017;8(15):25141‐25150. https://doi.org/10.18632/

oncotarget.15350.

12. Lee SJ, Park JH, Park BY, et al. Depression and suicide ideas of

cancer patients and influencing factors in South Korea. Asian Pacific J
Cancer Prev. 2014;15(7):2945‐2950. https://doi.org/10.7314/APJCP.
2014.15.7.2945.

13. Shim EJ, Park JH. Suicidality and its associated factors in cancer

patients: results of a multi‐center study in Korea. Int J Psychiatry
Med. 2012;43(4):381‐403. https://doi.org/10.2190/PM.43.4.g.

14. Hong Kong Hospital Authority. Overview of Hong Kong Cancer Sta-
tistics of 2016; 2018. http://www3.ha.org.hk/cancereg/pdf/overview/
SummaryofCanStat2016.pdf.

15. Ma Y, Mak K. Management of depression in cancer patients. Hong
Kong Pract 2012;34(4):154‐159.

16. Yeung SL. Assessing and Managing Potentially Suicidal Patients: Prac-
tical Guidelines for Doctors; 2015. https://csrp.hku.hk/wp‐content/
uploads/2015/06/DoctorGuidelines.pdf.

17. Cheung G, Douwes G, Sundram F. Late‐life suicide in terminal can-

cer: a rational act or underdiagnosed depression? J Pain Symptom
Manage. 2017;54(6):835‐842. https://doi.org/10.1016/j.jpainsymman.
2017.05.004.

18. Massetti GM, Holland KM, Jack SPD, Ragan KR, Lunsford NB.

Circumstances of suicide among individuals with a history of cancer.

Psycho Oncol. 2018;27(7):1750‐1756. https://doi.org/10.1002/pon.
4720.

19. Stenbacka M, Jokinen J. Violent and non‐violent methods of

attempted and completed suicide in Swedish young men: the role of

early risk factors. BMC Psychiatr. 2015;15:196. https://doi.org/

10.1186/s12888‐015‐0570‐2.
20. Hong Kong Cancer Registry. Overview of 2009 Hong Kong Cancer

Statistics; 2017. Published. https://www3.ha.org.hk/cancereg/pdf/

overview/SummaryofCanStat2009.pdf.

21. Schneider B, Lukaschek K, Baumert J, Meisinger C, Erazo N,

Ladwig KH. Living alone, obesity, and smoking increase risk for

suicide independently of depressive mood findings from the pop-

ulation‐based MONICA/KORA Augsburg cohort study. J Affect
Disord. 2014;152‐154(1):416‐421. https://doi.org/10.1016/j.jad.

2013.10.007.

22. Granek L, Nakash O, Ben‐David M, Shapira S, Ariad S. ‘Oncologists’,

nurses’, and social workers’ strategies and barriers to identifying

suicide risk in cancer patients. Psycho Oncol. 2018;27(1):148‐154.
https://doi.org/10.1002/pon.4481.

23. Shen Y‐C, Zhang M‐Y, Huang Y‐Q, et al. Twelve‐month prevalence,

severity, and unmet need for treatment of mental disorders in

metropolitan China. Psychol Med. 2006;36(2):257‐267. https://doi.
org/10.1017/S0033291705006367.

24. Ajdacic‐Gross V, Weiss MG, Ring M, et al. Methods of suicide: in-

ternational suicide patterns derived from the WHO mortality

database. Bull World Health Organ. 2008;86(9):726‐732. https://doi.
org/10.2471/BLT.07.043489.

25. Elnour AA, Harrison J. Lethality of suicide methods. Inj Prev.
2008;14(1):39‐45. https://doi.org/10.1136/ip.2007.016246.

26. Theobald DE. Cancer pain, fatigue, distress, and insomnia in

cancer patients. Clin Cornerstone. 2004;6(1):S15‐S21. https://doi.org/
10.1016/S1098‐3597(05)80003‐1.

27. Strong V, Waters R, Hibberd C, et al. Emotional distress in cancer

patients: the Edinburgh Cancer Centre symptom study. Br J

MEN ET AL. – 1057

Cancer. 2007;96(6):868‐874. https://doi.org/10.1038/sj.

bjc.6603626.

28. Wilson KG, Dalgleish TL, Chochinov HM, et al. Mental disorders and

the desire for death in patients receiving palliative care for cancer.

BMJ Support Palliat Care. 2016;6(2):170‐177. https://doi.org/

10.1136/bmjspcare‐2013‐000604.
29. Nakaya N, Nakamura T, Tsuchiya N, Tsuji I, Hozawa A, Tomita H.

Unemployment risk among individuals undergoing medical treat-

ment for chronic diseases. Occup Med (Chic Ill). 2016;66(2):143‐149.
https://doi.org/10.1093/occmed/kqv159.

30. Hansen DG, Larsen PV, Holm LV, Rottmann N, Bergholdt SH,

Søndergaard J. Association between unmet needs and quality of

life of cancer patients: a population‐based study. Acta Oncol
(Madr). 2013;52(2):391‐399. https://doi.org/10.3109/0284186X.

2012.742204.

31. Faller H, Schuler M, Richard M, Heckl U, Weis J, Kuffner R. Effects of

psycho‐oncologic interventions on emotional distress and quality of
life in adult patients with cancer: systematic review and meta‐
analysis. J Clin Oncol. 2013;31(6):782‐793. https://doi.org/10.1200/
JCO.2011.40.8922.

32. Centers for Disease Control and Protection. Risk and Protective
Factors for Suicide; 2019. Published. https://www.cdc.gov/violence
prevention/suicide/riskprotectivefactors.html.

33. Weinberger MI, Roth AJ, Nelson CJ. Untangling the complexities

of depression diagnosis in older cancer patients. Oncol.
2009;14(1):60‐66. https://doi.org/10.1634/theoncologist.2008‐
0147.

34. Filiberti A, Ripamonti C, Totis A, et al. Characteristics of terminal

cancer patients who committed suicide during a home palliative care

program. J Pain Symptom Manage. 2001;22(1):544‐553. https://doi.
org/10.1016/S0885‐3924(01)00295‐0.

How to cite this article: Men VY, Emery CR, Yip PSF.

Characteristics of cancer patients who died by suicide: A

quantitative study of 15‐year coronial records. Psycho‐
Oncology. 2021;30:1051–1058. https://doi.org/10.1002/
pon.5634

1058 – MEN ET AL.

Copyright of Psycho-Oncology is the property of John Wiley & Sons, Inc. and its content
may not be copied or emailed to multiple sites or posted to a listserv without the copyright
holder’s express written permission. However, users may print, download, or email articles for
individual use.

  • Characteristics of cancer patients who died by suicide: A quantitative study of 15‐year coronial records
    • 1 | BACKGROUND
    • 2 | METHODS
      • 2.1 | Statistical methods
    • 3 | RESULTS
    • 4 | DISCUSSION
      • 4.1 | Study limitations
      • 4.2 | Clinical implications
    • 5 | CONCLUSIONS
    • ACKNOWLEDGMENTS
    • DATA AVAILABILITY STATEMENT

Psychosomatics 2020:61:698–706 ª 2020 Academy of Consultation-Liaison Psychiatry. Published by Elsevier Inc. All rights reserved.

Original Research Article

698

The Feasibility and Impact of a Suicide Risk
Screening Program in Rural Adult Primary Care:

A Pilot Test of the Ask Suicide-Screening
Questions Toolkit

Mary A. LeCloux, Ph.D., Mathew Weimer, M.D., Stacey L. Culp, Ph.D.,
Karissa Bjorkgren, B.S., Samantha Service, M.S., John V. Campo, M.D.

Objective: The purpose of this study was to evaluate
the feasibility and impact of a suicide risk screening
program in a rural West Virginia primary care prac-
tice. Methods: Patients presenting for routine and sick
visits were asked to participate in electronic suicide
risk screening using the Ask Suicide-Screening Ques-
tions tool; screen positive individuals were assessed
with the Ask Suicide-Screening Questions Brief Suicide
Safety Assessment. Screening program feasibility was
evaluated by the proportion of patients consenting to
participate, participant Ask Suicide-Screening Ques-
tions and Brief Suicide Safety Assessment completion
rates, and response to a question asking whether pri-
mary care providers should ask about suicide.
Screening impact was evaluated quasi-experimentally
by comparing electronic medical record documentation
of suicide risk screening, assessment, and risk deter-
mination in practice patients before and after

www.psychosomaticsjournal.org

implementing the screening program. Results: Over half
of the patients approached agreed to participate in
a research study about suicide (N = 196; 57.7%).
Feasibility of the screening program was demonstrated
by the high completion rates for the Ask Suicide-
Screening Questions (99.0%) and the Brief Suicide
Safety Assessment (100.0%) among study participants.
Additionally, 95.4% (N = 187) of participants agreed
primary care providers should screen patients for suicide.
Suicide screening rates rose significantly between the
baseline and intervention phases (5.8% to 61.0%;
X2 = 200.61, P , 0.001), as did suicide risk detection
rates (0.7% to 6.2%; X2 = 12.58, P , 0.001).
Conclusion: Suicide risk screening was feasible and
well accepted by adult patients in rural primary care
and has potential to improve suicide risk detection in
this setting.

(Psychosomatics 2020; 61:698–706)

Key words: suicide, primary care, mental health, behavioral medicine.

ReceivedFebruary 27, 2020; revisedMay7, 2020; acceptedMay 7, 2020.
From the School of Social Work (M.A.L., K.B.), West Virginia Uni-
versity, Morgantown, WV; Family Medicine, Valley Health Systems
(M.W.), Milton, WV; Department of Statistics (S.L.C., S.S.), West
Virginia University, Morgantown, WV; Department of Behavioral
Medicine and Psychiatry (J.V.C.), School of Medicine, West Virginia
University, Morgantown, WV. Send correspondence and reprint re-
quests toMary A. LeCloux, PhD, School of SocialWork,West Virginia
University, 105 Knapp Hall, Morgantown, WV 26506; e-mail: mary.
[email protected]

ª 2020Academy ofConsultation-Liaison Psychiatry. Published

INTRODUCTION

Suicide is a significant global public health problem.
The World Health Organization reports that 793,000
people died by suicide worldwide in 2016,1 and the
Centers for Disease Control and Prevention indicates
that more than 47,000 people died by suicide in the
United States in 2017.2 Suicide rates were significantly
higher in rural than in urban areas in 2013–2015 (19.74
per 100,000 compared with 12.72 per 100,000).3 In

by Elsevier Inc. All rights reserved.

Psychosomatics 61:6, November/December 2020

LeCloux et al.

addition, suicide rates in rural communities are
increasing at a higher rate over the past decade,3,4 and
rural communities are disproportionately impacted by
social deprivation compared with metropolitan areas.5

Suicide risk screening programs can contribute to
suicide prevention efforts in community healthcare
settings by increasing screening6,7 and risk detection
rates,7 decreasing acute care utilization,8 and increasing
behavioral health referrals.9 The acceptability of suicide
risk screening programs has been repeatedly demon-
strated in adult emergency departments,10 inpatient
settings,11,12 pediatric emergency rooms,13–15 and adult
primary care.16 Importantly, a universal suicide risk
screening program in the emergency department com-
bined with a brief intervention for at-risk individuals
decreased subsequent suicidal behavior by 30%
compared with usual care.10 The Joint Commission
currently includes universal suicide risk screening for
patients presenting with behavioral health concerns
among its National Patient Safety Goals17 and suicide
risk screening is now part of the most recent National
Strategy for Suicide Prevention.18

Implementation of suicide risk screening programs
may be particularly crucial for rural settings. First, in-
dividuals who live in rural areas are more likely to visit
primary care providers (PCPs) than mental health
professionals because of limited access to behavioral
healthcare services, high levels of stigma associated
with mental health help-seeking in rural communities,
and concerns about confidentiality.19–21 Moreover, up
to 83% of people who die by suicide22 and 95% of
suicide attempters23 have visited a healthcare provider
in the previous year, and half of suicide decedents have
visited a healthcare provider in the month before
death.23 To date, most adult suicide risk screening
studies have been conducted in urban settings.

The Ask Suicide-Screening Questions (ASQ)
screening tool and corresponding Toolkit were devel-
oped by Horowitz et al.24 at the National Institute of
Mental Health in response to a paucity of validated
standardized suicide risk screening and follow-up
assessment tools for medical settings.25,26 The Toolkit
includes the Brief Suicide Safety Assessment (BSSA), a
brief guide to follow-up safety assessment and man-
agement. The ASQ screening tool has demonstrated
high sensitivity (83–98%)24,27,28 and good specificity28

in samples of youths aged 10 to 21 years. Preliminary
data also indicate that the ASQ has high sensitivity
and specificity among adult inpatients.29

Psychosomatics 61:6, November/December 2020

The ASQ and BSSA are designed to be used in
tandem as part of a 3-tiered screening program.30 Risk is
first identified with the ASQ brief screen, followed by a
10–12 minute follow-up risk assessment using the BSSA
as a guide, and then disposition planning and aftercare
are provided as needed depending on the patient’s level
of risk. Patients determined to be at imminent risk of
suicide are referred for more extensive crisis evaluation
and/or are administered appropriate suicide precautions.

The primary aim of this study was to explore the
feasibility and impact of a suicide risk screening pro-
gram using the ASQ and the BSSA in an adult, primary
care practice in rural West Virginia.

METHODS

A 2-phase design was used in which data were collected
from patients of one specific PCP in a rural outpatient
primary care clinic during a baseline phase (treatment
as usual) and a screening intervention phase that invited
patients of the participating PCP to participate in a
research study that applied the ASQ Toolkit.

Baseline Phase

During the baseline phase, which lasted for 1 month, a
behavioral health clinician/research assistant mined data
from the electronic medical record (EMR) for consecu-
tive patients who were provided usual care by the PCP.
Only data from patients older than 18 years were used.
Data for baseline patients were logged and deidentified
by the clinician/research assistant onto Excel spread-
sheets and then entered into SPSS for analysis.

Data recorded from the EMR included basic de-
mographics as well as any documentation of suicide risk
screening, the resulting risk determination, and any asso-
ciated disposition plan. Specifically, data were recorded
from the EMR regarding the following: (1) whether the
PCP had asked the patient any questions about suicide
during the visit, (2) any identified level of suicide risk
(“acute,” “nonacute,” “no/low risk,” and “undeter-
mined”), and (3) any disposition plans related to suicide
risk (“follow-up with PCP,” “behavioral health referral,”
“emergency department,” “safety plan,” and “other”).

Intervention Phase

During the 3-month long intervention phase, primary
care patients were invited to participate in a “research
study about suicide” while being checked in for well or

www.psychosomaticsjournal.org 699

Suicide Risk Screening in Rural Primary Care

sick visits at the primary care office. Patients who were
too medically ill to participate were excluded. Inter-
ested patients were given an electronic tablet, watched a
brief consent video, and were given the option to con-
sent electronically. Consenting patients were directed to
an electronic survey including the ASQ screening
questions and a 5-item Screening Opinions Question-
naire (see Figure 1)* developed by the research team.

Results from the ASQ were uploaded in real time
and viewed by the PCP before meeting with each
patient. For patients who screened positive for sui-
cide risk, the PCP administered a brief suicide risk
assessment using the BSSA as a guide and then
provided disposition planning as needed. Before the
intervention phase, the PCP participated in a brief
webinar training that described the BSSA and how to
best administer it with patients. The PCP recorded
the risk level and disposition plans for patients in the
EMR.

The research assistant then mined data from the
EMR for the intervention patients that included pa-
tient demographics, risk level, and disposition plans.
Data were also mined, logged, and deidentified for
nonconsenting patients regarding whether the PCP
asked about suicide risk during their visits and any
associated risk level and disposition plans. Finally,
demographic data were collected from the EMR for
patients who were approached during the interven-
tion phase but declined to participate. All study
procedures received full institutional review board
approval by the supporting university before the start
of the study.

Measures

Feasibility

Feasibility of the screening program was evaluated using:
(1) the participant consent rate during the intervention
phase, (2) the ASQ and BSSA completion rates for
consenting participants, and (3) screening participant
responses to the questionnaire item that asked “Do you
think primary care providers should ask adults about
suicidal thoughts when they are at the doctor’s office?”

* Results from additional items in the Screening Opinions Question-
naire are described in a separate manuscript.

700 www.psychosomaticsjournal.org

Impact of Screening Program Implementation on Suicide
Risk Assessment and Detection

To measure the impact of the screening program on the
frequency of suicide screening/assessment during each
phase, the proportion of patients with documentation of
being asked about suicide in the EMR during the base-
line phase was compared with the proportion of practice
patients who were screened for suicide either via the use
of the ASQ or, for nonconsenting patients, who were
asked about suicide risk directly by the PCP during the
intervention phase. To compare suicide risk detection
rates between phases, the proportion of patients who had
documentation of some sort of suicide risk in the EMR
during the baseline phase was compared with the pro-
portion of patients who were identified as at risk either
throughASQ screening or by documentation by the PCP
in the EMR during the intervention phase.

Data Analysis

All data were uploaded and analyzed using SPSS 26.
Descriptive statistics were computed to evaluate the 3
measures of feasibility (consent rate, completion rate,
and responses to the screening opinions item). Chi-
square analyses using Fisher’s exact test were
computed to compare the proportions of patients who
were screened for suicide risk and who were detected as
at risk for suicide during both phases.

RESULTS

Sample

Baseline

The baseline sample included 274 patients, 99.6% of
whom identified as white, and 52.2% of whom were
women. The mean age of the baseline sample was 58.24
years (standard deviation = 16.3).

Intervention

There were initially 340 subjects recruited to the inter-
vention sample, but because of on-site data entry er-
rors, 2 cases had to be deleted from the analysis,
resulting in a final intervention sample of N = 338 pa-
tients. Of the 338 patients who were approached to
participate in the study, 204 (60.4%) agreed to watch
the consent video and 196 (58.0%) gave consent to

Psychosomatics 61:6, November/December 2020

FIGURE 1. The Screening Opinions Questionnaire.

1.) Has anyone ever asked you about suicide before? ____ Yes ____ No

1a.) If yes, who asked you? ______________________________

2.) Do you think primary care providers should ask adults about suicidal thoughts when they are at

the doctor’s office? _____ Yes _____ No

2a.) Why or why not? ____________________

3.) What was it like to be asked about suicide today? ___________________________________

4.) In the future, which would you prefer, having the provider or nurse ask you questions or to fill

out a questionnaire yourself on an electronic device or paper survey?

____ Provider ask you

If checked, then respondent is asked:

“Who would you prefer ask you?”

___Nurse/Medical Assistant

___Primary care provider

___No preference

____ Electronic device

____ Paper survey

4a.) Why do you think this way is better? ______________________________

5.) Please note any other comments here. _________________________________________

LeCloux et al.

participate. There were no significant differences in
gender between consenting participants and those who
declined, but consenting participants were significantly
younger (M = 54.0, standard deviation = 15.8) than
nonconsenters (M = 61.8, standard deviation = 16.5,
P , 0.001). There were no significant gender or age
differences across phases. Complete sample de-
mographics for both phases are included in Table 1.

Responses to the Individual ASQ Items

Responses to the individual screening items are depic-
ted in Table 2. Responses were fairly evenly split

Psychosomatics 61:6, November/December 2020

between the first 4 items, with the exception of the item
that asked participants about recent suicidal ideation,
which fewer participants endorsed (N = 2). Only 1
participant endorsed the fifth acuity question.

Feasibility of Screening Using the ASQ and BSSA

During the intervention phase, a total of 206 patients
were screened for suicide (61.0%). Of the total inter-
vention sample, 194 patients were screened with the
ASQ (56.2%), and 12 were asked about suicide risk
directly by the provider (3.6%). Of the patients

www.psychosomaticsjournal.org 701

TABLE 1. Sample Demographics

Phase

Baseline
(N = 274)

Intervention (N = 338)

Consenting
participants
(N = 196)

Nonconsenting
participants
(N = 142)

N % N % N %

Race
White 273* 99.6% 196 100.0% 144 100.0%

Gender
Female 143 52.2% 97 49.5% 60 42.3%

M SD M SD M SD
Age

58.2† 16.3 54.0 15.8 61.8‡ 16.3

SD = standard deviation.

* Includes 1 missing response.
† P , .01.
‡ P , .001.

Suicide Risk Screening in Rural Primary Care

identified as at risk during the intervention phase, 20
(5.9%) were nonacute positive screens and 1 (0.3%) was
an acute positive screen. All consenting participants
who screened positive on the ASQ were administered
the full BSSA by the PCP, resulting in a completion
rate of 100.0%. One patient identified as at-risk after an
informal screen by the PCP did not consent to
completing the BSSA. All but 1 consenting participant
completed the full ASQ screen, resulting in a partici-
pant completion rate of 99.5%.

Of the patients who were surveyed (consenting
patients in the intervention sample), 95.4% (N = 196)
agreed that PCPs should ask patients about suicide,
including all 20 screen-positive individuals.

Suicide Risk Assessment, Detection, and Disposition
by Phase

Rates of practice efforts to detect and assess suicide risk
by phase are displayed in Table 3. The likelihood of a
documented effort to detect suicide risk, or screening,
was significantly greater during the intervention phase
(61.0%) than the baseline phase (5.8%; X2 = 200.6,
P , 0.001). In addition, suicide risk was detected in
6.2% of the cases during the intervention phase
compared to only 0.7% of cases in the baseline phase, a
significant between group difference (X2 = 12.58,
P , 0.001).

Management and disposition plans for patients
determined to be clinically at risk for suicide are

702 www.psychosomaticsjournal.org

depicted in Table 4. Most dispositions followed non-
acute presentations, where participants received safety
planning, external behavioral health referral, and/or
follow-up from the PCP. Only 1 patient was sent to the
emergency department during the intervention phase
(0.3%) after screening positive on the ASQ and
completion of the BSSA. All but 1 of the patients who
were identified as at risk during the intervention phase
received a safety planning intervention, whereas neither
of the 2 patients identified as being at risk for suicide
during the baseline phase were exposed to safety plan-
ning. Disposition plans were uniform across phases,
with the exception of the safety planning intervention,
which was informed by training provided in the BSSA.

DISCUSSION

Feasibility of the ASQ/ASQ BSSA in Rural Primary
Care

The results of this pilot study provide preliminary evi-
dence that suicide risk screening using the ASQ and the
BSSA is feasible and acceptable to patients in the rural,
adult primary care setting. The overall consent rate for
the screening program was high, with approximately
60% of patients approached agreeing to participate in a
“general research study about suicide,” and 96% of
consenting participants completing the ASQ and the
study after learning it involved completing a suicide risk
screening questionnaire. The vast majority (96%) of
consenting patients agreed that PCPs should ask pa-
tients about suicide. Although it is important to note
that 40% of the total intervention sample were not
surveyed about their views regarding PCP screening, it
is still encouraging that the majority of those who were
surveyed endorsed the practice of screening in primary
care. These findings in a rural, adult ambulatory pri-
mary care setting are consistent with findings from
other empirical studies that suicide risk screening is
feasible and acceptable to most patients, including
those in inpatient adult medical units, pediatric emer-
gency rooms, and urban adult primary care.12–16,29

Given the high levels of stigma associated with
mental health help-seeking in rural areas,21 it is reas-
suring that a significant proportion of patients in this
rural Appalachian primary care setting agreed to sui-
cide risk screening with a standardized measure. This
finding suggests that suicide risk screening programs in

Psychosomatics 61:6, November/December 2020

TABLE 2. Response to Individual ASQ Items

Item response (N = 196) No Yes

N % N %

ASQ Item
Item 1: “In the past few weeks, have you wished you were dead? 188 95.9% 8 4.1%
Item 2: “In the past few weeks, have you felt you or your family would be better off if you were dead?” 182 93.8% 12 6.1%*
Item 3: “In the past few weeks, have you been having thoughts about killing yourself?” 194 99.0% 2 1.0%
Item 4: “Have you ever tried to kill yourself?” 183 93.4% 12 6.1%
Item 5: “Are you having thoughts of killing yourself right now?” 20† 95.2% 1 4.8%

ASQ = Ask Suicide-Screening Questions.

* Includes 2 missing responses.
† Includes only participants who answered “yes” to one of the first 4 items.

LeCloux et al.

rural primary care settings are likely amenable to pa-
tients and deserve further study.

Participant completion rates for both the ASQ and
BSSA were also high. All but 1 participant completed
the full ASQ, and a PCP subsequently assessed all 20
patients who screened positive for suicide risk on the
ASQ with the BSSA. Given the time pressures and
heavy workloads in primary care,31 these high
completion rates are useful findings regarding the
feasibility of implementing the ASQ Toolkit, as they
suggest that application of the ASQ Toolkit was
acceptable to the provider and did not significantly
disrupt clinical flow in a busy primary care office.
Although completion rates by themselves may not
demonstrate feasibility in all clinics (e.g., the comple-
tion of the ASQ and BSSA might interfere with other
tasks or extend the workday), follow-up communica-
tions with the research team indicated that integration
of the ASQ and BSSA did not significantly disrupt
workflow in this particular clinic. It is important to note
that this was one specific practice, and certain factors,
such as the use of electronic screening and focused
provider training in risk assessment and management
using the BSSA may have facilitated this integration.
Such efforts may inform the implementation of future
screening programs, as they likely streamlined the
process of screening and subsequent risk assessment
and management.

Administration of the ASQ identified positive
screens for suicide risk in approximately 6% of all of
patients seen by the provider during the intervention
phase, and in 10.3% of the patients who consented to be
screened, a rate similar to other screening studies of
suicide risk in adolescents and adults in primary care
and the emergency department.9,32-35 In addition, it

Psychosomatics 61:6, November/December 2020

appears that the process of implementing a suicide risk
screening program using the ASQ and the BSSA
increased the extent to which the PCP considered
asking about suicide risk with the nonconsenting pa-
tients in his practice, highlighting the potentially
broader impact a routinized screening program can
have beyond the screening procedure alone. Moreover,
only 1 patient was determined to be at imminent risk
after completion of the follow-up risk assessment. The
fact that most positive screens were nonacute was an
especially encouraging finding for rural primary care
practices, as these communities often lack access to
specialty behavioral health services,36 making care
provided by the PCP especially relevant to rural pop-
ulation health. Brief interventions such as safety plan-
ning,37 caring contacts,38 and scheduled follow-up
appointments, as well as the proactive recognition
and management of associated mental disorders such as
depression,39 have potential to be adapted and applied
in rural primary care even in the absence of robust
specialty behavioral health referral options.

The Impact of the ASQ/BSSA Program on Screening
and Detection Rates

A secondary study aim was to describe how the imple-
mentation of a suicide risk screening program using the
ASQ and ASQ BSSA would impact the likelihood of
clinical suicide risk screening and detection compared
with treatment as usual in a rural primary care clinic.
Consistent with other studies of pediatric emergency
rooms and primary care settings, the implementation of
a suicide risk screening program for adults in rural pri-
mary care was found to increase the likelihood of suicide
risk screening6,7 and detection.7 Overall, rates of both

www.psychosomaticsjournal.org 703

TABLE 4. Disposition Plans by Phase for Patients Identified as
at Risk for Suicide

Disposition plan Phase

Baseline
(N = 274)

Intervention
(N = 338)

Safety plan 0 0.0% 20 5.9%
Follow-up with PCP 2 0.7% 17 5.0%
Home with mental health referral 2 0.7% 4 1.2%
Sent to emergency room 0 0.0% 1 0.3%
Other (medication) 0 0.0% 1 0.3%

PCP = primary care provider.

TABLE 3. Suicide Risk Assessment and Detection Rates by
Phase

Phase

Baseline
(N = 274)

Intervention
(N = 338)

N % N %

Screened for suicide risk 16 5.8% 206 61.0%*
Suicide risk detected 2 0.7% 21 6.2%*

* P , .001.

Suicide Risk Screening in Rural Primary Care

increased significantly during the intervention phase
relative to baseline, with almost 60% of patients screened
for suicide risk during the intervention phase compared
with only about 6% at baseline. In addition, suicide risk
was detected in 6% of patients in the intervention phase
compared with only 0.7% at baseline, a conservative
finding given that suicide risk was detected in more than
10% of patients who consented to complete the ASQ.
One might reasonably expect greater willingness of rural
patients to complete a brief suicide screen as the culture
of primary care practice evolves to make such screening
routine and not encumbered by a formalized consent
process that began with an invitation to participate in “a
research study about suicide.”

The observed increase in risk detection during the
intervention phase has important clinical implications.
Although data regarding patient outcomes of detailed
clinical follow-up were not available, all but one of the
patients determined to be at-risk during the interven-
tion phase received a safety plan, an evidence-based
intervention that has been found to mitigate suicide
risk,37 compared with none in the baseline phase.
Future studies should consider the link between
increased risk detection, management, and outcomes.

The importance of the BSSA as an element of the
screening program should also be noted. The only pa-
tients who received a safety planning intervention from
the PCP were ones who were administered the BSSA by
the PCP. This supports the notion that having a
structured protocol for PCPs, such as the BSSA, en-
sures that evidence-based practices are followed while
completing a suicide risk assessment. Moreover, it
suggests that providing even brief training and guidance
can significantly impact provider behavior and service
provision. This finding replicates other studies that have
demonstrated the impact of provider training on con-
fidence, comfort, and service provision related to sui-
cide risk screening.9,40

704 www.psychosomaticsjournal.org

Although this study was only a pilot in a single
practice, it provides preliminary support that suicide risk
screening using theASQToolkit can increase levels of risk
detection among adult patients in primary care. In fact,
had the screening program been implemented simply as a
standard of care measure, rather than as a research study
that required informed consent, it is likely that screening
and detection rates would have been even higher had
patients been offered screening as a practice expectation.

This study did not explore other outcomes, such as
suicidal behavior or mental health service use of patients
after screening. Future studies evaluating the efficacy
and validity of suicide risk screening programs could do
so by gathering longitudinal follow-up data from pa-
tients on these variables. Moreover, it would be helpful
to evaluate the impact of brief interventions provided in
combination with screening on later suicide risk and
acute care utilization. Finally, screening programs such
as the one described in this study should be evaluated in
larger, more diverse samples of rural adult primary care.

Study Limitations

This study did have limitations. First, this was a pilot
study conducted with a small, racially homogenous
sample cared for by a single PCP in an isolated rural
practice. Accordingly, this was a unique provider, and
future research is needed before these results can be
generalized to other populations. Second, there may
have been outside variables, such as mental health
diagnosis, treatment history, and whether participants
were attending the clinic for a well or sick visit, that
impacted patient willingness to consent as well as out-
comes related to screening. Third, as this was a 2-phase
design, some of the differences in the intervention
condition could be attributed to factors other than

Psychosomatics 61:6, November/December 2020

LeCloux et al.

implementation of the screening program, such as the
training that was provided, availability of the Toolkit
elements, or increased attunement to the topic of sui-
cide. Finally, because nonconsenting patients were not
surveyed, it is impossible to know their views regarding
suicide risk screening, which may have biased the re-
sults regarding the acceptability question. Accordingly,
patients who consented to the study may have been
favorably biased toward screening, which may have
further impacted the results regarding acceptability.

CONCLUSION

Study results provide preliminary evidence that a sui-
cide risk screening program using the ASQ and the
BSSA is feasible and acceptable to adults in rural pri-
mary care and that implementation of the ASQ Toolkit
has potential to improve suicide risk assessment and
detection rates in these settings. Future studies should
expand these findings by conducting randomized

Psychosomatics 61:6, November/December 2020

experimental trials of screening on suicide risk detection
and management in diverse medical settings with
diverse populations and in relation to longitudinal
outcomes.

Funding: This project was supported by a pilot grant
from the West Virginia Clinical and Translational Sci-
ence Institute (WVCTSI) [Award #C1006000AW] and
received administrative support from the West Virginia
Practice-Based Research Network (WVPBRN). The
WVCTSI and WVPBRN provided resource support for
provider and site recruitment and technological assis-
tance related to the electronic screening and data
collection, but were not directly involved with imple-
mentation of the study, analysis of the data, or writing of
the final manuscript.

Disclosure: The authors report no proprietary or
commercial interest in any product mentioned or concept
discussed in this article.

References

1. World Health Organization (WHO). 2016. Global Health
Estimates 2016: Estimated Deaths by Age, Sex, and Cause.
Available from: https://www.who.int/healthinfo/global_
burden_disease/estimates/en/. Accessed February 12, 2020.

2. Centers for Disease Control and Prevention (CDC). 2017a.
Fatal Injury Reports. WISQARS Data Visualization Tool.
Available from: https://www.cdc.gov/injury/wisqars/index.
html. February 12, 2020.

3. Ivey-Stephenson AZ, Crosby AE, Jack SPD,
Haileyesus T, Kresnow-Sedacca M: Suicide trends among
and within urbanization levels by sex, race/ethnicity, age
group, and mechanism of death-United States, 2001-
2015. Mortal Morb Wkly Rep Surveill Summ 2017;
66:1–16

4. Kegler SR, Stone DM, Holland KM: Trends in suicide by
level of urbanization- United States, 1999-2015. MMWR
Morb Mortal Wkly Rep 2017; 66:270–273

5. Steelesmith DL, Fontanella CA, Campo JV, Bridge JA,
Warren KL, Root ED: Contextualfactors associated with
county-level suicide rates in the United States, 1999 to 2016.
JAMA Netw Open 2019; 2:e1910936

6. Betz ME, Arias SA, Miller M, et al: Change in emergency
department providers’ beliefs and practices after use of new
protocols for suicidal patients. Psychiatr Serv 2015; 66:625–631

7. Diamond GS, Herres JL, Krauthamer Ewing ES, et al:
Comprehensive screening for suicide risk in primary care. Am
J Prev Med 2017; 53:48–54

8. Wintersteen MB, Diamond GS: Youth suicide prevention in
primary care: a model program and its impact on psychiatric
emergency referrals. Clin Pract Pediatr Psychol 2013; 1:295–305

9. Frick MG, Butler SA, deBoer DS: Universal
suicide screening in college primary care. J Am Coll Health
2019:1–6

10. Miller IW, Camargo CA, Arias SA, et al: Suicide prevention
in an emergency department population: the ED-SAFE
study. JAMA Psychiatry 2017; 74:563–570

11. Horowitz LM, Snyder D, Ludi E, et al: Ask suicide-screening
questions to everyone in medical settings: the asQ’em quality
improvement project. Psychosomatics 2013; 54:239–247

12. Snyder DJ, Ballard ED, Stanley IH, et al: Patient opinions
about screening for suicide risk in the adult medical inpatient
unit. J Behav Health Serv Res 2017; 44:364–372

13. Horowitz L, Ballard E, Teach SJ, et al: Feasibility of
screening patients with nonpsychiatric complaints for suicide
risk in a pediatric emergency department: a good time to
talk? Pediatr Emerg Care 2010; 26:787–792

14. Ballard ED, Bosk A, Snyder D, et al: Patients’ opinions
about suicide screening in a pediatric emergency department.
Pediatr Emerg Care 2012; 28:34–38

15. Ballard ED, Stanley IH, Horowitz LM, Cannon EA,
Bridge JA: Asking youth questions about suicide risk in the
pediatric emergency department: results from a qualitative
analysis of patient opinions. Clin Pediatr Emerg Med 2013;
14:20–27

16. Richards JE, Hohl SD, Whiteside U, et al: If you listen, i will
talk: the experience of being asked about suicidality during
routine primary care. J Gen Intern Med 2019; 34:2075–2082

17. The Joint Commision: National Patient Safety Goal for Suicide
Prevention. R3 Report: Requirement, Rationale, Reference.
2018; 18. Available from: https://www.jointcommission.org/
assets/1/18/R3_18_Suicide_prevention_HAP_BHC_5_6_19v5.
pdf. Accessed December February 12, 2020

www.psychosomaticsjournal.org 705

Suicide Risk Screening in Rural Primary Care

18. U.S. Department of Health and Human Services (HHS)
Office of the Surgeon General and National Action Alliance
for Suicide Prevention: 2012 National strategy for suicide
prevention: goals and objectives for action. Washington, DC:
HHS; 2012

19. Hirsch JK, Cukrowicz KC: Suicide in rural areas: an updated
review of the literature. J Rural Ment Health 2014; 38:65–78

20. Ogbeide S, Stermensky G II, Rolin S: Integrated primary
care behavioral health for the rural older adult. Pract Innovat
2016; 1:145–153

21. Hill SK, Cantrell P, Edwards J, Dalton W: Factors influ-
encing mental health screening and treatment among women
in a rural south central Appalachian primary care clinic.
J Rural Health 2016; 32:82–91

22. Ahmedani BK, Simon GE, Stewart C, et al: Health care
contacts in the year before suicide death. J Gen Intern Med
2014; 29:870–877

23. Ahmedani BK, Stewart C, Simon GE, et al: Racial/ethnic
differences in health care visits made before suicide attempt
across the United States. Med Care 2015; 53:430–435

24. Horowitz LM, Bridge JA, Teach SJ, et al: Ask suicide-
screening questions (ASQ): a brief instrument for the pedi-
atric emergency department. Arch Pediatr Adolesc Med
2012; 166:1170–1176. 1177

25. Haney EM, O’Neil ME, Carson S, et al: Suicide risk factors
and risk assessment tools: a systematic review. Washington,
DC: Department of Veterans Affairs; 2012. Available from:
https://www.ncbi.nlm.nih.gov/books/NBK92671/

26. National Collaborating Centre for Mental Health
(NCCfMH): Self-harm: longer term management. In: NICE
clinical guidelines, Vol 133. Leicester, UK: British Psycho-
logical Society; 2012. Available from: https://www.ncbi.nlm.
nih.gov/books/NBK126777/

27. Ballard ED, Cwik M, Van Eck K, et al: Identification of at-
risk youth by suicide screening in a pediatric emergency
department. Prev Sci 2017; 18:174–182

28. Newton AS, Soleimani A, Kirkland SW, Gokiert RJ:
A systematic review of instruments to identify mental health
and substance use problems among children in the emergency
department. Acad Emerg Med 2017; 24:552–568

29. Horowitz LM, Snyder DJ, Boudreaux ED, et al: Validation
of the ask suicide-screening questions (ASQ) for adult med-
ical inpatients: a brief tool for all ages. Psychosomatics 2020.
https://doi.org/10.1016/j.psym.2020.04.008

706 www.psychosomaticsjournal.org

30. Brahmbhatt K, Kurtz BP, Afzal KI, et al: Suicide risk
screening in pediatric hospitals: clinical pathways to address a
global health crisis. Psychosomatics 2019; 60:1–9

31. Bae J, Encinosa WE: National estimates of the impact of
electronic health records on the workload of primary care
physicians. BMC Health Serv Res 2016; 16:172

32. Etter DJ, McCord A, Ouyang F, et al: Suicide screening in
primary care: use of an electronic screener to assess suici-
dality and improve provider follow-up for adolescents.
J Adolesc Health 2017; 62:191–197

33. Lanzillo EC, Horowitz LM, Wharff EA, Sheftall AH,
Pao M, Bridge JA: The importance of screening preteens for
suicide risk in the emergency department. Hosp Pediatr 2019;
9:305–307

34. DeVylder JE, Ryan TC, Cwik M, et al: Assessment of
selective and universal screening for suicide risk in a pe-
diatric emergency department. JAMA Netw Open 2019;
2:e1914070

35. Fortney JC, Pyne JM, Turner EE, et al: Telepsychiatry
integration of mental health services into rural primary care
settings. Int Rev Psychiatry 2015; 27:525–539

36. Gamm L, Stone S, Pittman S: Mental health and mental
disorders—a rural challenge: a literature review. In:
Gamm LD, Hutchinson LL, Dabney BJ, Dorsey AM, (eds).
Rural healthy people 2010: a companion document to rural
health people 2010 (vol. 2, pp. 97-113). College Station, TX:
The Texas A&M University System Health Science Center,
School of Rural Public Health, Southwest Rural Health
Research Center; 2003.

37. Stanley B, Brown GK, Brenner LA: Comparison of the
safety planning intervention with follow-up versus usual care
of suicidal patients treated in the emergency department.
JAMA Psychiatry 2017; 75:894–900

38. Reger MA, Luxton DD, Tucker RP, et al: Implementation
methods for the caring contacts suicide prevention interven-
tion. Prof Psychol Res Pr 2017; 48:369–377

39. Alexopoulos GS, Reynolds CF 3rd, Bruce ML, et al.,
PROSPECT Group: Reducing suicidal ideation and
depression in older primary care patients: 24-month out-
comes of the PROSPECT study. Am J Psychiatry 2009;
166:882–890

40. Graham RD, Rudd MD, Bryan CJ: Primary care providers’
views regarding assessing and treating suicidal patients. Sui-
cide Life Threat Behav 2011; 41:614–623

Psychosomatics 61:6, November/December 2020

  • The Feasibility and Impact of a Suicide Risk Screening Program in Rural Adult Primary Care: A Pilot Test of the Ask Suicide …
    • Introduction
    • Methods
      • Baseline Phase
      • Intervention Phase
      • Measures
        • Feasibility
        • Impact of Screening Program Implementation on Suicide Risk Assessment and Detection
      • Data Analysis
    • Results
      • Sample
        • Baseline
        • Intervention
      • Responses to the Individual ASQ Items
      • Feasibility of Screening Using the ASQ and BSSA
      • Suicide Risk Assessment, Detection, and Disposition by Phase
    • Discussion
      • Feasibility of the ASQ/ASQ BSSA in Rural Primary Care
      • The Impact of the ASQ/BSSA Program on Screening and Detection Rates
      • Study Limitations
    • Conclusion
    • References

General Hospital Psychiatry 68 (2021) 52–58

Available online 13 November 2020
0163-8343/© 2020 Elsevier Inc. All rights reserved.

Validation of the ask suicide-screening questions (ASQ) with youth in
outpatient specialty and primary care clinics

Laika D. Aguinaldo a,*, Shayla Sullivant b, Elizabeth C. Lanzillo c, Abigail Ross d, Jian-Ping He c,
Andrea Bradley-Ewing b, Jeffrey A. Bridge e,f, Lisa M. Horowitz c, Elizabeth A. Wharff g,h

a Department of Psychiatry, University of California, San Diego, 9500 Gilman Drive (MC 0862), La Jolla, CA 92093, USA
b Children’s Mercy Kansas City, 2401 Gillham Rd, Kansas City, MO 64108, USA
c National Institute of Mental Health, 10 Center Drive, Bethesda, MD 20892, USA
d Graduate School of Social Service, Fordham University, 113 W 60th St #7, New York, NY 10023, USA
e The Abigail Wexner Research Institute at Nationwide Children’s Hospital, 575 Children’s Crossroad, Columbus, OH 43215, USA
f Department of Pediatrics, Ohio State University, 370 W. 9th Avenue, Columbus, OH 43210, USA
g Department of Psychiatry, Boston Children’s Hospital, 300 Longwood Avenue, Boston, MA 02115, USA
h Department of Psychiatry, Harvard Medical School, 401 Park Drive, Boston, MA 02215, USA

A R T I C L E I N F O

Keywords:
Screening
Prevention
Instrument validation

A B S T R A C T

Objective: Validate the Ask Suicide-Screening Questions (ASQ) with youth in outpatient specialty and primary
care clinics.
Method: This is a cross sectional instrument validation study assessing the validity of the ASQ with respect to the
standard criterion, Suicidal Ideation Questionnaire (SIQ/SIQ Jr.). The sample included 515 English speaking
youth ages 10–21 years old from outpatient specialty and primary care clinics. ASQ sensitivity, specificity,
positive and negative predictive values (PPV/NPV), positive and negative likelihood ratios, c statistic and
respective receiver operating characteristic curves were assessed.
Results: A total of 335 outpatient specialty and 180 primary care clinic participants completed the study. In
outpatient specialty clinics, the ASQ showed a sensitivity of 100.0% (95% CI: 80.5–100.0%), specificity of 91.2%
(95% CI: 87.5–94.1%), and NPV of 100.0% (95% CI: 98.7–100.0). In the primary care clinic, the ASQ showed a
sensitivity of 100.0% (95% CI: 59.0–100.0%), specificity of 87.9% (95% CI: 82.0–92.3%), and NPV of 100.0%
(95% CI: 97.7–100.0). Forty-five (13.4%) outpatient specialty clinic participants and 28 (15.6%) primary care
clinic participants screened positive for suicide risk on the ASQ.
Conclusions: The ASQ is a valid screening tool for identifying youth at elevated suicide risk in outpatient clinical
settings.

1. Introduction

Suicide is an increasingly devastating public health problem in the
United States (U.S.) and the current rate is the highest it has been in
decades [1,2]. In the U.S., the age-adjusted suicide rate increased by
33% from 10.5 to 14.0 per 100,000 between 1999 and 2017 [3]. While
suicide is a leading cause of death across age groups, suicidal thoughts
and behaviors in youth are a major public health problem and warrant
particular attention. In 2017, suicide was the second leading cause of
death among 10–24-year-olds and 6769 youth ages 10 to 24 died by
suicide in the U.S [4]. Intervention during adolescence presents a critical
opportunity to identify those at risk for suicide and potentially save

lives.
Early identification of patients at risk for suicide is a key prevention

strategy; however, high-risk patients often go unrecognized by health-
care providers. Death registry studies show that 80% of youth who died
by suicide had contact with a healthcare provider within months of their
death, and 38% of these youth (up to age 19) had visited a medical
provider in the 4 weeks before a death by suicide [5–7]. Youth with
medical concerns (e.g., diabetes, sports medicine, medical illness) are at
increased risk for suicide and primary care is the de facto behavioral
health services and care system [21–24]. Recognizing the medical
setting as a critical venue to identify at-risk patients, the Joint Com-
mission issued a Sentinel Event Alert in 2016 recommending that all

* Corresponding author.
E-mail address: [email protected] (L.D. Aguinaldo).

Contents lists available at ScienceDirect

General Hospital Psychiatry

journal homepage: www.elsevier.com/locate/genhospsych

https://doi.org/10.1016/j.genhosppsych.2020.11.006
Received 10 September 2020; Received in revised form 7 November 2020; Accepted 9 November 2020

General Hospital Psychiatry 68 (2021) 52–58

53

medical patients be screened for suicide risk in medical settings [8].
There are several screening instruments healthcare providers can use

for detecting suicide risk (e.g. Columbia Suicide Severity Rating Scale
[33], the Patient Safety Screener [34] and the Suicide Behaviors Ques-
tionnaire Revised [35]), however, only the Ask Suicide-Screening
Questions (ASQ) [9,36] was developed specifically for medical pa-
tients. Most tools developed for mental health screening for outpatient
and primary care clinics are either costly because they require a great
deal of time and special training or software to administer, or are
intended to screen for depression [10,25,26].

The Ask Suicide-Screening Questions (ASQ) [9], a 4-item screening
instrument, is a brief suicide risk screening tool developed for both
medical and psychiatric patients presenting to a pediatric emergency
department [9]. To make sure the tool was valid for other settings, the
ASQ was then validated in inpatient medical /surgical units36. This
study followed previous validation studies using a standard criterion
measure that predicts suicide risk by assessing proxy measures of future
suicidal behavior [9]. For this purpose, we chose the Suicidal Ideation
Questionnaire (SIQ/SIQ Jr.), a longer instrument that measures severity
of suicidal ideation (a strong predictor of suicidal behavior [27]). The
SIQ/SIQ Jr. has demonstrated high reliability (SIQ: r = 0.97; SIQ-JR: r =
0.94), validity, and predictive ability [10,12] using the standard crite-
rion the Suicidal Behavior Interview, a semi-structured clinical inter-
view that assesses current and past suicidal behavior [8]. The
concordance of the ASQ with the SIQ/SIQ Jr. in identifying individuals
with elevated risk of suicide has previously been assessed for pediatric
ED patients. In the pediatric emergency department ASQ development
study, the ASQ was found to have high sensitivity (96.9%) and speci-
ficity (87%), and high negative predictive values for both medical
(99.7%) and psychiatric patients (96.9%) [9]. A recent study has also
shown the ASQ to improve the ability to predict future suicide-related
behaviors (i.e., suicide-related emergency department visits and sui-
cide deaths) beyond clinical judgment alone in both pediatric emer-
gency department patients presenting with psychiatric problems as well
as medical emergencies [11]. However, the ASQ has not been validated
in outpatient non-emergency settings. It is important to assess its val-
idity in these settings because of potential differences between emer-
gency and non-emergency clinical settings in the context of the
screening (e.g., administered by a triage nurse in a relatively stressful
environment with the youth completing the screen while experiencing
an urgent medical issue, versus in a more routine clinical setting and
with less medical urgency) as well as potential differences in the
respective samples (e.g., patients treated in emergency departments may
have more acute or severe medical and psychological issues).

The primary aim of this study was to test the validity of the ASQ with
youth in non-emergency settings of outpatient specialty and primary
care clinics to identify individuals with elevated suicide risk, using the
SIQ/SIQ Jr. as the standard criterion measure.

2. Methods

2.1. Ethics statement

This study was approved by scientific and institutional review boards
at the National Institute of Mental Health and the individual study sites.
For participants younger than 18 years of age, we obtained written or
verbal informed consent from the parent/guardian and written informed
assent from the participant. For participants aged 18 and older, we ob-
tained written informed consent. Participant responses were kept
confidential; however, parents were notified if their minor child
screened at risk for suicide. All participants who screened positive for
suicide risk received an additional suicide risk assessment administered
by a healthcare provider.

2.2. Study sites

Data came from two independent samples at two urban tertiary care
academic hospitals, with study site 1 consisting of four outpatient spe-
cialty clinics at Children’s Mercy Hospital Kansas City (diabetes, endo-
crinology, sports medicine, orthopedics) in the Midwest of the U.S.
Study site 2 consisted of one primary care clinic at Boston Children’s
Hospital in the Northeastern U.S.

2.3. Instruments

Ask Suicide-Screening Questions (ASQ)9. The ASQ, is a four-item
nonproprietary, self-report instrument intended to identify individuals
at elevated risk of suicide. The four ASQ items assess major facets of
established suicide risk factors, including suicidal ideation, perceived
burdensomeness on others and past history of suicide attempt. Specif-
ically, the four items on the ASQ are; (1) In the past few weeks, have you
wished you were dead? (2) In the past few weeks, have you felt that you
or your family would be better off if you were dead? (3) In the past week,
have you been having thoughts about killing yourself? and (4) Have you
ever tried to kill yourself? In this study, a “yes” response to any of the
four questions constitutes a positive screen for suicide risk requiring
further assessment from a healthcare provider, per usual care. Partici-
pants who screened positive for suicide risk were then asked an acuity
question, “Are you having thoughts of killing yourself right now?”.
Participants who responded yes to the acuity question were considered
an acute positive screen for suicide risk, whereas participants who
responded “no” to the acuity item were considered non-acute positive
screens.

Suicidal Ideation Questionnaire (SIQ/SIQ Jr.)10[28]. While the ASQ
measures perceived burdensomeness, suicidal ideation and attempt, the
Suicidal Ideation Questionnaire is a self-report instrument measuring
severity of suicidal ideation in adolescents. In this study, the 30-item SIQ
was administered to participants 15 years of age and older and the 15-
item SIQ Jr. was administered to younger participants. Each item on
the SIQ begins with “I thought…”, “I wondered…”, “I wished…” fol-
lowed by a statement indicative of suicidal ideation. Respondents indi-
cate the frequency with which they had that particular thought within
the last month on a scale from 1 (“I never had this thought”) to 7
(“Almost every day”). The items range from general thoughts indicating
a general wish to not be alive (e.g., “I wish I was never born”) to specific
thoughts/plans about ending one’s life (e.g., “I thought of when I would
kill myself”). The SIQ contains eight “critical” items (six on the SIQ Jr.)
designated as such because they directly assess serious thoughts of self-
destructive behavior: (1) “I thought about killing myself,” (2) “I thought
about how I would kill myself”, (3) “I thought about when I would kill
myself”, (4) “I thought about what to write in a suicide note”, (5) “I
thought about writing a will”, (6) “I thought about telling people I plan
to kill myself”, (7) “I thought about how easy it would be to end it all”,
and (8) “I thought if I had the chance I would kill myself”. A participant
with a total score of 41 or greater (31 or greater for the SIQ Jr.) is
considered at risk for suicide. If a participant responds positively to 3 or
more critical items (2 or more on SIQ Jr.), they are also considered to be
positive for suicide risk on this measure [10].

Demographic/Clinical Variables Questionnaire. Participants completed
a survey containing questions on demographic and clinical information
(e.g. gender, age, diagnosis, reason for visit), history of medical or
psychiatric illness, and prior healthcare usage, and opinions about sui-
cide risk screening.

2.4. Procedure

We prospectively enrolled a convenience sample of 10- to 21-year-
old patients from two different children’s hospitals, November 2015
through July 2017. Exclusion criteria (assessed informally by parent/
guardian (minors) or self-report (adult participants ≥18) included: (1)

L.D. Aguinaldo et al.

General Hospital Psychiatry 68 (2021) 52–58

54

developmental disability, cognitive impairment, or communication
disorder such that the patient was unable to comprehend questions/
communicate answers; (2) medical illness suggesting that the patient
was not physiologically stable enough to be approached; (3) parent/
legal guardian of patients younger than 18 years unavailable for con-
sent; and (4) non-English speaking parents/guardians and/or patients.

2.4.1. Site 1, outpatient specialty clinics
Site 1 recruited patients who presented for an outpatient specialty

visit to diabetes, endocrinology, sports medicine, or orthopedic clinics.
Patients were approached for recruitment and enrolled upon check-in
for their visit. Participants completed the study interview either before
or after being seen by their provider. A trained research assistant (RA)
with a minimum of a bachelor’s degree verbally administered the ASQ,
the SIQ/SIQ Jr., and other measures to participants, and responses were
entered into REDCap [13] via tablet computers.

2.4.2. Site 2, primary care clinic
Site 2 recruited patients who presented for an outpatient visit to

primary care. Patients were approached for recruitment and enrolled
upon check-in for their visit. Participants completed their primary care
appointment with their provider and then the study interview began. A
trained master’s level research social worker (RSW) verbally adminis-
tered the ASQ and the SIQ/SIQ Jr. to participants. Responses were
recorded via paper instruments and subsequently entered into Microsoft
Excel.

Sites 1 and 2 employed essentially the same procedures. To control
for possible fatigue order effects, RA/RSWs alternated the order in
which the ASQ and SIQ/SIQ Jr. were administered. In addition to the
ASQ and SIQ/SIQ Jr., participants completed a demographic question-
naire that included questions about sociodemographic characteristics,
history of medical and psychiatric illness, prior health care usage, and
opinions about suicide risk screening. Interviews were conducted
without the parent/guardian in the room, but participants under 18
years of age were told that if the RA/RSWs had any concerns about their
safety, their parents would be notified, and pertinent information would
be shared with the provider(s). As a safety measure, any participant who
responded positively to any one of the 4 ASQ screening questions, or
scored positive on the SIQ/SIQ Jr., or responded positively to any of the
8 SIQ/SIQ Jr. critical items required a further suicide risk assessment,
which was conducted per standard of care for suicide risk at both study
sites. Study participants were thanked for their participation, received a
$10 gift card and were given a suicide prevention resource sheet that
included facts, warning signs, and information about where to get help
for suicidal ideation (including the National Suicide Prevention Lifeline
and the Crisis Text Line) in addition to useful websites (e.g. American
Foundation for Suicide Prevention, Suicide Prevention Resource
Center).

2.5. Data analysis

Sample size estimates were conducted using general sample size
calculation for proportions [14]. We applied the central limit theorem
with a sensitivity of 96.9%, prevalence rate of 15.8% [8,15,16] and a
maximum acceptable margin of error d = 3%. The total number of
participants needed was 514.

A total of 918 patients were approached during the study period
between the 2 sites; 827 patients (90.1%) were eligible for participation;
525 (response rate = 63.5%) consented to participate. Seven patients
had completed the consent/assent process and attended their outpatient
specialty or primary care appointment, however did not complete the
study interview for personal reasons (e.g. time) and another three par-
ticipants were excluded due to RA/RSW error in conducting study
interview or REDCap data entry and thus were excluded from analyses.
All remaining participants (total n = 515, comprised of 335 outpatient
specialty clinic participants and 180 primary care participants)

completed the screening protocol and were included in the analysis. (See
Fig. 1).

We examined the distributions of demographic characteristics and
endorsement of four individual ASQ items by study site. Between-group
differences (diabetes/endocrine v. sports medicine/orthopedic; outpa-
tient specialty clinics v. primary care) were assessed using t-test for age
(treated as a continuous variable), and chi-square tests or fisher’s exact
tests for categorical variables (sex/gender, race/ethnicity and insurance
status). Traditional validity statistics were estimated in the package ‘Epi’
[17], a statistical analysis package used in epidemiology, including
sensitivity, specificity, positive predictive value and the negative pre-
dictive value, LR +, LR –, area under curve (AUC) and the kappa sta-
tistic, where SIQ/SIQ Jr. used as ‘true’ test and ‘ASQ’ as ‘screen’ test
[17,20]. Multivariate binary logistic regression modeling was used to
examine the associations of suicide risk as detected by the ASQ (i.e.,
elevated risk present or absent) with the demographic, clinical and other
variables (i.e., age, sex, race/ethnicity, insurance status, and site)
entered into the model simultaneously. Adjusted odds ratios were the
exponentiated values of multivariate logistic regression coefficients.
Statistical significance was based on two-sided tests evaluated at the p =
0.05 level of significance. Data preparation and data analyses were
performed in R, version 1.1.442 [18].

3. Results

Study participant characteristics are shown in Table 1. Approxi-
mately half of the participants (52.8%) were female, 39.6% were White,
19.1% were Black, and the mean age was 15.3 (SD 2.7) years. Most of
participants (62.9%) reported having government-sponsored insurance.
Compared to those from outpatient specialty sites, participants from the
primary care site were more likely to be older, female, non-white and
have private insurance. The indicators of the clinical validity of the ASQ,
as measured by the concordance of the ASQ with the SIQ/SIQ Jr. in
identifying individuals at elevated risk of suicide, are shown by study
site in Table 2.

Fig. 1. Sample Flowchart.

L.D. Aguinaldo et al.

General Hospital Psychiatry 68 (2021) 52–58

55

3.1. Site 1 outpatient specialty clinics

Forty-five (13.4%) of 335 participants screened positive for suicide
risk on the ASQ. Of the participants who were positive on the ASQ, four
participants were acute positive screens (1.2% of total sample). In the
outpatient specialty clinics, the ASQ had a sensitivity of 100.0% (95%
CI: 80.5–100.0%), specificity of 91.2% (95% CI: 87.5–94.1%), and
negative predictive values at 100.0% (95% CI: 98.7–100.0). The positive
LR was 11.4 (95% CI, 8.0–16.2), indicating that a positive screen on the
ASQ was 11.4 times more likely to be seen in someone actually at sui-
cidal risk than in someone not at risk. The 4-item ASQ screen had sub-
stantial agreement with the SIQ/SIQ Jr. for detecting suicide risk (AUC
= 95.6%).

3.2. Site 2 primary care clinic

Twenty-eight (15.5%) of the 180 participants screened positive for
suicide risk on the ASQ. All 28 positive screens were non-acute positive
screens. In the primary care clinic, the ASQ had a sensitivity of 100.0%

(95% CI: 59.0–100.0%), specificity of 87.9% (95% CI: 82.0–92.3%), and
negative predictive values at 100.0% (95% CI: 97.7–100.0). The positive
LR was 8.2 (95% CI, 7.7–13.1), indicating that a positive screen on the
ASQ was 8.2 times more likely to be seen in someone actually at suicidal
risk than in someone not at risk. The 4-item ASQ screen had substantial
agreement with the SIQ/SIQ Jr. for detecting suicide risk (AUC =
93.9%).

We examined the association between screening positive for suicide
risk, as determined by the ASQ, and demographic variables (age, sex/
gender, race, insurance status and site) in Table 3. For every one unit
(year) increase in age, the odds of screening positive increased by 1.166
(95% CI: 1.036–1.316). Odds of screening positive for females were
2.358 (95% CI: 1.341–4.292) times higher compared to the odds for
males. Odds of screening positive for publicly-insured were 0.540 (95%
CI: 0.288–0.969) times higher compared to the odds for participants
with private insurance.

Table 1
Sample characteristics of clinic settings and study sites.

Characteristics No. (%)

Site 1 Site 2 Tests for Group Differences

Outpatient Specialty (1) vs. (2) (3) vs. (4)

Diabetes/
Endocrine

Sports Med/
Ortho

Subtotal Primary
Care

Dia/Endo vs.
SportsOrtho

Outpatient Specialty vs. Primary
Care

Total
Sample

(1) (2) (3) (4) p-values

Sample Size 515 197 138 335 180
Age in years, MEAN

(SD)
15.3 (2.7) 14.6 (2.7) 14.0 (2.1) 14.4 (2.4) 17.2 (2.1) 0.042* <0.0001***

Sex 0.029* 0.233
Female 272 (52.8) 107 (54.3) 60 (43.5) 167

(49.9)
105 (58.3)

Male 231 (44.9) 80 (40.6) 76 (55.1) 156
(46.6)

75 (41.7)

Transgender 9 (1.7) 9 (4.6) 0 (0.0) 9 (2.7) 0 (0.0)
Unknown 3 (0.6) 1 (0.5) 2 (1.4) 3 (0.9) 0 (0.0)

Race/ethnicity 0.383 <0.0001***
White 258 (50.1) 132 (67.0) 98 (71.0) 230

(68.7)
28 (15.6)

Black 95 (18.4) 22 (11.2) 15 (10.9) 37 (11.0) 58 (32.2)
Hispanic/Latino 105 (20.4) 25 (12.7) 14 (10.1) 39 (11.6) 66 (36.7)
Asian 36 (7.0) 11 (5.6) 2 (1.4) 13 (3.9) 23 (12.8)
Other/Unknown 21 (4.1) 7 (3.6) 9 (6.5) 16 (4.8) 5 (2.8)

Insurance 0.785 0.726
Private 324 (62.9) 123 (62.4) 85 (61.6) 208

(62.1)
116 (64.4)

Public 170 (33.0) 66 (33.5) 40 (29.0) 106
(31.6)

64 (35.6)

None/Unknown 21 (4.1) 8 (4.1) 13 (9.4) 21 (6.3) 0 (0.0)

Note: “*” p ≤ 0.05, “***” p ≤ 0.001.

Table 2
Concordance of SIQ/SIQ Jr. and suicide risk assessed by ASQ for total sample and by study site.

Suicide Risk Sample
Size

ASQ
(Screen)

SIQ/SIQ Jr.
(True)

Sensitivity Specificity PPV NPV LR+ LR- Kappa AUC

Positive
(n, %)

Positive (n,
%)

% (95% CI) % (95% CI) % (95% CI) % (95% CI) % (95% CI) % (95%
CI)

Site I:
Outpatient
Specialty

335 45 (13.4) 17 (5.0) 100.0
(80.5–100.0)

91.2
(87.5–94.1)

37.8
(23.8–53.5)

100.0
(98.7–100.0)

11.4
(8.0–16.2)

0.0
(0.0-
NaN)

0.512 0.956

Site II: Primary
Care

180 28 (15.5) 7 (3.9) 100 (59.0–1) 87.9
(82.0–92.3)

25.0
(10.7–44.9)

100.0
(97.7–100.0)

8.2
(5.5–12.3)

0.0
(0.0-
NaN)

0.360 0.939

Note: NaN = Not a number [20].

L.D. Aguinaldo et al.

General Hospital Psychiatry 68 (2021) 52–58

56

4. Discussion

The ASQ appears to be a valid instrument for detecting elevated
suicide risk in pediatric outpatient specialty or primary care clinics
when compared to the standard criterion SIQ/SIQ Jr. This study
demonstrated that the ASQ appeared to have good content validity. The
four questions together assessed established suicide risk factors,
including suicidal ideation, burdensomeness, and the most critical risk
factor for future suicidal behavior—a previous suicide attempt. Positive
responses to one or more of these four questions identified 100% of the
youth at risk for suicide, as assessed by the SIQ/SIQ Jr., a much longer
criterion standard proprietary instrument typically administered by
mental health clinicians. Of note, the high specificity demonstrated by
the four questions, reflecting the ASQ’s ability to minimize false posi-
tives, or to efficiently and correctly identify youth who are currently not
at risk for suicide (~91% outpatient specialty/~88% primary care), is
important for a busy outpatient specialty and primary care clinic with
limited and sometimes overburdened mental health resources. Given the
consequences of failing to detect elevated risk for suicide, the high NPV
(100.0% outpatient specialty/100% primary care), reflects the ability of
the ASQ to avoid false negatives, or to correctly identify youth who are
actually at elevated risk for suicide. Of note, the fairly high specificity
demonstrated by the four questions, reflecting the ASQ’s ability to
efficiently and correctly identify youth who are currently not at risk for
suicide (~91% outpatient specialty/~88% primary care), is important
for a busy outpatient specialty and primary care clinic with limited and
sometimes overburdened mental health resources. In contrast, the low
PPV, or the proportion of positive screens on the ASQ that are also
positive on the SIQ (~38% outpatient specialty/ 25% primary care),
indicates that positive screens do not always reflect current severity of
suicidal ideation (as assessed by the SIQ). This discordance is possibly
attributable to the time parameter (lifetime) associated with item 4 of
the ASQ, which assesses past suicide attempt (e.g. “Have you ever tried
to kill yourself?). If a respondent has ever attempted suicide, they would
screen positive on any subsequent ASQ screens, regardless of changes in
suicidal ideation or actual suicide risk over time. Although a recent
suicide attempt is certainly indicative of current acute suicide risk, one
that occurred several years prior raises the patient’s baseline risk for
suicide but may not be very informative about current acute suicide risk.
Therefore, patients who endorse Item 4 on the ASQ should be asked
when the suicide attempt(s) occurred, and this information should be
considered with respect to responses on Items 1–3, so that current risk
can better be assessed.

Overall, 13.4% of the outpatient specialty clinic participants and
15.5% of the primary care

participants screened positive for non-acute suicide risk (outpatient
specialty n = 45, primary care n = 28), as assessed by the ASQ. Acute, or
imminent risk for suicide was detected in only 4 participants in the
outpatient specialty participants (1.2%) but was not detected for any
participants in the primary care sample. This relatively small number of
participants will likely not overburden a busy outpatient specialty or
primary care clinic. Proactively identifying participants at elevated
suicide risk with a brief screening instrument may protect a large
number of youth in crisis.

The ASQ is intended to be utilized as a primary screening tool to
identify patients who require further evaluation. Rational management
of patients who screen positive can be conducted in a way that does not
impede medical care, overwhelm the patient or overburden a busy
outpatient pediatric practice. As found in this sample, the majority of
patients screened will be negative and the ones that screen positive will
be mostly non-acute positives in need of mental health linkage. It will be
important to follow evidence-informed clinical pathways, such as the
one developed by Brahmbhatt and colleagues, who recommend that
non-acute positive screens be followed up with a brief suicide safety
assessment to determine if any safety precautions or a more in-depth
psychiatric evaluation are needed [32]. Conversely, acute positive
screens, which are extremely rare among medical outpatients, as evi-
denced in this sample, do not require a brief suicide safety assessment
because they are automatically considered an emergency and will
require full safety precautions and full mental health/safety evaluation.

These results should be interpreted with consideration of a few
limitations of this study. First, there are potential differences between
our convenience sample and the general population of youth in non-
emergency outpatient settings. As these differences would presumably
impact participants’ responses on the ASQ and the SIQ equally, they
might affect the percent of patients that screen positive for suicide risk,
but seem unlikely to impact the validation metrics, which are the focus
of this study. One potential difference is that our sample was in urban,
tertiary care academic hospitals and did not include other outpatient
specialty and primary care clinics. However, we are aware of no a priori
reason to think that the validation metrics we obtained for the ASQ
would not generalize to other clinics. In addition, we used convenience
sampling, which may have introduced selection bias into our findings.
As we were not able to collect data on non-participants, we could not
assess this possibility statistically. If bias exists, it is likely that the
percentage of non-study outpatient samples screening positive for sui-
cide risk would be lower than with our convenience sample, but again,
this potential difference likely would not affect the validation metrics.

Second, there may have been an effect of the timing of study
participation (i.e. after their clinic appointment in primary care and
either before or after their clinic appointment in outpatient specialty
care) on the likelihood of screening positive for suicide risk. Participants
may be more or less likely to report suicidality before or after an
appointment. For example, appointment-related anticipatory anxiety
and time pressure, or post-appointment fatigue, could theoretically
affect results. In an actual clinical setting, patients would be screened
before or during the appointment so that clinicians can review the re-
sponses. Data on the timing of the screen was not available, so we could
not address the possibility of a timing effect. Again, if it is the case that
timing of the screener influences the likelihood of endorsing suicide-
related questions, presumably it affects the ASQ and the SIQ similarly,
as they were administered at the same time (either before or after the
appointment). Thus, this potential timing issue might affect the rate of
positive screenings but probably not the validation metrics.

Third, there may have been a fatigue effect by asking the participants
repeatedly about suicide ideation; however, any fatigue would have
affected the two instruments equally as the order of the ASQ and SIQ/
SIQ Jr. were rotated. Lastly, the tool used to validate the ASQ, the SIQ/
SIQ-JR, is primarily intended to identify risk for suicide ideation and
may not be predictive of suicide behavior.

Table 3
The associations of risk for suicide as detected by the ASQ and demographic
characteristics Note: “.” Denotes 0.1 significance (trending towards), “*” De-
notes ≤0.05 significance, “**” Denotes ≤0.01 significance, “***” Denotes
≤0.001 significance.

Variable OR 95% CI p-value

Lower Upper

Site
Primary Care Reference
Outpatient 1.117 0.544 2.293 0.762

Age 1.166 1.036 1.316 0.012*
Sex

Male Reference
Female 2.358 1.341 4.292 0.004**

Race
White Reference
Black 0.525 0.206 1.231 0.154
Other/Unknown 0.979 0.49 1.925 0.951

Insurance
Private Reference
Public 0.540 0.288 0.969 0.045*

L.D. Aguinaldo et al.

General Hospital Psychiatry 68 (2021) 52–58

57

5. Conclusion

The ASQ is a brief, valid and useful tool for screening for suicide risk
in pediatric outpatient specialty and primary care clinics, which has
been recommended by the Joint Commission and the American Acad-
emy of Pediatrics. The ASQ instrument and materials to guide imple-
mentation can be found in the ASQ Toolkit, a publicly available resource
that can facilitate the detection of youth at risk for suicide (www.nimh.
nih.gov/asq).

Future studies measuring the impact of suicide risk screening with
youth in outpatient specialty and primary care clinics on outcomes such
as linkage with mental health services and future suicidal behavior are
warranted. Additionally, evaluating the acceptance of such screening
instruments by clinicians and the costs associated with implementing
universal suicide risk screening in outpatient specialty and primary care
clinics would inform ASQ implementation strategies. A secondary
exploration of the ASQ on an item level would provide nuanced infor-
mation regarding the endorsement of individual items for the same
subject and between subjects, allowing us to better understand risk for
suicide within different hospital settings. Moreover, it would be useful to
examine the utility of the ASQ in conjunction with brief suicide in-
terventions similar to the Screening and Brief Intervention to Treatment
model [19], given the push for integrated behavioral health in these
settings.

Author contributions

Drs. Aguinaldo, He, Bridge, and Horowitz had full access to all the
data in the study and take responsibility for the integrity of the data and
the accuracy of the data analysis. Study concept and design: Drs. Hor-
owitz, Bridge, Wharff, Sullivant, Ross, Aguinaldo and Ms. Bradley-
Ewing. Acquisition of data: Drs. Horowitz, Wharff, Sullivant, Ross,
Aguinaldo and Ms. Bradley-Ewing.

Analysis and interpretation of data

Drs. Aguinaldo, He, Horowitz, Bridge and Wharff. Drafting of the
manuscript: Horowitz, Bridge, Wharff, Ross, Aguinaldo, Sullivant,
Bradley-Ewing and Lanzillo. Critical revision of the manuscript for
important intellectual content: Horowitz, Bridge, Wharff, Ross, Agui-
naldo and Sullivant. Statistical analysis: Bridge, He and Aguinaldo.
Administrative, technical, and material support: Horowitz, Bridge,
Wharff, Lanzillo, Aguinaldo, Bradley-Ewing and Sullivant.

Acknowledgements

The authors would like to thank all the patients, nurses, mental
health and medical teams that helped make the study run smoothly. The
authors would also like to acknowledge and thank the following people
who were instrumental to the success of the study, including Annabelle
Mournet, Maryland Pao, Mary Tipton, Jeanne Radcliffe, and Shirley Yu.
The views expressed in this abstract are those of the authors and do not
reflect the official policy of the National Institute of Mental Health,
National Institutes of Health, or the Department of Health and Human
Services. Research reported in this publication was supported by the
National Institutes of Health under Award Number T32AA013525
(Aguinaldo), Program for Patient Safety and Quality at Boston Chil-
dren’s Hospital (Wharff, Ross), and the Children’s Research Institute at
Children’s Mercy Kansas City (Sullivant, Bradley-Ewing). Dr. Bridge
receives research support from the National Institute of Mental Health
(NIMH), the Centers for Disease Control and Prevention (CDC), and the
Patient-Centered Outcomes Research Institute (PCORI). Dr. Bridge is a
member of the Scientific Advisory Board of Clarigent Health. Dr. Hor-
owitz is supported by the National Institute of Mental Health
(ZIAMH002922).

References

[1] World Health Organization. Preventing suicide: a global imperative. World Health
Organization; 2014.

[2] World Health Organization. Suicide data World Health Organization. http://www.
who.int/mental_health/prevention/suicide/suicideprevent/en/; 2017 (Accessed
December 14, 2017).

[3] Hedegaard H, Curtin SC, Warner M. Suicide Mortality in the United States,
1999–2017. NCHS Data Brief, no 330. Hyattsville, MD: National Center for Health
Statistics; 2018.

[4] Hedegaard H, Curtin SC, Warner M. Suicide Rates in the United States Continue to
Increase NCHS Data Brief, no 309. Hyattsville, MD: National Center for Health
Statistics; 2018. Available from, https://www.cdc.gov/nchs/data/databriefs/d
b309.pdf.

[5] Luoma JB, Martin CE, Pearson JL. Contact with mental health and primary care
providers before suicide: a review of the evidence. Am J Psychiatry 2002;159(6):
909–16.

[6] Rhodes A, Khan S, Boyle M, Tonmy L, Wekerle C, Goodman D, et al. Sex differences
in suicides among children and youth: the potential impact of help-seeking
behaviour. Can J Psychiatry 2013;58(5):274–82.

[7] Ahmedani BK, Simon GE, Stewart C, Beck A, Waitzfelder BE, Rossom R, et al.
Health care contacts in the year before suicide death. J Gen Intern Med 2014;29(6):
870–7.

[8] The Joint Commission. Detecting and Treating Suicide Ideation in All Settings.
February 24, 2016.

[9] Horowitz LM, Bridge JA, Teach SJ, Ballard E, Klima J, Rosenstein D, et al. Ask
Suicide-Screening Questions (ASQ): a brief instrument for the pediatric emergency
department. Arch Pediatr Adolesc Med 2012;166(12):1170–6.

[10] Reynolds WM. Suicidal Ideation Questionnaire (SIQ). Odessa, FL: Psychological
Assessment Resources; 1987.

[11] DeVylder JE, Ryan TC, Cwik M, Wilson ME, Jay S, Nestadt PS, et al. Assessment of
selective and universal screening for suicide risk in a pediatric emergency
department. JAMA Netw Open 2019;2(10):e1914070.

[12] Keane EM, Dick RW, Bechtold DW, Manson SM. Predictive and concurrent validity
of the suicidal ideation questionnaire among American Indian adolescents.
J Abnorm Child Psychol 1996;24(6):735–47.

[13] Harris P, Taylor R, Thielke R, Payne J, Gonzalez N, Conde J. Research electronic
data capture (REDCap) – A metadata-driven methodology and workflow process for
providing translational research informatics support. J Biomed Inform 2009;42:
377–81.

[14] Hajian-Tilaki K. Sample size estimation in diagnostic test studies of biomedical
informatics. J Biomed Inform 2014;48:193–204.

[15] Reynolds WM, Mazza JJ. Assessment of suicidal ideation in inner-city children and
young adolescents: reliability and validity of the Suicidal Ideation Questionnaire-
JR. School Psychol Rev 1999;28(1):17–30.

[16] Zhang Y, Yip PS, Fu KW. Validation of the Chinese version of the Reynolds’ suicidal
ideation questionnaire: psychometric properties and its short version. Health Qual
Life Outcomes 2014;12:33.

[17] Carstensen B, Plummer M, Laara E, Hills M. Epi: A Package for Statistical Analysis
in Epidemiology. In: 2.32. Rpv. 2018.

[18] RCoreTeam. R: A language and environment for statistical computing. In: R
Foundation for Statistical Computing. 2013.

[19] Office of National Drug Control P. Screening, brief intervention, and referral to
treatment (SBIRT). Washington, D.C.]: Office of National Drug Control Policy
(ONDCP), Substance Abuse and Mental Health Services Administration (SAMHSA);
2012.

[20] Simel DL, Samsa GP, Matchar DB. Likelihood ratios with confidence: sample size
estimation for diagnostic test studies. J Clin Epidemiol 1991;44(8):763–70.

[21] Kessler R, Stafford D. Primary care is the de facto mental health system. In:
Collaborative medicine case studies: Evidence in practice. New York, NY, US:
Springer Science + Business Media; 2008. p. 9–21.

[22] Wang B, An X, Shi X, Zhang J-A. Management of endocrine disease: suicide risk in
patients with diabetes: a systematic review and meta-analysis. Eur J Endocrinol
2017;177(4):R169–81.

[23] Sabo D, Miller KE, Melnick MJ, Farrell MP, Barnes GM. High school athletic
participation and adolescent suicide: a Nationwide US Study. Int Rev Sociol Sport
2005;40(1):5–23.

[24] Druss B, Pincus H. Suicidal ideation and suicide attempts in general medical
illnesses. Arch Intern Med 2000;160(10):1522–6.

[25] Posner K, Brent D, Lucas C, Gould M, Stanley B, Brown G. The Columbia-Suicide
Severity Rating Scale: initial validity and internal consistency findings from three
multisite studies with adolescents and adults. Am J Psychiatry 2011;168(12):
1266–77.

[26] Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression
severity measure. J Gen Intern Med 2001;16(9):606–13.

[27] Klonsky ED, May AM, Saffer BY. Suicide, suicide attempts, and suicidal ideation.
Annu Rev Clin Psychol 2016;12:307–30.

[28] Reynolds WM. Development of a semistructured clinical interview for suicidal
behaviors in adolescents. Psychol Assess J Consult Clin Psychol 1990;2(4):382–90.

[32] Brahmbhatt K, Kurtz BP, Afzal KI, et al. Suicide risk screening in pediatric
hospitals: clinical pathways to address a Global Health crisis. Psychosomatics.
2019;60(1):1–9.

[33] Posner K, Brown GK, Stanley B, et al. The Columbia-suicide severity rating scale:
initial validity and internal consistency findings from three multisite studies with
adolescents and adults. Am J Psychiatry 2011;168(12):1266–77.

L.D. Aguinaldo et al.

General Hospital Psychiatry 68 (2021) 52–58

58

[34] Boudreaux ED, Jaques ML, Brady KM, Matson A, Allen MH. The patient safety
screener: validation of a brief suicide risk screener for emergency department
settings. Arch Suicide Res 2015;19(2):151–60.

[35] Osman A, Bagge CL, Gutierrez PM, Konick LC, Kopper BA, Barrios FX. The suicidal
behaviors questionnaire-revised (SBQ-R):validation with clinical and nonclinical
samples. Assessment. 2001;8(4):443–54.

[36] Horowitz LM, Wharff EA, Mournet AM, Ross AM, McBee-Strayer S, He J, et al.
Validation and feasibility of the Ask Suicide-Screening Questions (ASQ) among
pediatric medical/surgical inpatients. Hosp Pediatr 2020;10(9):750–7. https://doi.
org/10.1542/hpeds.2020-0087.

L.D. Aguinaldo et al.

  • Validation of the ask suicide-screening questions (ASQ) with youth in outpatient specialty and primary care clinics
    • 1 Introduction
    • 2 Methods
      • 2.1 Ethics statement
      • 2.2 Study sites
      • 2.3 Instruments
      • 2.4 Procedure
        • 2.4.1 Site 1, outpatient specialty clinics
        • 2.4.2 Site 2, primary care clinic
      • 2.5 Data analysis
    • 3 Results
      • 3.1 Site 1 outpatient specialty clinics
      • 3.2 Site 2 primary care clinic
    • 4 Discussion
    • 5 Conclusion
    • Author contributions
    • Analysis and interpretation of data
    • Acknowledgements
    • References

Patient Opinions About Screening for Suicide
Risk in the Adult Medical Inpatient Unit

Deborah J. Snyder, MSW
Elizabeth D. Ballard, PhD
Ian H. Stanley, BA
Erica Ludi, BS
Julie Kohn-Godbout, MSN, RN, PMHCNS-BC
Maryland Pao, MD
Lisa M. Horowitz, PhD, MPH

Abstract

As hospital clinicians and administrators consider implementing suicide risk screening on
medical inpatient units, patient reactions to screening can provide essential input. This post hoc
analysis examined patient opinions about screening for suicide risk in the medical setting. This
analysis includes a subsample of a larger quality improvement project designed to screen
medically hospitalized patients for suicide risk. Fifty-three adult medical inpatients at a clinical
research hospital provided opinions about suicide risk screening. A qualitative analysis of
responses to an opinion question about screening was conducted to identify major themes. Forty-
three (81%) patients supported screening medical inpatients for suicide risk. Common themes
emphasized asking patients directly about suicide, connection between mental/physical health, and
the role of screening in suicide prevention. Adult medical inpatients supported screening for
suicide risk on medical/surgical inpatient units. Behavioral health clinicians are uniquely poised to
champion suicide detection and intervention in the general medical hospital setting. Patient
opinions can be utilized to inform thoughtful implementation of universal suicide risk screening in
the medical setting.

Address correspondence to Lisa M. Horowitz, PhD, MPH, National Institute of Mental Health Intramural Research
Program, Bethesda, Maryland, USA. Email: [email protected]
Deborah J. Snyder, MSW, National Institute of Mental Health Intramural Research Program, Bethesda, Maryland, USA.
Elizabeth D. Ballard, PhD, National Institute of Mental Health Intramural Research Program, Bethesda, Maryland, USA.
Maryland Pao, MD, National Institute of Mental Health Intramural Research Program, Bethesda, Maryland, USA.
Ian H. Stanley, BA, Florida State University, Tallahassee, Florida, USA.
Erica Ludi, BS, Emory University, Atlanta, Georgia, USA.
Julie Kohn-Godbout, MSN, RN, PMHCNS-BC, Research and Practice Development, Clinical Center Nursing

Department, National Institutes of Health, Bethesda, Maryland, USA.

Journal of Behavioral Health Services & Research, 2016. 364–372. c) 2016 National Council for Behavioral Health (outside

364 The Journal of Behavioral Health Services & Research 44:3 July 2017

the USA). DOI 10.1007/s11414-016-9498-7

Introduction

Individuals with medical illnesses are at elevated risk for suicide.1–4 Diagnoses such as cancer,
diabetes, HIV/AIDS, lung disease, and gastric ulcers have been associated with increased rates of
suicidal thoughts, behaviors, and death by suicide.5–11 Symptoms and psychosocial effects of
medical illnesses, such as uncontrolled pain, poor prognoses, poor social support, and physical and
cognitive impairments, are also correlates of suicidal thoughts and behaviors.12–14 Furthermore,
death registry studies reveal that individuals who have killed themselves often visit their medical
providers months before their death.15 Therefore, settings that treat medically ill patients represent
an important venue for suicide detection and prevention.

In 2010, the Joint Commission (JC) issued a Sentinel Event Alert highlighting the need to detect
suicide risk in nonbehavioral health patients in medical settings.16 Over the past 18 years, nearly
1300 inpatient deaths by suicide have been reported to the JC. Most striking, approximately 25%
of these deaths occurred in nonbehavioral health settings such as the emergency department and
inpatient medical units.17 Although the JC has made recommendations to screen medical patients at
elevated risk, the specific implementation of suicide prevention strategies has been left up to the
discretion of hospitals.

In light of the JC recommendation to broaden screening for suicide risk to include nonbehavioral
health patients, determining patient acceptability of screening is a critical next step. Studies have
shown that pediatric patients in the emergency department favor suicide risk screening;18, 19

however, it is not known how adult patients hospitalized for medical reasons will react to questions
about suicide, particularly in the context of medical treatment. The objective of this paper is to
describe patients’ opinions and reactions to universal screening for suicide risk in an inpatient
medical hospital setting using qualitative analysis. Data were obtained from a larger quality
improvement project (QIP), Ask Suicide-Screening Questions to Everyone in Medical Settings
(asQ’em), which examined suicide risk screening in adult medical inpatients.20 The use of open-
ended questions allowed for identification of themes which can be used to assess feasibility and
inform how hospitals can implement suicide risk screening and other behavioral health initiatives.

Methods

Patient population

The current findings represent a subanalysis of data collected from the asQ’em QIP.20 The QIP,
which incorporated the suicide risk screen into standard of care, was determined to be Bexempt^
from IRB review by the NIH Office of Human Subjects Research. A convenience sample of 56
adult medical/surgical patients, age 18 years or older, who were admitted to one of three select
inpatient units at the National Institutes of Health (NIH) Clinical Research Center (CRC) during a
single week in January 2012, were included in this subanalysis. The NIH CRC is a 200-bed clinical
research hospital. Patients were excluded if they were unable to communicate fluently in English,
were cognitively impaired, or were experiencing acute worsening medical status.

Procedure

As part of the provision of standard care, nurses administered the asQ’em, a two-item suicide
risk screening tool, and a patient feedback survey to newly admitted patients on one of three
selected inpatient units (oncology, general medical/surgical, and infectious disease). The screening
was embedded into the nurses’ initial intake assessment; no patients declined answering the two
screening questions. The asQ’em was composed of the following items: (1) BIn the past month,
have you had thoughts about suicide?^ (Yes/No) and (2) BHave you ever made a suicide attempt?^

Patient Opinions About Screening for Suicide Risk SNYDER ET AL. 365

(Yes/No). If patients answered yes to either of these two questions, a follow-up question was asked
to assess acuity, BAre you having thoughts of suicide right now?^ (Yes/No). Of note, the asQ’em
tool is not yet a validated assessment instrument; the multisite validation study is underway. The
purpose of this analysis was to describe the opinions of medically ill adults about suicide risk
screening. A 15-item feedback survey was given verbally by the asQ’em data collector to all
patients who participated in the QIP. The following two questions are the focus of this subanalysis:
(1) BDo you think all medical/surgical patients in a hospital should be asked about suicide; why or
why not?^ and (2) BDo you have any additional comments?^ Further methodologies and the
quantitative results of asQ’em have been described elsewhere.20

Qualitative data analysis

Responses to these open-ended questions were transcribed verbatim and uploaded into NVivo
9.2 qualitative software for coding and content analysis. A group composed of a masters-level
clinical social worker (DJS), a research assistant (EL), and an advanced doctoral clinical
psychology student (EDB) used grounded theory and open coding, a technique of naming and
categorizing phenomena, to analyze the responses.21 Two members of the team (DJS and EL) used
the open coding procedure to code the comments and to identify themes that emerged from the
responses. A constant comparison method was used to refine the codes into larger themes. A third
coder (EDB) reviewed the responses and discussed with the team to further refine themes. Eleven
themes were established. Interrater agreement was 90.5%. All discrepancies were resolved by
discussion and consensus.

Results

Overall, 56 patients ranging in age from 18 to 89 were screened. Three patients were excluded
due to emergent medical status changes or early discharge from the hospital. Fifty-three patients
(95%) completed the feedback survey and were included in this subanalysis (see Table 1). Patients

Table 1
Demographics of QIP patients

Pilot patients

Demographics N (%) [total = 53]
Diagnosis

General med/surg 24 (45%)
Cancer 20 (38%)
Infectious disease 9 (17%)

Age, mean (SD) 48.3 (19.5) years
Range 18–89 years

Gender
Female 35 (66%)

Race
Caucasian 33 (62%)
African-American 13 (25%)
Other 7 (13%)

366 The Journal of Behavioral Health Services & Research 44:3 July 2017

were 66% female and 62% white; average age was 48.3 years (±19.5 years). Patients were
recruited from the following units: 45% general medical/surgical, 38% cancer, and 17% infectious
disease. Four of the 53 patients in this sample screened positive for suicide risk necessitating
follow-up mental health evaluation.

Of note, 98% of patients reported the experience of being screened for suicide risk as either
positive (79%) or neutral (19%). Only one patient reported the experience as negative; this patient
screened positive on the asQ’em screen. The remaining three patients who screened positive did
not report the experience as negative. Further, 81% of patients reported that they believed all
medical/surgical patients in a hospital should be screened for suicide risk; 9.5% disagreed, and
another 9.5% reported they Bdon’t know.^ Reactions to being screened for suicide risk in this
subsample are presented in Table 2.

The three most common themes that emerged from the qualitative analysis included the following:
(1) patients should be asked directly about suicide; (2) mental health should be an integral component in
the delivery of medical care; and (3) importance of intervening, protecting, and keeping patients safe in
a hospital setting. Descriptions of the themes as well as quotes from the patients follow.

Patient reactions to screening for suicide risk: major themes

The most commonly identified response was the importance of asking patients directly about
suicide in order to detect a problem which might otherwise have remained hidden (N=39; 74%).
Comments indicated that patients may not spontaneously offer information about suicidal thoughts
or behaviors unless asked openly.

BThe questions will help you help them. You can’t read a person’s mind. You don’t know what they are thinking until
you ask.^

– an 89-year-old man admitted to the oncology unit
BI think anyone should be asked. You never know who has thoughts. [You] don’t want to miss one. You could save a
life by asking.^
– a 20-year-old man admitted to the medical/surgical unit
BIt’s never crossed my mind, but you have to ask for people who may be hiding something.^
– a 45-year-old woman admitted to the oncology unit

Another major theme that emerged from this analysis was the notion that mental health should
be an integral component in the delivery of holistic care to the medical patient in a hospital setting
(N=31; 59%). Patients recognized that medical illness can significantly impact not only physical

Table 2
Overall reactions to suicide sreening in an inpatient hospital setting in subsample used in

qualitative analysis

Overall, how would you rate your experience of being screened for suicide: positive, neutral or
negative? N (%)
Positive 42 (79%)
Neutral 10 (19%)
Negative 1 (2%)

Do you think all medical/surgical patients in a hospital should be asked about suicide?
Yes 43 (81%)
No 5 (9.5%)
Don’t know 5 (9.5%)

Patient Opinions About Screening for Suicide Risk SNYDER ET AL. 367

well-being, but also psychological well-being. Many patients felt that both physical and emotional
functioning should be monitored in the hospital setting.

BI think it’s such an emotional strain to be in a situation where you are vulnerable. Hospitalization is when you feel
the most vulnerable and scared…. If you are predisposed [to suicidal thoughts], it could tip the scale and … this
could be the kind of intervention that helps.^
– a 48-year-old woman admitted to the medical/surgical unit
BIt’s important that it is coming from a health care professional. If they are in this setting [hospital] they are very ill.
Facing death, mortality. Asking and honesty are key.^
– a 43-year-old woman admitted to the infectious disease unit
BMedical patients may be stoic…. With all of the emotional stuff going on … patients may feel like they have to
keep it together….^
– a 48-year-old woman admitted to the medical/surgical unit

Patients also expressed their belief that asking suicide screening questions could lead to
intervention, protection, and safety (N=24; 42%).

BBecause I think that we know risk for suicide is a chronic illness. In a hospital, patients are more likely to be
depressed. It’s an opportunity for intervention.^
– a 32-year-old woman admitted to the medical/surgical unit
BBecause I think that anyone should be asked. You never know who has thoughts. You don’t want to miss one. You
could save a life by asking.^
– a 20-year-old man admitted to the medical/surgical unit
BIt’s hard to generalize, but it can’t hurt to ask. If there is a 1/1000 chance that you catch somebody, it’s worth a little
awkwardness.^
– a 63-year-old man admitted to the oncology unit
BWhen you are in a hospital, people are there to help you.^
– a 64-year-old woman admitted to the oncology unit

Related to this intervention theme, several patients alluded to the responsibility the health care
system would have after asking about suicide.

BWhen asking these kinds of questions, you have to make sure you have the help once you open the door.^
– a 40-year-old woman admitted to the oncology unit

Additional findings

In addition to these major themes, other topics were less frequently reported. Nine patients
(17%) mentioned in their responses that they knew someone who died by suicide. One patient
suggested that mental illness and suicide can be hereditary and repetitively seen in families thus
requiring additional oversight and monitoring.

BI watched my cousins’ children after he died by suicide [because] I have learned that it [runs in families]…. medical
patients… may need someone to check in with them.^
– a 48-year-old woman admitted to the medical/surgical unit

Eight patients (15%) connected suicide risk screening to stigma around mental health. Patients
mentioned concerns related to the rising rates of mental disorders, particularly with regard to
military personnel returning from active duty overseas. Several patients commented that while
suicide has been traditionally seen as a Btaboo^ topic, there appears to be growing national
awareness around the rising problem of suicide. They noted that this has helped to de-stigmatize
mental illness more generally and suicide in particular.

BTwenty years ago people didn’t want to talk about cancer [and we’ve come a long way since then]. Hopefully, the
same could be true in 20 years for suicide.^
– a 63-year-old woman admitted to the oncology unit

368 The Journal of Behavioral Health Services & Research 44:3 July 2017

BSeveral patients touched on the importance of screening for suicide not only in the adult setting, but in children and
extending to the entire family as well.^
B[Suicide screening] would be good for little kids. It strikes all ages and [screening] would open up a dialogue in the
family and has a ripple effect.^
– a 64-year-old woman admitted to the oncology unit

Concerns about screening for suicide risk

Five patients (9%) did not agree with universal suicide risk screening. Four suggested that
screening may not be relevant for all patients, stating that it Bdepends on the case.^ One
patient remarked about the timing of screening: BPeople feel fragile waiting for serious
surgery and that would not be the time to ask.^ Additionally, three patients had concerns
about being stigmatized or treated Bdifferently^ by staff if they screened positive for suicidal
ideation. Specifically, one patient who screened positive for suicide risk stated it Bmade me
feel as though [the medical team] would put me in a different category than the person not
having these difficulties. They are going to think I’m cuckoo.^ Some patients felt that
screening for suicide risk should occur only in those cases where there was a Bwarning sign^
or some clinical indication in the form of behavior, expressed thoughts, clinical history, or
certain risk variables like low levels of social support. One stated: BIt was not expected. Am
I here for my lungs or my mental health?^

Discussion

In a qualitative analysis subsample of a larger QIP, adult medical inpatients supported screening
for suicide risk during a medical hospital stay. When asked to elaborate on their opinions about
screening, patients highlighted the importance of directly asking about suicide. As a second theme,
they emphasized the link between physical and mental health as part of comprehensive hospital
care. Lastly, the importance of connecting screening to suicide intervention and prevention efforts
was underscored. These responses provide further support for asking about suicide in medical
settings, if implemented carefully and thoughtfully.

Figure 1
The script for nurses who administered the asQ’em20

Patient Opinions About Screening for Suicide Risk SNYDER ET AL. 369

In the first identified theme, patients reported that if medical patients are not asked directly about
suicide, they may not talk disclose these thoughts to the healthcare providers. This theme supports
findings from prior research that psychiatric symptoms often go unrecognized and untreated in
medically ill populations1–4 as well as research recommending the development of suicide risk
screening instruments for the medically ill.22, 23

Raising awareness and comfort about mental health screening is not only important for patients
with medical illness, but also critical for health care professionals. As part of the implementation of
the asQ’em QIP, in-service trainings with nurses, social workers, and physicians, provided by
multidisciplinary members of the behavioral health team, were vital for raising awareness and
increasing knowledge about suicide risk in the medical setting (see Horowitz et al., 2013).
Furthermore, a carefully scripted paragraph introducing asQ’em was created to reduce discomfort
both with the patient participants and the nursing staff who administered the risk of suicide
screening questions (see Fig. 1).

Another major theme that emerged was the notion that medical illness and mental health are
inextricably linked and as such should be integrated into the holistic care of the hospitalized
medical patient. It is common for patients to present to their primary care providers with somatic
complaints that may be a marker for psychological distress like depression, anxiety, and suicidal
thoughts.24 In fact, there has been a growing trend in hospitals and primary care settings to identify
socio-behavioral problems such as screening for depression and distress, domestic violence and
alcohol use or dependence.25–30 Therefore, identification of suicide risk could be incorporated into
comprehensive care for medically hospitalized inpatients.

Patient opinions highlight the importance of suicide risk screening in order to intervene and
provide safeguards for patients who might be harboring suicidal thoughts and behaviors. As stated
by the patients, once the Bdoor is opened^ to talk about suicide, health care providers will have the
responsibility to respond. In the case of acute suicidal risk, this may include an emergent
assessment by behavioral health staff, including possible one-to-one observer status or psychiatric
hospitalization. If an assessment indicates a more chronic risk of suicide, linking the patient with
outpatient mental health resources may be more appropriate. Of note, medical patients at elevated
risk for suicide may require additional assessment and attention after major medical procedures or
events, such as communication of a new diagnosis or poor prognosis. Guidelines around suicide
risk assessment and care have been published31–33 and likely need modification for use in medical
settings.

Limitations of this analysis include the fact that data was collected using a small
convenience sample of patients from a single setting as part of a QIP. Furthermore, as this
setting is a clinical research hospital, it is important to recognize that participants who enroll
in clinical trials represent a self-selected subsample of medical patients. Responses may not
generalize to other nonresearch settings. In addition, non-English-speaking patients were
excluded; this is an important population to assess opinions of in future studies. Lastly, the
qualitative analysis was conducted on open-ended responses to a single question, and
responses were often brief. Further qualitative analyses using more in-depth interviews or
focus groups may have elucidated additional themes.

Conclusions

Qualitative findings suggest that adult hospitalized medical patients support suicide risk
screening. The importance of asking directly the connection between physical and mental health as
well as the critical role of follow-up interventions were emphasized. Patient opinions lend further
support for the implementation of universal suicide risk screening and prevention strategies in
medical hospital settings.

370 The Journal of Behavioral Health Services & Research 44:3 July 2017

Implications for Behavioral Health

With support from medical patients for suicide risk screening, hospital systems can begin to take
the next steps in acting on the Joint Commission’s recommendation for screening in the general
medical setting.16 Risk of suicide screening will create opportunities for behavioral health
clinicians to work collaboratively with nonpsychiatric staff on inpatient medical units in several
ways. Targeted education about the importance of early detection and training on how to respond
to positive screens may reduce discomfort in managing suicide risk in the general medical setting.
Effective implementation of suicide risk screening will require coordinated effort with key hospital-
based stakeholders including behavioral health nursing, social work, and psychiatry. Behavioral
health nurse champions can help bridge relationships between nursing administration and staff
nurses in regards to mental health concerns. Hospital-based behavioral health clinicians can
position themselves to take an active role in the evaluation of patients who screen positive
and to manage acute psychiatric needs. Furthermore, linking at risk patients with community
mental health resources will require a coordinated effort with behavioral health staff taking
the lead.

Acknowledgments

The Quality Improvement Project described in this article was supported by the Intramural
Research Program of the National Institutes of Health and the National Institute of Mental
Health (ZIAMH002922) and by the NIH Clinical Center Nursing Department. The authors
disclosed no proprietary or commercial interest in any product mentioned or concept discussed in
this article. The opinions expressed in the article are the views of the authors and do not necessarily
reflect the views of the Department of Health and Human Services or the US government. The
authors wish to thank Daniel Powell, BA, Don Rosenstein, MD, and Jeanne Radcliffe, RN;
National Institutes of Health Clinical Center Nursing Leadership; the physicians associated with
the three inpatient units; and a special thanks to the bedside nurses, the clinical social workers, and
all the patients for their participation.

References

1. Bostwick JM & Rackley SJ. Completed suicide in medical/surgical patients: Who is at risk? Current Psychiatry Reports. 2007; 9(3): 242-
246.

2. Recklitis CJ, Lockwood RA, Rothwell MA, et al. Suicidal ideation and attempts in adult survivors of childhood cancer. Journal of
Clinical Oncology. 2006; 24(24): 3852-3857.

3. Ohaeri JU, Shokunbi WA, Kehinde SA, et al. The psychological problems of sickle cell disease sufferers and their methods of coping.
Social Science and Medicine. 1995; 40(7): 955-960.

4. Druss B & Pincus H. Suicidal ideation and suicide attempts in general medical illness. Archive of Internal Medicine. 2000; 160(10):
1522-1526.

5. Qin P, Webb R, Kapur N, et al. Hospitalization for physical illness and risk of subsequent suicide: A population study. Journal of Internal
Medicine. 2012; 273(1): 48-58.

6. Furlanetto LM & Stefanello B. Suicidal ideation in medical inpatients: Psychosocial and clinical correlates. General Hospital Psychiatry.
2011; 33(6):572-578.

7. Robson A, Scrutton F, Wilkinson L, et al. The risk of suicide in cancer patients: A review of the literature. Psycho-Oncology. 2010;
19(12): 1250-1258.

8. Brinkman TM, Zhang NM, Recklitis CJ, et al. Suicide ideation and associated mortality in adult survivors of childhood cancer. Cancer.
2014; 120(2): 271-277.

9. Pompili M, Forte A, Lester D, et al. Suicide risk in type 1 diabetes mellitus: A systematic review. Journal of Psychosomatic Research.
2014; 76(5): 352-360.

10. Bolton JM, Walld R, Chateau D, et al. Risk of suicide and suicide attempts associated with physical disorders: A population-based,
balancing score-matched analysis. Psychological Medicine. 2015; 45(3): 495-504.

11. Qin P, Hawton K, Mortensen PB, et al. Combined effects of physical illness and comorbid psychiatric disorder on risk of suicide in a
national population study. British Journal of Psychiatry. 2014; 204(6): 430-435.

12. Aiello-Laws LB. Assessing the risk for suicide in patients with cancer. Clinical Journal of Oncology Nursing. 2010; 14(6): 687-91.

Patient Opinions About Screening for Suicide Risk SNYDER ET AL. 371

13. Johnson TV, Garlow SJ, Brawley OW, et al. Peak window of suicide occurs within the first months of diagnosis: implications for clinical
oncology. Psycho-Oncology. 2012; 21(4): 351-356.

14. Turaga KK, Malafa MP, Jacobsen PB, et al. Suicide in patients with pancreatic cancer. Cancer. 2011; 117(3): 642-647.
15. Luoma JB, Martin CE, & Pearson JL. Contact with mental health and primary care providers before suicide: a review of the evidence.

American Journal of Psychiatry. 2002; 159(6): 909-916.
16. The Joint Commission. A follow-up report on preventing suicide: Focus on medical/surgical units and the emergency department. Oak

Brook, IL: Joint Commission Resources, 2010.
17. The Joint Commission. National Patient Safety Goals Effective January 1, 2014. Oak Brook, IL: Joint Commission Resources, 2014.
18. Ballard ED, Bosk A, Snyder D, et al. Patients’ opinions about suicide screening in a pediatric emergency department. Pediatric

Emergency Care. 2012; 28(1): 34-38.
19. Ballard ED, Stanley IH, Horowitz LM, et al. Asking youth questions about suicide risk in the pediatric emergency department: Results

from a qualitative analysis of patient opinions. Clinical Pediatric Emergency Medicine. 2013; 14(1): 20-27.
20. Horowitz LM, Snyder D, Ludi E, et al. Ask suicide-screening questions to everyone in medical settings: The asQ’em Quality

Improvement Project. Psychosomatics. 2013; 54(3): 239-247.
21. Braun V & Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 2006; 3(2): 77-101.
22. Boudreaux ED & Horowitz LM. Suicide risk screening and assessment: Designing instruments with dissemination in mind. American

Journal of Preventative Medicine. 2014; 47(3): S163-S169.
23. Horowitz LM, Bridge JA, Pao M, et al. Screening youth for suicide risk in medical settings: Time to ask questions. American Journal of

Preventative Medicine. 2014; 47(3): S170-S175.
24. Pan YJ, Lee MB, Chiang HC et al. The recognition of diagnosable psychiatric disorders in suicide cases’ last medical contacts. General

Hospital Psychiatry. 2009; 31(2): 181-184.
25. Jacobson PB, Donovan KA, Trask PC, et al. Screening for psychological distress in ambulatory cancer patients. Cancer. 2005; 103(7):

1494-1502.
26. Recklitis C, O’Leary T, & Diller L. Utility of routine psychological screening in the childhood cancer survivor clinic. Journal of Clinical

Oncology. 2003; 21(5): 787-792.
27. Siegel RM, Hill TD, Henderson VA, et al. Screening for domestic violence in the community pediatric setting. Pediatrics. 1999; 104(4):

874-877.
28. Rumpf HJ, Hapke U, Hill A, et al. Development of a screening questionnaire for the general hospital and general practices. Alcoholism:

Clinical and Experimental Research. 1997; 21(5): 894-898.
29. Gielen AC, O’Campo PJ, Campbell JC, et al. Women’s opinions about domestic violence screening and mandatory reporting. American

Journal of Preventative Medicine. 2000; 19(4):279-285.
30. Miller PM, Spies C, Neumann T, et al. Alcohol biomarker screening in medical and surgical settings. Alcoholism: Clinical and

Experimental Research. 2006; 30(2): 185-193.
31. Jobes DA. Managing Suicidal Risk: A Collaborative Approach. New York: The Guildford Press, 2006.
32. Bongar B & Sullivan GR. The Suicidal Patient: Clinical and Legal Standards of Care, Third Edition. Washington, D.C.: American

Psychological Association, 2013.
33. Joiner TE, Jr, Walker RL, Rudd MD, et al. Scientizing and routinizing the assessment of suicidality in the outpatient practice.

Professional Psychology: Research and Practice. 1999; 30(5): 447-453.

372 The Journal of Behavioral Health Services & Research 44:3 July 2017

Journal of Behavioral Health Services & Research is a copyright of Springer, 2017. All Rights
Reserved.

  • Patient Opinions About Screening for Suicide Risk in the Adult Medical Inpatient Unit
    • Abstract
    • Introduction
    • Methods
      • Patient population
      • Procedure
      • Qualitative data analysis
    • Results
      • Patient reactions to screening for suicide risk: major themes
      • Additional findings
      • Concerns about screening for suicide risk
    • Discussion
    • Conclusions
    • Implications for Behavioral Health
    • Acknowledgments
    • References

ORIGINAL ARTICLE

Contact between patients with suicidal ideation
and nurses in mental health wards: Development
and psychometric evaluation of a questionnaire

Joeri Vandewalle,1,2 Veerle Duprez,1 Dimitri Beeckman,1,3,4,5,6,7 Ann Van Hecke1,8 and
Sofie Verhaeghe1,9
1Department of Public Health and Primary Care, University Centre for Nursing and Midwifery, Ghent University,
Ghent, 2Research Foundation-Flanders (FWO), Brussel, 3Department of Public Health and Primary Care, Skin
Integrity Research Group (SKINT), University Centre for Nursing and Midwifery, Ghent, Belgium, 4School of
Health Sciences, €Orebro University, €Orebro, Sweden, 5School of Nursing and Midwifery, Royal College of Surgeons
in Ireland, Dublin, Ireland, 6Research Unit of Plastic Surgery, Department of Clinical Research, Faculty of Health
Sciences, Copenhagen, Denmark, 7School of Nursing and Midwifery, Monash University, Melbourne, Australia,
8Nursing Department, Ghent University Hospital, Ghent, and 9Department of Nursing, VIVES University College,
Roeselare, Belgium

ABSTRACT: Suicide prevention and treatment opportunities often depend on interpersonal
contact between patients and professionals. Presently, there is a lack of valid and reliable
instruments to obtain the perspective of patients with suicidal ideation regarding their contact
with professionals in mental health wards. This was a three-stage study to develop and
psychometrically evaluate a questionnaire: the Contact with Nurses from the perspective of
Patients with Suicidal ideation (CoNuPaS). First, the construct was defined by a systematic
review, qualitative study, and face validity among experts. Second, the content was validated
through a Delphi procedure with professional experts (n = 14) and cognitive interviews with
hospitalized patients (n = 12). Third, using a sample of adult patients with suicidal ideation in the
past year (n = 405), the psychometric properties were assessed by an exploratory factor analysis,
a test-retest procedure, and the internal consistency. The CoNuPaS comprises 23 items and two
subsections, to examine patients’ perceptions of contact experiences with nurses (CoNuPaS-
experience) and what they find important in that contact (CoNuPaS-importance). The subsections
comprise four components: encountering a space to express suicidal thoughts and explore needs,
being recognized as a unique and self-determining individual, encountering nurses’ availability/
information-sharing/transparency on expectations, and trusting nurses in communication about

Correspondence: Joeri Vandewalle, University Centre for Nursing and Midwifery, Department of Public Health and Primary Care, Ghent
University, Corneel Heymanslaan 10, 9000 Ghent, Belgium. Email: [email protected]; [email protected]
Declaration of Conflict of interest: None.
Authorship statement: All authors made substantial contributions to the conception and design of the study; Joeri Vandewalle and Sofie
Verhaeghe were involved in the process of data collection; all authors contributed to the analysis and interpretation of data. Joeri Vande-
walle and Veerle Duprez wrote the draft of the article. All authors critically revised the article, gave final approval for the submission of the
article, and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of
the work are appropriately investigated and resolved.

Joeri Vandewalle, RN, MSc, PhD Candidate.
Veerle Duprez, RN, MSc, PhD.
Dimitri Beeckman, RN, MSc, PhD.
Ann Van Hecke, RN, MSc, PhD.
Sofie Verhaeghe, RN, MSc, PhD.

Accepted July 21 2020.

© 2020 Australian College of Mental Health Nurses Inc.

International Journal of Mental Health Nursing (2021) 30, 219–234 doi: 10.1111/inm.12776

bs_bs_banner

suicidality. Content validity scores were excellent (0.78–1.00); test-retest intraclass correlation
coefficient and internal consistency were >0.90. Thus, the CoNuPaS demonstrated good
psychometric properties. The availability of a valid questionnaire to examine patient-nurse contact
in mental health wards is central to improving understanding of nurses’ contributions to suicide
prevention and suicidal ideation treatment.

KEY WORDS: nurses, patients, psychometric, questionnaire design, suicidal ideation.

INTRODUCTION

In psychiatric hospitals, suicide prevention and treat-
ment of suicidal ideation are imperative, given patients’
high suicide risk during and after hospitalization (Hunt
et al. 2013; Madsen et al. 2017). With suicide preven-
tion and treatment of suicidal ideation high on the pol-
icy agenda, mental health wards continue to focus on a
medical and risk-dominated paradigm (Fitzpatrick &
River 2018, Heller, 2015). This paradigm is evident in
efforts to implement prediction models and suicide risk
assessments, promote pharmacological treatments, and
formalize surveillance and containment strategies
(Belsher et al. 2019; Bolton et al. 2015; Manuel et al.
2018; Slemon et al. 2017).

In contrast to this emphasis on medical and risk-re-
lated factors of suicide prevention and suicidal ideation
treatment, interpersonal aspects of clinical encounters
receive limited attention (Cutcliffe & McKenna 2018).
However, the importance of well-considered contact
with patients cannot be overlooked, in terms of either
its suicide preventative effect or its meaning for
patients’ care experiences (Berg et al. 2017). If patients
perceive contact with professionals as trustful, non-
judgemental, understanding, and emotionally support-
ive, this could help them express their suicidal ideation
(Hom et al. 2017; Richards et al. 2019). Moreover, car-
ing contact with professionals can enable patients to
feel safe from their suicidal impulses and take the first
steps towards (re)connecting with themselves, relatives,
and treatment (Berg et al. 2017; Sellin et al. 2017).

Background

While all healthcare professionals should invest in con-
tact with patients experiencing suicidal ideation, this is
particularly evident for nurses in mental health wards.
Within multidisciplinary teams, nurses are referred to
as ‘front-line carers’ who provide direct care and are
appreciated by patients for being accessible and spend-
ing time with them (Cutcliffe & Stevenson 2008;

Vandewalle et al. 2020). Nurses’ closeness to patients
can serve as an interpersonal endeavour characterized
by being with patients ‘in the here and now’, listening
to and exploring individual needs, and building trusting
partnerships (Cutcliffe & Stevenson 2008; Santangelo
et al. 2018). Regarding a clinical perspective, nurses
are well positioned to develop personal knowledge of
patients, create conditions for discussing suicide, assess
and respond to patients’ risks and problems in daily life
(Sellin et al. 2017; Vandewalle et al. 2019).

However, nurses’ front-line position is described as
emotionally demanding, and nurses can encounter diffi-
culties during contact with patients who experience sui-
cidal ideation (Cutcliffe & Stevenson 2008). Nurses
may distance themselves from patients, lack empathy
and understanding (Samuelsson et al. 2000), lack
engagement (Lees et al. 2014), and avoid conversations
about suicide (Meerwijk et al. 2010). Moreover,
patients report being watched and controlled by nurses
imposing surveillance and containment strategies (Lees
et al. 2014), while also overlooking patients’ basic needs
and desires to be understood as individuals (Berg et al.
2017).

This raises questions as to why impersonal, observa-
tion-led, and containment-focused nursing often pre-
vails over interpersonal aspects of care (Cutcliffe &
McKenna 2018). Nursing researchers assert that this is
partly attributable to a failure to incorporate interper-
sonal aspects of care into valid and reliable instru-
ments, while simultaneously, nursing fundamentals
remain unarticulated and undervalued (McAndrew
et al. 2014; Sitzman & Watson 2019). Within the con-
text of suicidality, the authors conducted a systematic
review, which suggested that there are no valid instru-
ments from the perspective of patients with suicidal
ideation regarding their contact with nurses in mental
health wards (Vandewalle et al. 2020). Such an instru-
ment could generate insight into interpersonal aspects
often overlooked in nursing care (McAndrew et al.
2014), and in suicide prevention and treatment of suici-
dal ideation (Fitzpatrick & River 2018).

© 2020 Australian College of Mental Health Nurses Inc.

220 J. VANDEWALLE ET AL.

Thus, the authors considered other valid instruments
that could potentially examine patient-nurse contact in
mental health wards, including the Caring Attributes
Questionnaire (Arthur et al. 2004) and Therapeutic
Engagement Questionnaire (Chambers et al. 2017).
However, while these instruments incorporate essential
interpersonal aspects of care and contact, they lack
specificity regarding suicidality. For example, these
tools do not include items on conversations about sui-
cide, which are essential to patient-nurse contact in
mental health wards (Cutcliffe & Stevenson 2008; Hom
et al. 2017). Given the lack of specific instruments and
the evidence that interpersonal aspects of care are
often overlooked, the present study aimed to develop
and psychometrically evaluate a questionnaire to exam-
ine contact with nurses from the perspective of patients
with suicidal ideation.

METHODS

Design

The study used a three-stage process to develop and
psychometrically evaluate a questionnaire to examine
contact with nurses from the perspective of patients
with suicidal ideation (Fig. 1). The checklist for report-
ing a questionnaire research was used (Kelley et al.
2003).

Stage one: questionnaire development and face validity
The questionnaire’s conceptual foundation was
informed by a systematic review (Vandewalle et al.
2020) and qualitative study (Vandewalle et al. 2019).
Draft items were generated based on both studies.
Subsequently, two group-based evaluations were orga-
nized to establish face validity.

Systematic review and qualitative study. The
systematic review aimed to synthesize the perspectives
of people with suicidal ideation regarding interactions
with nurses in mental health and emergency services
(Vandewalle et al. 2020). The review involved a search
of the databases PubMed, Web of Science, Embase,
and PsycARTICLES, and included 26 studies. Most
studies used qualitative approaches and focused on
inpatient mental health care in Western countries.
Three key themes were identified: ‘being cared for and
acknowledged as a unique individual’, ‘giving voice to
myself in an atmosphere of connectedness’, and
‘encountering a nurturing space to address my
suicidality’. For questionnaire development, core

elements pertaining to patient-nurse contact in mental
health wards were extracted.

Additionally, a qualitative study using a grounded
theory approach was conducted (Vandewalle et al.
2019). This study aimed to elucidate the core elements
of how nurses in mental health wards make contact
with patients experiencing suicidal ideation. Nineteen
nurses were interviewed. The findings revealed that
nurses make contact with patients by ‘creating condi-
tions for open and genuine communication’, while
focusing on ‘developing an accurate and meaningful
picture of the patients’. These interconnected core ele-
ments represented nurses’ attention to both interper-
sonal and clinical aspects of practice, including
building trust and assessing suicide risk. Incorporating
nurses’ perspectives when constructing the question-
naire enhanced attention to the reciprocal nature of

FIG. 1 Developmental and psychometric evaluation process of the

CoNuPaS.

© 2020 Australian College of Mental Health Nurses Inc.

MAKING CONTACT WITH SUICIDAL PATIENTS 221

interpersonal contact and the items’ clinical appropri-
ateness. Figure 2 presents core elements derived from
the systematic review and qualitative study.

Group-based evaluations. Two group-based
evaluations were organized to revise draft items,
establish the questionnaire format, and establish face
validity. First, four research team members held open
discussions to develop the questionnaire to examine
Contact with Nurses from the perspective of Patients
with Suicidal ideation (CoNuPaS). These researchers
held academic and clinical positions, and had diverse
areas of expertise (e.g. nursing, mental health,
psychometric evaluation).

During the revision process, the researchers added
one item to focus more on patients’ involvement in
decision-making about their care and treatment. Sub-
sequently, they focused on the questionnaire’s format
and decided to score the items according to two sub-
sections: to examine patients’ perceptions of contact
experiences with nurses (CoNuPaS-experience) and
what they find important in that contact (CoNuPaS-
importance). Adding these subsections allowed for
detecting potential differences in – and discrepancies
between – patients’ experiences of contact with
nurses and what patients find important in this con-
tact.

Furthermore, subsection items were scored on a
five-point Likert scale, ranging from one (strongly dis-
agree/not important at all) to five (strongly agree/very
important). A number of items were negatively formu-
lated to minimize response-set bias (Polit & Beck
2017). Finally, instructions for completion were added.
For example, patients were instructed to keep in mind
the nurse with whom they had the most contact.

Second, the questionnaire was revised by a mental
health nursing steering committee (n = 6), including
two psychiatric hospital directors, two nursing aca-
demics, and two nursing specialists. They discussed the
items’ relevance, clarity, and the questionnaire’s com-
pleteness during a two-hour meeting with the first and
last authors. Seven items were removed due to overlap-
ping meanings. Other adjustments reflected a need for
more specific formulations. For example, ‘I can tell
everything to the nurse’ was reformulated as ‘I can talk
honestly about my suicidal thoughts with the nurse’.
Overall, feedback from the steering committee sup-
ported the questionnaire’s face validity. They did not
report missing aspects and indicated the content was
representative of contact between patients with suicidal
ideation and nurses in mental health wards. The two

group-based evaluations resulted in a 36-item question-
naire with two subsections.

Ethical considerations

The Ethical Committees of the Ghent University
Hospital and participating hospitals approved this study
(B670201630531). Permission was obtained through
informed consent from the hospital directors. Two hos-
pital directors declined to participate, and in two par-
ticipating hospitals, three ward managers stated
participation was not possible or desirable for their
patients. All participants were informed about the
study’s objectives, procedures, and assured about their
confidentiality. Regarding the cognitive interviews and
psychometric evaluation, patients were informed ver-
bally and through informed consent that participation
was voluntary and their care would not be affected by
their decision (not) to participate. Additionally, a
researcher explained the instructions, time was pro-
vided for questions, and participants were given as
much time as required to complete the questionnaire.
All participants provided written informed consent
before completing the questionnaire.

The researchers recognized that the CoNuPaS cov-
ers a sensitive topic. Although the items do not focus
on suicidality per se, it was anticipated that completing
the CoNuPaS might evoke distress. Therefore, the
researchers ensured that a supportive network of multi-
disciplinary team members was present on each ward.
Additionally, the data collection organized during group
meetings provided the researchers with the opportunity
to identify patients’ emerging distress and respond
accordingly. Participants were also informed verbally
and in the questionnaire they could contact a multidis-
ciplinary team member whenever they needed to talk
or had questions about suicide.

Stage two: content validation
In stage two, the content of the CoNuPaS was vali-
dated using a Delphi procedure among professional
experts and cognitive interviews with patients.

Delphi procedure. In a double Delphi procedure, 14
experts assessed the 36-item questionnaire (Hsu &
Sandford 2007). None of them participated in the
previous stage. To provide a differentiated perspective
on the content of the CoNuPaS, the experts differed in
terms of gender, age, and education level, and
represented clinical and academic positions, including
nurses (n = 4) and nursing specialists (n = 4) in

© 2020 Australian College of Mental Health Nurses Inc.

222 J. VANDEWALLE ET AL.

psychiatric hospitals, researchers and educators in
mental health care (n = 5), and one coordinator of
community mental health services.

The experts were asked to evaluate the items’ rele-
vance and clarity on a four-point Likert scale (irrele-
vant to very relevant) and a dichotomous scale (unclear
vs. clear). To guide this process, the experts were
emailed a Delphi form that incorporated the scoring
system and sufficient space to propose new items or
improvements. The obtained feedback was analysed
within the research team, leading to a feedback

summary and revised questionnaire, which were com-
municated back to the experts for assessment (Hsu &
Sandford 2007). The item-level Content Validity Index
(I-CVI) was calculated; I-CVI scores ≥ 0.78 were con-
sidered evidence of good content validity (Polit et al.
2007).

During the double Delphi procedure, 28 items were
adjusted, 8 items were removed, and 5 items were
added. Item adjustments were mainly focused on using
more understandable and specific language. The main
reasons for item removal were minimizing meaning

FIG. 2 Core elements of contact: perspectives of individuals with suicidal ideation and nurses.

© 2020 Australian College of Mental Health Nurses Inc.

MAKING CONTACT WITH SUICIDAL PATIENTS 223

overlap and I-CVI scores < 0.78. Additionally, the
expert feedback reflected a need to broaden the scope
of patient-nurse contact. For example, some experts
suggested adding an item about contact between
nurses and patients’ relatives. Following the double
Delphi procedure, the questionnaire comprised 34
items, with I-CVI scores between 0.78–1.00 for rele-
vance and clarity.

Cognitive interviews. Individual cognitive interviews
were conducted with 12 hospitalized adult patients.
The aim was to evaluate each item’s clarity and
relevance, the questionnaire’s completeness, feasibility,
and user-friendliness, and the time needed to complete
it (Beatty & Willis 2007). Ward managers invited
eligible patients based on inclusion criteria: being
admitted to a mental health ward, experiencing suicidal
ideation in the past year, aged 18–65 years, having the
mental capacity to consent to participate, and a good
command of the Dutch language. As shown in Table 1,
participants varied in terms of age, gender, number
and duration of admission(s), and self-reported suicidal
ideation.

The cognitive interviews were divided into three
rounds with four, five, and three participants,

TABLE 1 Participant demographics in the cognitive interviews and
psychometric evaluation

Cognitive

interviews

Psychometric

evaluation

n = 12 % n = 405 %

Gender

Male 4 33.3 168 41.5

Female 8 66.7 237 58.5

12 405

Age (years)

18–25 4 33.3 71 17.5

26–35 5 41.7 95 23.5

36–45 1 8.3 103 25.4

46–55 1 8.3 83 20.5

56–65 1 8.3 53 13.1

12 405

Previous admission to a mental health ward?

Yes 10 83.3 282 69.6

No 2 16.7 123 30.4

12 405

Previous admission to the current ward?

Yes 4 33.3 116 28.6

No 8 66.7 289 71.4

12 405

Duration of current admission

≤1 week – – 28 7.0

1 week – 1 month 1 8.3 77 19.2

1 month – 3 months 4 33.3 109 27.1

3 months – 6 months 3 25.0 95 23.6

>6 months 4 33.3 93 23.1

12 403

Frequency of suicidal ideation

Past year

seldom 1 8.3 70 17.5

sometimes 3 25.0 118 29.5

often 7 58.3 177 44.2

always 1 8.3 35 8.8

12 400

Past month

never 1 8.3 62 16.0

seldom 2 16.7 70 18.0

sometimes 4 33.3 114 29.4

often 4 33.3 111 28.6

always 1 8.3 31 8.0

12 388

Past week

never 6 50.0 104 26.9

seldom 1 8.3 78 20.2

sometimes 3 25.0 103 26.7

often – – 70 18.1

always 2 16.7 31 8.0

12 386

Severity of suicidal ideation (past week)

I had no thoughts of suicide 6 50.0 150 37.0

I had only brief passing thoughts 5 41.7 155 38.3

I had strong thoughts of suicide 1 8.3 74 18.3

(Continued)

TABLE 1 (Continued)

Cognitive

interviews

Psychometric

evaluation

n = 12 % n = 405 %

I made plans to attempt suicide – – 21 5.2

I made a suicide attempt – – 5 1.2

12 405

Type of ward

Ward for mood, anxiety,

personality disorders

6 50 119 29.4

Ward for day services and

treatment

– – 51 12.6

Ward for rehabilitation services 3 25 47 11.6

Ward for substance abuse

treatment

– – 58 14.3

Ward for treatment of psychotic

disorders

1 8.3 42 10.4

Secure wards (crisis and forensic

services)

– – 25 6.2

Psychiatric ward in a general

hospital

– – 21 5.2

Ward for young adults 2 16.7 15 3.7

Ward for short/intensive

treatment

– – 27 6.7

12 405

© 2020 Australian College of Mental Health Nurses Inc.

224 J. VANDEWALLE ET AL.

respectively, to allow for intermittent adjustments
(Beatty & Willis 2007). The interviews were conducted
in a quiet room in each ward, audio-recorded, and
transcribed verbatim. Participants were invited to
express their thoughts aloud when considering each
item and response options. Verbal probes such as ‘Can
you tell me more about. . .’ were used to elicit details
regarding how patients interpreted and answered
items. Discussions based on researcher triangulation
were organized to analyse the transcripts and partici-
pants’ suggestions (Beatty & Willis 2007).

Overall, participants perceived completing the
questionnaire to be feasible. They found the instruc-
tions clear and indicated the questionnaire covered
‘the most crucial aspects’ of their contact with nurses.
No participants expressed problems with the question-
naire’s length. As evaluated in the third round, time
needed to complete the questionnaire was, on aver-
age, 20–25 min. Participant feedback led to the
adjustment of six items. All items were (re)phrased
positively, because participants expressed difficulties
interpreting negatively phrased items. Additionally,
one item – ‘the nurse controls me more when my sui-
cidal thoughts are stronger’ – was added, because par-
ticipants suggested a double interpretation of the
item: ‘The nurse contacts me more when my suicidal
thoughts are stronger’. They called into question
whether ‘more contact’ stemmed from a nurse’s con-
trolling attitude or interpersonal engagement. Further-
more, the cognitive interviews provided insight into
the middle answer option’s meaning, which used the
formulation ‘neither/nor’. Participants endorsed the
middle option when they both agreed and disagreed
with a particular item. For example, they indicated a
nurse might take time to discuss suicide one moment,
but not the next. No participants indicated they
endorsed the middle option because they felt indiffer-
ent, did not want to answer, or did not understand
the item (Chyung et al. 2017).

Stage three: psychometric evaluation
In stage three, construct validity of the CoNuPaS was
assessed by exploratory factor analysis. Additionally,
reliability of the CoNuPaS was assessed using a test-
retest procedure and measuring internal consistency.

Sample. A convenience sample of 430 patients in 55
mental health wards of one general hospital and ten
psychiatric hospitals was established to determine the
questionnaire’s psychometric properties. The hospitals
were geographically spread across Flanders, the Dutch-

speaking part of Belgium. Inclusion criteria were the
same as for the cognitive interviews.

Data collection. Data were collected between April
2018 and May 2019. Participants completed the self-
report questionnaire and placed it in a sealed box.
Data collection was organized once in each ward, in a
quiet room where participants were convened for a
group meeting and completed the questionnaire
individually.

Only the researcher was present, to avoid any bias
in responses due to staff presence. Questionnaires
with < 75% of the items answered were removed from
data analysis (n = 15). All questionnaires were also
checked for potential response-set bias (Polit & Beck
2017). Questionnaires for which all items received the
same score were deleted (n = 10). Ultimately, 405
questionnaires were used for analysis.

In a test-retest procedure to determine the ques-
tionnaire’s stability, 63 patients completed the ques-
tionnaire twice. They were recruited from 13 wards of
three psychiatric hospitals. A short time interval of
three to five hours was set between the tests to mini-
mize confounding factors (Polit & Beck 2017), includ-
ing effects resulting from intermittent therapeutic
sessions or participants’ suicidal ideation, which can
fluctuate hourly (Kleiman et al. 2017).

Data analysis. Data were analysed using SPSS
Statistics 25 (IBM Corp., Armonk, NY, USA). Item
normality was assessed both visually (histogram and
quantile-quantile plot) and through normality tests
(skewness/kurtosis values and Shapiro-Wilk tests)
(Ghasemi and Saleh Zahediasl 2012). The data were
also checked for floor and ceiling effects. The authors
determined a priori that floor and ceiling effects
occurred when > 15% of participants scored in the
12.5% lower and upper bound, respectively, on
subsection and component levels.

First, construct validity of the CoNuPaS-experience
and CoNuPaS-importance was assessed using explora-
tory factor analysis. This method was used to identify
the underlying structure and inform item reduction.
The Kaiser–Meyer–Olkin measure of sampling ade-
quacy (≥0.80) and Bartlett’s test of sphericity
(P < 0.05) were used to determine the appropriateness
of exploratory factor analysis. The sample size
(n = 405) was adequate for factor analysis on the 34
items, given the recommendation to pursue ten partici-
pants per item (Costello & Osborne 2005). The princi-
pal axis factoring method and varimax rotation were

© 2020 Australian College of Mental Health Nurses Inc.

MAKING CONTACT WITH SUICIDAL PATIENTS 225

used. Eigenvalues over one and an inspection of the
scree plot were applied to determine the number of
extracted components. As per rule of thumb, items
were retained if they loaded > 0.40 on the component,
and cross-loading items were considered for removal to
obtain a unidimensional structure (Costello & Osborne
2005).

Second, reliability of the CoNuPaS-experience and
CoNuPaS-importance were assessed by a test-rest pro-
cedure and internal consistency analysis. For the test-
retest procedure, the intraclass correlation coefficient
(ICC) was calculated using a single-measurement –
two-way mixed-effects model with absolute-agreement
(Koo & Li 2016). Stability was assessed as moderate
(ICC 0.50–0.75), good (0.75–0.90), or excellent (>0.90;
Koo & Li 2016). Cronbach’s a was calculated to mea-
sure internal consistency. Cronbach’s a value >0.75
was considered satisfactory (Gliem & Gliem 2003).
Additionally, mean inter-item correlations were
reviewed, with scores >0.3 indicating an acceptable
correlation (Tabachnick & Fidell 2019).

RESULTS

Sample characteristics

The psychometric evaluation sample (n = 405) com-
prised 237 women and 168 men. Most participants
were between 36–45 years old (n = 103, 25.4%). Most
participants reported they sometimes (n = 118,
29.5%) or often (n = 177, 44.2%) experienced suicidal
ideation during the past year. Additionally, for the
past week, most participants reported no suicidal
thoughts (n = 163, 38%) or brief passing thoughts
(n = 161, 37.5%), while 18% reported strong suicidal
thoughts (n = 76). Participants were admitted to
wards mostly divided according to psychiatric diag-
noses (e.g. psychotic disorders), age (e.g. young
adults), or service delivery focus (e.g. crisis vs. reha-
bilitation services). Participant demographics are
shown in Table 1.

Results: psychometric evaluation

The conditions for performing an exploratory factor anal-
ysis were satisfied, including the Kaiser–Meyer–Olkin
measure (CoNuPaS-experience: 0.94; CoNuPaS-mpor-
tance: 0.93) and Bartlett’s test of sphericity (CoNuPaS-
experience: v2 = 4600.421, df = 253, P < 0.001; CoNu-
PaS-importance: v2 = 5136.624, df = 253, P < 0.001).

Factor analysis resulted in a four-component model.
The item loadings on each component were >0.40.
Item reduction appeared to be possible. Respecting
the conceptual foundation of the CoNuPaS, item
reduction was based not only on statistical measures
but also on careful assessment by the research team.
The main reason for removing items was overlapping
meaning, especially for cross-loading items. Moreover,
some items were removed because they appeared to
focus on potential ‘effects’ of patient-nurse contact
(e.g. increased hope/insight) rather than on ‘how’
nurses make contact, which is the focus of the CoNu-
PaS. The item ‘I have a good relationship with the
nurse’ was removed because, conceptually, patient-
nurse contact is not necessarily linked with a ‘relation-
ship’, which might require long-term contact (Priebe
& McCabe 2006). Overall, 11 items were removed,
resulting in a 23-item questionnaire with four compo-
nents. Table 2 shows descriptive statistics and factor
loadings for all items.

The explained variance of the four components was
63.76% for the CoNuPaS-experience and 61.24% for
the CoNuPaS-importance. Factor analysis resulted in a
similar fit between the CoNuPaS-experience and CoN-
uPaS-importance, except for two items. Given that
these items had a cross loading, they were classified in
the same component to preserve congruency between
subsections. Component 1 (eight items) was labelled
‘Encountering a space to express suicidal thoughts and
explore needs’. It indicates the importance of contact
with nurses who initiate conversations about suicide,
explore patients’ needs, and take patients’ suicidal
expressions seriously. Component 2 (seven items) was
labelled ‘Being recognized as a unique and self-deter-
mining individual’. This component emphasizes an
encounter between two individuals, wherein nurses
demonstrate care and concern, and patients can exer-
cise self-determination in decision-making about their
care and treatment. Component 3 (five items) was
labelled ‘Encountering nurses’ availability/information-
sharing/transparency on expectations’. It highlights
patients’ ability to access a nurse when needed and
experience nurses’ engagement in providing informa-
tion and discussing expectations. Component 4 (three
items) was labelled ‘Trusting nurses in communication
about suicidality’. It underlines the necessity of trust in
patient-nurse contact, especially for patients’ perceived
ability to communicate openly and honestly about their
suicidality. Table 3 presents the explained variance on
subsection and component levels.

© 2020 Australian College of Mental Health Nurses Inc.

226 J. VANDEWALLE ET AL.

Item distribution

For the CoNuPaS-experience, 16 of 23 items followed
a normal distribution. All items of the CoNuPaS-impor-
tance followed a non-normal distribution. In Table 2,
non-normally distributed items are described by their
median and interquartile range; normally distributed
items are described by their mean and standard devia-
tion (SD). Additionally, Table 3 presents the propor-
tion of patients who achieved the 12.5% lower and
upper bound, respectively, on the subsection and com-
ponent levels. No floor effects were found. Ceiling
effects were found in the second component of the
CoNuPaS-experience (23.30%) and in all components
of the CoNuPaS-importance (range 25.20%-53.10%).

Stability

Sixty-three patients completed the questionnaire twice.
On the first administration, the mean score was 3.59
(SD 0.71) for the CoNuPaS-experience and 4.26 (SD
0.49) for the CoNuPaS-importance. At retest, the cor-
responding scores were 3.65 (SD 0.73) and 4.28 (SD
0.51), respectively. The overall ICC was 0.95 (95% CI
= [0.92–0.97]) for the CoNuPaS-experience and 0.91
(95% CI = [0.85–0.94]) for the CoNuPaS-importance.
Table 3 displays ICCs for subsections and components.

Internal consistency

Cronbach’s a was 0.93 for the CoNuPaS-experience
and 0.93 for the CoNuPaS-importance. Cronbach’s a
values for the components ranged between 0.79 and
0.93 (Table 3). Mean inter-item correlation was 0.38
for the CoNuPaS-experience and 0.35 for the CoNu-
PaS-importance, which reflected acceptable correla-
tions (Tabachnick & Fidell 2019).

DISCUSSION

Evidence from the perspective of patients with suicidal
ideation regarding their contact with nurses is largely
restricted to qualitative research (Vandewalle et al.
2020). While qualitative research is crucial to ‘under-
stand’ the dynamics and processes involved in patient-
nurse contact (Hjelmeland and Knizek, 2010), a valid
instrument could facilitate the numerical visibility of
patient-nurse contact in suicide prevention and suicidal
ideation treatment, and quality of care (McAndrew
et al. 2014). Therefore, this study developed and psy-
chometrically evaluated the CoNuPaS, a questionnaire

to examine contact with nurses from the perspective of
patients with suicidal ideation. The CoNuPaS includes
23 items (scored on a five-point Likert scale) and two
subsections to examine and compare patients’ experi-
ences of contact with nurses (CoNuPaS-experience)
and what they find important in that contact (CoNu-
PaS-importance).

The CoNuPaS includes four components, which
represent patients’ opportunities for expressing suicidal
ideation and exploring needs, being recognized as a
unique and self-determining individual, encountering
nurses’ availability/information-sharing/transparency on
expectations, and trusting nurses in communication
about suicidality. The significance of these aspects must
be recognized. Regarding the patient’s perspective,
being able to express suicide-related experiences and
explore needs as part of open, validating, and trusting
contact is often the first step in recovering from suici-
dal ideation, including alleviating distress, regaining
hope, and (re)connecting with oneself (Cutcliffe &
Stevenson 2008; Sellin et al. 2017). Simultaneously,
prevention and treatment efforts often depend on
patients’ expression of suicidal ideation, and the effects
depend on meeting patients’ needs, including needs for
connection, support, acceptance, and validation (Berg
et al. 2017; Hom et al. 2017; Van Orden et al. 2010).

For clinical purposes, the CoNuPaS can be com-
pleted upon admission – and repeated during admis-
sion – to generate data about the presence and
development of crucial aspects of patient-nurse contact,
including trust and communication about suicide.
Moreover, data pertaining to patients’ needs to be rec-
ognized as a unique and self-determining individual
(i.g. component 2) might point to stigmatic views of
nurses. This is important because patients expressed
that nurses might minimize their feelings or approach
them as a ‘risk object’ (Vandewalle et al. 2020), and
nurses may listen and talk to patients as part of an
instrumental approach rather than an interpersonal
approach (Vandewalle et al. 2019). Considering this,
the CoNuPaS can make the interpersonal and commu-
nicative skills and qualities that patients value in nurses
more explicit and visible. Such data can inform training
and feedback initiatives that aim to foster nurses’ skills,
qualities, and confidence to discuss suicide and assess
suicide risk as part of an interpersonal approach (Berg
et al. 2017; Vandewalle et al. 2019). For research pur-
poses, the instrument can encourage studies that iden-
tify multilevel factors – such as severity of suicidality
and type of ward – which may influence patients’ per-
ceptions of their contact with nurses. Such knowledge

© 2020 Australian College of Mental Health Nurses Inc.

MAKING CONTACT WITH SUICIDAL PATIENTS 227

T
A
B
L
E

2
D
es
cr
ip
ti
ve

st
at
is
ti
cs

an
d
fa
ct
or

lo
ad

in
gs

of
th
e
23
-i
te
m

qu
es
ti
on

na
ir
e

C
oN

u
P
aS
-e
xp
er
ie
n
ce

C
oN

u
P
aS
-i
m
p
or
ta
n
ce

N
va
lid

M
ea
n

S
D

M
ed

ia
n

In
te
rq
u
ar

ti
le

ra
n
ge

F
ac
to
r

lo
ad
in
g

N
va
lid

M
ed

ia
n

In
te
rq
u
ar

ti
le

ra
n
ge

F
ac
to
r

lo
ad
in
g

25
%

75
%

25
%

75
%

C
om

p
on

en
t
1:

E
n
co
u
n
te
ri
n
g
a
sp
ac
e
to

ex
p
re
ss

su
ic
id
al

th
ou

gh
ts

an
d
ex
p
lo
re

n
ee
d
s

1
T
h
e
n
u
rs
e
ta
ke
s
ti
m
e
to

ta
lk

w
it
h
m
e
ab
ou

t
m
y
su
ic
id
al

th
ou

gh
ts

40
5

3.
34

1.
21

4
3

4
0.
80

40
1

4
4

5
0.
80

2
T
h
e
n
u
rs
e
in
vi
te
s
m
e
to

h
av
e
a
co
n
ve
rs
at
io
n
ab
ou

t
m
y
su
ic
id
al

th
ou

gh
ts

40
5

3.
04

1.
25

3
2

4
0.
79

40
1

4
3

5
0.
80

3
T
h
e
n
u
rs
e
as
ks

m
e
ab
ou

t
m
y
p
ro
b
le
m
s
th
at

gi
ve

ri
se

to
m
y
su
ic
id
al

th
ou

gh
ts

40
4

3.
47

1.
18

4
3

4
0.
77

40
1

4
4

5
0.
72

4
T
h
e
n
u
rs
e
b
ri
n
gs

m
y
th
ou

gh
ts

ab
ou

t
su
ic
id
e
in
to

co
n
ve
rs
at
io
n
w
h
en

I

n
ee
d
it

40
4

2.
90

1.
15

3
2

4
0.
74

40
2

4
4

5
0.
77

5
T
h
e
n
u
rs
e
ex
p
lo
re
s
w
it
h
m
e
w
h
at

I
n
ee
d
to

m
ak
e
m
y
su
ic
id
al

th
ou

gh
ts

m
or
e
to
le
ra
b
le

40
5

3.
30

1.
17

3
3

4
0.
72

40
0

4
4

5
0.
70

6
T
h
e
n
u
rs
e
co
n
ta
ct
s
m
e
m
or
e
w
h
en

m
y
su
ic
id
al

th
ou

gh
ts

ar
e
st
ro
n
ge
r

40
3

3.
20

1.
17

3
2

4
0.
65

40
1

4
4

5
0.
64

7
T
h
e
n
u
rs
e
ta
ke
s
m
e
se
ri
ou

sl
y
w
h
en

I
ta
lk

ab
ou

t
m
y
su
ic
id
al

th
ou

gh
ts

40
5

N
/A

N
/A

4
3

5
0.
64

40
2

5
4

5
0.
71

8
T
h
e
n
u
rs
e
as
ks

m
e
ab
ou

t
m
y
su
ic
id
al

th
ou

gh
ts

40
4

3.
20

1.
17

3
2

4
0.
62

40
0

4
4

5
0.
48

C
om

p
on

en
t
2:

B
ei
n
g
re
co
gn

iz
ed

as
a
u
n
iq
u
e
an
d
se
lf
-d
et
er
m
in
in
g
in
d
iv
id
u
al

9
T
h
e
n
u
rs
e
p
ay
s
at
te
n
ti
on

to
m
y
fe
el
in
gs

39
9

N
/A

N
/A

4
4

5
0.
77

39
9

5
4

5
0.
70

10
T
h
e
n
u
rs
e
tr
ea
ts

m
e
as

a
u
n
iq
u
e
p
er
so
n
w
it
h
in
d
iv
id
u
al

n
ee
d
s

40
2

N
/A

N
/A

4
4

5
0.
76

39
8

5
4

5
0.
69

11
T
h
e
n
u
rs
e
lis
te
n
s
to

m
e
w
it
h
ou

t
p
re
ju
d
ic
e

40
5

N
/A

N
/A

4
4

5
0.
75

40
2

5
4

5
0.
70

12
T
h
e
n
u
rs
e
ta
ke
s
m
y
op

in
io
n
in
to

ac
co
u
n
t

40
5

N
/A

N
/A

4
4

4
0.
72

40
2

4
4

5
0.
62

13
T
h
e
n
u
rs
e
is
ge
n
u
in
el
y
co
n
ce
rn
ed

ab
ou

t
h
ow

I
am

d
oi
n
g

40
3

N
/A

N
/A

4
4

5
0.
71

40
0

5
4

5
0.
69

14
I
h
av
e
th
e
op

p
or
tu
n
it
y
to

m
ak
e
d
ec
is
io
n
s
ab
ou

t
th
e
ca
re

I
n
ee
d

to
ge
th
er

w
it
h
th
e
n
u
rs
e

40
5

3.
78

0.
98

4
3

4
0.
59

40
1

4
4

5
0.
50

15
I
h
av
e
th
e
op

p
or
tu
n
it
y
to

as
k
fo
r
a
co
n
ve
rs
at
io
n
w
it
h
th
e
n
u
rs
e
w
h
en

I

n
ee
d
it

40
3

N
/A

N
/A

4
4

5
0.
54

39
9

5
4

5
0.
41

C
om

p
on

en
t
3:

E
n
co
u
n
te
ri
n
g
n
u
rs
es

av
ai
la
b
ili
ty
/i
n
fo
rm

at
io
n
-s
h
ar
in
g/
tr
an
sp
ar
en

cy
on

ex
p
ec
ta
ti
on

s

16
T
h
e
n
u
rs
e
d
is
cu
ss
es

w
it
h
m
e
w
h
at

I
ca
n
ex
p
ec
t
of

h
er
/h
im

d
u
ri
n
g
ou

r

co
n
ta
ct

40
4

3.
51

1.
01

4
3

4
0.
78

40
1

4
4

5
0.
81

17
T
h
e
n
u
rs
e
in
fo
rm

s
m
e
w
h
en

sh
e/
h
e
is
av
ai
la
b
le

on
th
e
w
ar
d

40
4

3.
61

1.
16

4
3

4
0.
74

40
1

4
4

5
0.
70

18
T
h
e
n
u
rs
e
d
is
cu
ss
es

w
it
h
m
e
w
h
at

sh
e/
h
e
w
ill

d
o
w
it
h
th
e
co
n
te
n
t
of

ou
r
co
n
ta
ct

40
4

3.
23

1.
11

3
2

4
0.
69

40
1

4
4

5
0.
72

19
I
kn

ow
w
h
en

ou
r
n
ex
t
co
n
ta
ct
-m

om
en

t
is
p
la
n
n
ed

40
4

3.
41

1.
23

4
2

4
0.
66

40
0

4
4

5
0.
64

20
T
h
e
n
u
rs
e
is
th
er
e
w
h
en

I
n
ee
d
h
er
/h
im

40
5

3.
48

0.
99

4
3

3
0.
50

40
2

4
4

5
0.
59

C
om

p
on

en
t
4:

T
ru
st
in
g
n
u
rs
es

in
co
m
m
u
n
ic
at
io
n
ab
ou

t
su
ic
id
al
it
y

21
I
am

op
en

ab
ou

t
m
y
su
ic
id
al

th
ou

gh
ts

w
it
h
th
e
n
u
rs
e

40
4

3.
19

1.
30

3
2

4
0.
82

39
9

4
4

5
0.
74

22
I
tr
u
st

th
e
n
u
rs
e
en

ou
gh

to
ta
lk

ab
ou

t
m
y
su
ic
id
al

th
ou

gh
ts

40
5

3.
67

1.
14

4
3

4.
50

0.
69

40
1

5
4

5
0.
56

23
I
ca
n
ta
lk

h
on

es
tl
y
ab
ou

t
m
y
su
ic
id
al

th
ou

gh
ts

w
it
h
th
e
n
u
rs
e

40
5

3.
60

1.
17

4
3

4
0.
68

40
1

4
4

5
0.
71

N
/A

=
n
ot

ap
p
lic
ab
le
:
th
e
m
ea
n
an
d
st
an
d
ar
d
d
ev
ia
ti
on

(S
D
)
ar
e
n
ot

re
p
or
te
d
fo
r
th
e
n
on

-n
or
m
al
ly

d
is
tr
ib
u
te
d
it
em

s.

© 2020 Australian College of Mental Health Nurses Inc.

228 J. VANDEWALLE ET AL.

can trigger reflection among healthcare policymakers,
professionals, educators, and researchers about the fun-
damental need to attune to patients’ interactional
needs.

While quantitative research can help visualize and
articulate interpersonal aspects of nursing care, the
pitfalls must be considered. The literature indicates
the risk that results obtained with instruments like the
CoNuPaS may be interpreted from a risk-dominated
and professionally led perspective (Slemon et al.
2017). Discussing suicide and building dialogue with
patients may represent mechanisms to enhance patient
compliance and control patients as ‘risk objects’, rather
than genuine ways to understand what patients are
experiencing, validate them as unique individuals, and
involve them in decision-making (Felton et al. 2018;
Fitzpatrick & River 2018). Likewise, in current health-
care systems where administrators are eager to meet
performance indicators (Kilbourne et al. 2018), making
contact with patients is easily viewed as ‘a duty’,
instead of an interpersonal expression of openness and
genuineness. To prevent the emergence of such unto-
ward mechanisms, the CoNuPaS must be used from a
holistic perspective, meaning that the obtained results
must be located back into the questionnaire’s concep-
tual foundation and into the context in which the
results acquire their meaning.

Methodological considerations

The CoNuPaS has a sound conceptual foundation and
demonstrated good psychometric properties in a men-
tal health ward context. A Delphi procedure with
experts and cognitive interviews with hospitalized
patients indicated that the questionnaire is user-
friendly and has excellent content validity. The con-
struct validity of the CoNuPaS-experience and CoNu-
PaS-importance was strong, with explained variances of
>60%. Additionally, the subsections and components
exhibited sound internal consistency (≥0.79) and mod-
erate to excellent stability (ICC = 0.59–0.95).

The rather high test-retest ICC values must be
interpreted in light of the three- to five-hour interval.
While this short time interval may have minimized con-
founding factors, as mentioned previously, one might
argue that such an interval inflates ICC values, due to
recall bias (Polit & Beck 2017). Simultaneously, com-
mon sense may tell us that the CoNuPaS items, which
reflect an interpersonal construct, may be not as sus-
ceptible to recall bias as items on, for instance, a
knowledge test (Althubaiti 2016).

Attention is needed for ceiling effects in the CoNu-
PaS-importance. As the CoNuPaS reflects meaningful
aspects of contact from a patient’s perspective, ceiling
effects were expected on the CoNuPaS-importance.
However, this ceiling effect impedes distinguishing
patients from each other; therefore, the CoNuPaS-im-
portance is less recommended for use in intervention
studies (Terwee et al. 2007). While the authors
attempted to minimize response-set bias by including
positively and negatively formulated items, participants
in the cognitive interviews informed us that this
induced interpretation difficulties. Additionally, the
self-report method is prone to overestimation and
social desirability bias, and this may partly explain why
CoNuPaS-experience item scores were relatively high
(Althubaiti 2016). Social desirability bias should be
considered, as people with suicidal ideation might per-
ceive themselves as a burden or have a need to feel
accepted by others (Van Orden et al. 2010). This may
influence patients’ responses to private and sensitive
items, such as whether they can discuss suicide with a
nurse.

Individuals with lived experience could have been
more explicitly involved throughout the study, to better
include and attune to their perspective in developing
and evaluating the questionnaire. Starting out with a
systematic review (Vandewalle et al. 2020) and con-
ducting cognitive interviews ensured that patients’ per-
spectives were prioritized. However, peer specialists
with lived experience of suicidal ideation (Huisman &
van Bergen 2019) were not involved in the Delphi pro-
cedure. This decision was pragmatic in nature to pre-
serve the study timeframe, partly because the ethical
committee requested a lengthy patient approval process
for involving peer specialists.

Furthermore, underrepresentation bias must be con-
sidered, given the lack of non-response data (Polit &
Beck 2017). Certain subpopulations may be underrep-
resented, including patients with severe suicidal intent
and severe mental disorders, and patients could not
participate if they did not have a good command of the
Dutch language, including people of non-Belgian ori-
gin. Recruitment issues are also relevant in this
respect. Two hospital directors declined to participate,
mainly out of concern that data collection in a group
would trigger adverse reactions, including manipulative
interactions between patients with personality disor-
ders. Additionally, three ward managers indicated that
their patients (e.g. patients with psychotic disorders)
lacked the mental capacity to participate and that par-
ticipation would exacerbate their distress. While the

© 2020 Australian College of Mental Health Nurses Inc.

MAKING CONTACT WITH SUICIDAL PATIENTS 229

T
A
B
L
E

3
Su

bs
ec
ti
on

an
d
co
m
po

ne
nt

va
lu
es
,
in
cl
ud

in
g
C
ro
nb

ac
h’
s
a,

ex
pl
ai
ne
d
va
ri
an

ce
,
an

d
flo

or
an

d
ce
ili
ng

ef
fe
ct
s

M
ed

ia
n

In
te
rq
u
ar

ti
le

ra
n
ge

F
lo
or
/c
ei
lin

g
ef
fe
ct

%

E
xp
la
in
ed

va
ri
an
ce

%
IC

C
(9
5%

C
I)

C
ro
n
b
ac
h
’s

a
25

%
75

%

C
oN

u
P
aS
-e
xp
er
ie
n
ce

C
om

p
on

en
t
1:

E
n
co
u
n
te
ri
n
g
a
sp
ac
e
to

ex
p
re
ss

su
ic
id
al

th
ou

gh
ts

an
d
ex
p
lo
re

n
ee
d
s

3.
38

2.
75

4.
00

7.
20
/5
.7
0

21
.3
0

0.
91

(0
.8
5–
0.
95
)

0.
91

C
om

p
on

en
t
2:

B
ei
n
g
re
co
gn

is
ed

as
a
u
n
iq
u
e
an
d
se
lf
-d
et
er
m
in
in
g
in
d
iv
id
u
al

4.
00

3.
71

4.
43

1.
50
/2
3.
30

19
.1
7

0.
94

(0
.9
0–
0.
96
)

0.
88

C
om

p
on

en
t
3:

E
n
co
u
n
te
ri
n
g
n
u
rs
es

av
ai
la
b
ili
ty
/i
n
fo
rm

at
io
n
-s
h
ar
in
g/
tr
an
sp
ar
en

cy
on

ex
p
ec
ta
ti
on

s

3.
40

3.
00

4.
00

1.
70
/9
.0
0

13
.2
1

0.
78

(0
.6
7–
0.
86
)

0.
79

C
om

p
on

en
t
4:

T
ru
st
in
g
n
u
rs
es

in
co
m
m
u
n
ic
at
io
n
ab
ou

t
su
ic
id
al
it
y

3.
50

2.
75

4.
00

4.
90
/1
1.
20

10
.0
8

0.
91

(0
.8
6–
0.
95
)

0.
84

S
u
b
se
ct
io
n
le
ve
l

3.
65

3.
17

4.
04

1.
20
/7
.4
0

63
.7
6

0.
95

(0
.9
2–
0.
97
)

0.
93

C
oN

u
P
aS
-i
m
p
or
ta
n
ce

C
om

p
on

en
t
1:

E
n
co
u
n
te
ri
n
g
a
sp
ac
e
to

ex
p
re
ss

su
ic
id
al

th
ou

gh
ts

an
d
ex
p
lo
re

n
ee
d
s

4.
25

3.
88

4.
63

1.
70
/2
8.
90

20
.9
2

0.
92

(0
.8
7–
0.
95
)

0.
91

C
om

p
on

en
t
2:

B
ei
n
g
re
co
gn

is
ed

as
a
u
n
iq
u
e
an
d
se
lf
-d
et
er
m
in
in
g
in
d
iv
id
u
al

4.
57

4.
14

4.
86

0.
20
/5
3.
10

15
.0
1

0.
80

(0
.7
0–
0.
88
)

0.
83

C
om

p
on

en
t
3:

E
n
co
u
n
te
ri
n
g
n
u
rs
es

av
ai
la
b
ili
ty
/i
n
fo
rm

at
io
n
-s
h
ar
in
g/
tr
an
sp
ar
en

cy
on

ex
p
ec
ta
ti
on

s

4.
00

3.
80

4.
60

0.
00
/2
5.
20

13
.1
4

0.
59

(0
.4
0–
0.
73
)

0.
79

C
om

p
on

en
t
4:

T
ru
st
in
g
n
u
rs
es

in
co
m
m
u
n
ic
at
io
n
ab
ou

t
su
ic
id
al
it
y

4.
33

4.
00

5.
00

1.
00
/4
3.
20

12
.1
6

0.
86

(0
.7
8–
0.
91
)

0.
80

S
u
b
se
ct
io
n
le
ve
l

4.
30

3.
96

4.
61

0.
20
/3
2.
40

61
.2
4

0.
91

(0
.8
5–
0.
94
)

0.
93

IC
C
,
In
tr
ac
la
ss

C
or
re
la
ti
on

C
oe
ffi
ci
en

t.

F
lo
or

an
d
ce
ili
n
g
ef
fe
ct
s
oc
cu
rr
ed

w
h
en

>
15
%

of
th
e
p
ar
ti
ci
p
an
ts

sc
or
ed

in
th
e
12
.5
%

lo
w
er

an
d
u
p
p
er

b
ou

n
d
,
re
sp
ec
ti
ve
ly
,
on

su
b
se
ct
io
n
an
d
co
m
p
on

en
t
le
ve
ls
.

© 2020 Australian College of Mental Health Nurses Inc.

230 J. VANDEWALLE ET AL.

authors acknowledge that some patients in the settings
that declined participation may not have been able to
participate or were susceptible to risk that justified
their exclusion (Emanuel et al. 2000), they questioned
whether this was true for all patients. If not, then
excluding patients who are able and willing to partici-
pate not only perpetuates bias but also precludes
patients’ possibilities of having a meaningful experi-
ence, making a social contribution, and managing their
own lives (Littlewood et al. 2019; Sellin et al. 2017).

Contextual considerations

The CoNuPaS was developed and tested in the context
of nursing care in mental health wards. This exclusive
focus is a limitation given that persons at risk of suicide
may not access mental health wards, and they interact
with many other professionals across different health-
care settings (Hom et al. 2015). Considering this,
efforts are required for cross-contextual adaptation and
validation of the CoNuPaS. In accordance with the
guideline of Sousa and Rojjanasrirat (2011), such
efforts require a comprehensive process, where the
development of a conceptual foundation is prioritized,
followed by rigorous expert evaluation and careful psy-
chometric testing.

To adapt the CoNuPaS for use across mental health
professions, the instrument should be elaborated through
conceptual understanding of the interactions between
patients and other professionals. This understanding
would enable sensitivity for differences in multidisci-
plinary practices. For example, compared to nurses, psy-
chiatrists’ contacts with patients might occur more often
within delineated therapies (Hagen et al. 2017), and psy-
chologists might perceive suicide prevention less as a
‘duty’ and may have fewer condemning attitudes towards
suicide (Norheim et al. 2016). Additionally, future
research should aim to adapt the CoNuPaS for use across
healthcare settings. This is necessary because crucial
aspects of patient-nurse contact, like discussing suicide,
are equally important, for instance, in community-based
services and emergency departments. Emergency depart-
ments in particular are considered a ‘starting point’,
where people often enter the healthcare system following
a suicide attempt, and where their contact with profes-
sionals influences future help-seeking and disclosure of
suicidal ideation (Hom et al. 2015).

Research efforts to adapt the CoNuPaS for use in
emergency departments should acknowledge the lack
of conceptual understanding of patient-nurse contact in
this context (Vandewalle et al. 2020). Enhancing this

understanding should be prioritized to identify relevant
cross-contextual differences. For example, patient-nurse
contacts in emergency departments may be influenced
by brief admissions and intense clinical requirements,
including suicide screening and early intervention
(Ceniti et al. 2020). Moreover, persons with suicidal
ideation and behaviour might experience more stigma
from emergency professionals than from mental health
professionals (Frey et al. 2016). Furthermore, a rigor-
ous approach will be needed to translate, adapt, and
validate the CoNuPaS for use across cultures (Sousa &
Rojjanasrirat 2011). For instance, it is likely that not all
CoNuPaS items account for patient-nurse contact in
non-Western cultures, where suicide may neither be
acknowledged nor discussed, or is still viewed as a
crime (WHO 2018).

CONCLUSION

The CoNuPaS is a self-report questionnaire with the
potential to generate new insight from the perspective
of patients with suicidal ideation regarding their con-
tact with nurses in mental health wards. The 23-item
questionnaire has a sound conceptual foundation and
demonstrated good psychometric properties. By includ-
ing two subsections and four components, the CoNu-
PaS can facilitate in-depth and differentiated
perspectives on the aspects of contact that patients
with suicidal ideation perceive as most helpful.

RELEVANCE TO CLINICAL PRACTICE

The availability of a valid questionnaire to examine
patient-nurse contact in mental health wards is central
to an improved understanding of nurses’ roles and con-
tributions to suicide prevention and suicidal ideation
treatment. The CoNuPaS can highlight those aspects of
contact that require attention and, subsequently, inform
developments in nursing practice that contribute to a
better fit between patients’ contact with nurses and what
they find important when experiencing suicidal ideation.
Overall, when used thoughtfully, the CoNuPaS can pro-
vide insights that stimulate person-centred and collabo-
rative approaches in nursing care (McAndrew et al.
2014), as well as suicide prevention and treatment of
suicidal ideation (Fitzpatrick & River 2018).

ACKNOWLEDGEMENTS

The Research Foundation Flanders (grant number:
1150117N) funded this study. We express our thanks

© 2020 Australian College of Mental Health Nurses Inc.

MAKING CONTACT WITH SUICIDAL PATIENTS 231

to Yentl Defour, Svea Geeroms and Evelyn Blanchard
for their involvement in the data collection. In addi-
tion, we are grateful to the experts who supported the
questionnaire development and to all the participants
for completing the questionnaire.

ETHICS APPROVAL

The Ethical Committees of the Ghent University
Hospital and the participating hospitals approved the
study (B670201630531).

FUNDING STATEMENT

The Research Foundation Flanders (grant number:
1150117N) funded this study. This funding had no
influence on the study design and process; the writing
of the article; and the decision to submit the article for
publication.

REFERENCES

Althubaiti, A. (2016). Information bias in health research:
definition, pitfalls, and adjustment methods. Journal of
Multidisciplinary Healthcare, 9, 211–217.

Arthur, D., Chong, C., Rujkorakarn, D., Wong, D. &
Wongpanarak, N. (2004). A profile of the caring attributes of
Hong Kong and Thailand psychiatric nurses. International
Journal of Mental Health Nursing, 13, 100–106.

Beatty, P. C. & Willis, G. B. (2007). Research Synthesis The
Practice of Cognitive Interviewing. Public Opinion
Quarterly, 71, 287–311.

Belsher, B. E., Smolenski, D. J., Pruitt, L. D. et al. (2019).
Prediction models for suicide attempts and deaths: a
systematic review and simulation. JAMA Psychiatry, 76,
642–651.

Berg, S. H., Rørtveit, K. & Aase, K. (2017). Suicidal patients’
experiences regarding their safety during psychiatric in-
patient care: a systematic review of qualitative studies.
BMC Health Services Research, 17, 73.

Bolton, J. M., Gunnell, D. & Turecki, G. (2015). Suicide risk
assessment and intervention in people with mental illness.
British Medical Journal, 351, h4978.

Ceniti, A. K., Heinecke, N. & McInerney, S. J. (2020).
Examining suicide-related presentations to the emergency
department. General Hospital Psychiatry, 63, 152–157.

Chambers, M., McAndrew, S., Nolan, F., Thomas, B., Watts,
P. & Kantaris, X. (2017). Service user involvement in the
coproduction of a mental health nursing metric: The
Therapeutic Engagement Questionnaire. Health
Expectations, 20, 871–877. https://doi.org/10.1111/hex.
12526.

Chyung, S. Y., Roberts, K., Swanson, I. & Hankinson, A.
(2017). Evidence-Based Survey Design: The Use of a

Midpoint on the Likert Scale. Performance Improvement,
56, 15–23.

Costello, A. B. & Osborne, J. W. (2005). Best practices in
exploratory factor analysis: Four recommendations for
getting the most from your analysis. Practical Assessment,
Research, & Evaluation, 10, 1–9.

Cutcliffe, J. R. & McKenna, H. (2018). Psychiatric/mental
health nursing in the 21st century: It’s ’d�ej�a vu’ all over
again! International Journal of Nursing Studies, 81 A3–A5.
https://doi.org/10.1016/j.ijnurstu.2018.03.004.

Cutcliffe, J. R. & Stevenson, C. (2008). Feeling our way in
the dark: the psychiatric nursing care of suicidal people–a
literature review. International Journal of Nursing Studies,
45, 942–953.

Emanuel, E. J., Wendler, D. & Grady, C. (2000). What
makes clinical research ethical? Journal of the American
Medical Association, 283, 2701–2711.

Felton, A., Repper, J. & Avis, M. (2018). Therapeutic
relationships, risk, and mental health practice.
International Journal of Mental Health Nursing, 27, 1137–
1148.

Fitzpatrick, S. J. & River, J. (2018). Beyond the Medical
Model: Future Directions for Suicide Intervention
Services. International Journal of Health Services:
Planning, Administration, Evaluation, 48, 189–203.

Frey, L. M., Hans, J. D. & Cerel, J. (2016). Perceptions of
suicide stigma: How do social networks and treatment
providers compare? Crisis: The Journal of Crisis
Intervention and Suicide Prevention, 37, 95–103.

Ghasemi, A. & Zahediasl, S. (2012). Normality tests for
statistical analysis: a guide for non-statisticians.
International Journal of Endocrinology and Metabolism,
10, 486–489.

Gliem, J. & Gliem, R. (2003). Calculating, interpreting and
reporting Cronbach’s alpha reliability coefficient for
Likert-type scales. Paper presented at the 2003. Midwest
Research-to-Practice Conference in Adult and Continuing
and Community Education, the Ohio State University,
Columbus, Ohio. Retrieved from https://scholarworks.
iupui.edu/bitstream/handle/1805/344/Gliem+&+Gliem.pdf?
sequence=1

Hagen, J., Hjelmeland, H. & Knizek, B. L. (2017). Relational
principles in the care of suicidal inpatients: Experiences of
therapists and mental health nurses. Issues in Mental
Health Nursing, 38, 99–106.

Heller, N. R. (2015). Risk, Hope and Recovery: Converging
Paradigms for Mental Health Approaches with Suicidal
Clients. The British Journal of Social Work, 45, 1788–
1803.

Hjelmeland, H. & Knizek, B. L. (2010). Why we need
qualitative research in suicidology. Suicide and Life-
Threatening Behavior, 40, 74–80.

Hom, M. A., Stanley, I. H. & Joiner, T. E. (2015). Evaluating
factors and interventions that influence help-seeking and
mental health service utilization among suicidal
individuals: A review of the literature. Clinical Psychology
Review, 40, 28–39.

© 2020 Australian College of Mental Health Nurses Inc.

232 J. VANDEWALLE ET AL.

Hom, M. A., Stanley, I. H., Podlogar, M. C. & Joiner, T. E.
Jr (2017). “Are You Having Thoughts of Suicide?”
Examining Experiences With Disclosing and Denying
Suicidal Ideation. Journal of Clinical Psychology, 73,
1382–1392.

Hsu, C.-C. & Sandford, B. A. (2007). The Delphi technique:
making sense of consensus. Practical Assessment, Research
& Evaluation, 12, 1–8.

Huisman, A. & van Bergen, D. D. (2019). Peer specialists in
suicide prevention: Possibilities and pitfalls. Psychological
Services, 16, 372–380.

Hunt, I. M., Bickley, H., Windfuhr, K., Shaw, J., Appleby, L.
& Kapur, N. (2013). Suicide in recently admitted
psychiatric in-patients: A case-control study. Journal of
Affective Disorders, 144, 123–128.

Kelley, K., Clark, B., Brown, V. & Sitzia, J. (2003). Good
practice in the conduct and reporting of survey research.
International Journal for Quality in Health Care, 15, 261–
266.

Kilbourne, A. M., Beck, K, Spaeth-Rublee, B., et al. (2018).
Measuring and improving the quality of mental health
care: a global perspective. World Psychiatry, 17, 30–38.

Kleiman, E. M., Turner, B. J., Fedor, S., Beale, E. E.,
Huffman, J. C. & Nock, M. K. (2017). Examination of
real-time fluctuations in suicidal ideation and its risk
factors: Results from two ecological momentary
assessment studies. Journal of Abnormal Psychology, 126,
726–738.

Koo, T. K. & Li, M. Y. (2016). A Guideline of Selecting and
Reporting Intraclass Correlation Coefficients for
Reliability Research. Journal of Chiropractic Medicine, 15,
155–163.

Lees, D., Procter, N. & Fassett, D. (2014). Therapeutic
engagement between consumers in suicidal crisis and
mental health nurses. International Journal of Mental
Health Nursing, 23, 306–315.

Littlewood, D. L., Harris, K., Gooding, P., Pratt, D.,
Haddock, G. & Peters, S. (2019). Using my Demons to
Make Good: The Short- and Long-Term Impact of
Participating in Suicide-Related Research. Archives of
Suicide Research, 1–25. https://doi.org/10.1080/13811118.
2019.1663330.

Madsen, T., Erlangsen, A. & Nordentoft, M. (2017). Risk
Estimates and Risk Factors Related to Psychiatric
Inpatient Suicide–An Overview. International Journal of
Environmental Research and Public Health, 14, 253.
https://doi.org/10.3390/ijerph14030253.

Manuel, J., Crowe, M., Inder, M. & Henaghan, M. (2018).
Suicide prevention in mental health services: A qualitative
analysis of coroners’ reports. International Journal of
Mental Health Nursing, 27, 642–651.

McAndrew, S., Chambers, M., Nolan, F., Thomas, B. &
Watts, P. (2014). Measuring the evidence: reviewing the
literature of the measurement of therapeutic engagement
in acute mental health inpatient wards. International
Journal of Mental Health Nursing, 23, 212–220.

Meerwijk, E. L., van Meijel, B., van den Bout, J., Kerkhof,
A., de Vogel, W. & Grypdonck, M. (2010). Development
and evaluation of a guideline for nursing care of suicidal
patients with schizophrenia. Perspectives in Psychiatric
Care, 46, 65–73.

Norheim, A. B., Grimholt, T. K., Loskutova, E. & Ekeberg,
O. (2016). Attitudes toward suicidal behaviour among
professionals at mental health outpatient clinics in
Stavropol, Russia and Oslo, Norway. BMC Psychiatry, 16.
https://doi.org/10.1186/s12888-016-0976-5.

Van Orden, K. A., Witte, T. K., Cukrowicz, K. C.,
Braithwaite, S., Selby, E. A. & Joiner, T. E. (2010). The
Interpersonal Theory of Suicide. Psychological Review,
117, 575–600.

Polit, D. F. & Beck, C. T. (2017). Nursing Research:
Generating and Assessing Evidence for Nursing Practice,
10th edn. Philadelphia: Wolters Kluwer Health.

Polit, D. F., Beck, C. T. & Owen, S. V. (2007). Is the CVI an
acceptable indicator of content validity? Appraisal and
recommendations. Research in Nursing and Health, 30,
459–467.

Priebe, S. & McCabe, R. (2006). The therapeutic relationship
in psychiatric settings. Acta Psychiatrica Scandinavica,
113, 69–72.

Richards, J. E., Whiteside, U., Ludman, E. J. et al. (2019).
Understanding why patients may not report suicidal
ideation at a health care visit prior to a suicide attempt: a
qualitative study. Psychiatric Services, 70, 40–45.

Samuelsson, M., Wiklander, M., Asberg, M. & Saveman, B. I.
(2000). Psychiatric care as seen by the attempted suicide
patient. Journal of Advanced Nursing, 32, 635–643.

Santangelo, P., Procter, N. & Fassett, D. (2018). Mental
health nursing: Daring to be different, special and leading
recovery-focused care? International Journal of Mental
Health Nursing, 27, 258–266.

Sellin, L., Asp, M., Wallsten, T. & Wiklund Gustin, L.
(2017). Reconnecting with oneself while struggling
between life and death: The phenomenon of recovery as
experienced by persons at risk of suicide. International
Journal of Mental Health Nursing, 26, 200–207.

Sitzman, K. & Watson, J. (2019). Assessing and Measuring
Caring in Nursing and Health Sciences: Watson’s Caring
Science Guide, 3rd edn. New York: Springer Publishing.

Slemon, A., Jenkins, E. & Bungay, V. (2017). Safety in
psychiatric inpatient care: The impact of risk management
culture on mental health nursing practice. Nursing
Inquiry, 24, e12199.

Sousa, V. D. & Rojjanasrirat, W. (2011). Translation,
adaptation and validation of instruments or scales for use
in cross-cultural health care research: a clear and user-
friendly guideline. Journal of Evaluation in Clinical
Practice, 17, 268–274.

Tabachnick, B. G. & Fidell, L. S. (2019). Using multivariate
statistics, 7th edn. Boston: Pearson Education.

Terwee, C. B., Bot, S. D., de Boer, M. R. et al. (2007).
Quality criteria were proposed for measurement

© 2020 Australian College of Mental Health Nurses Inc.

MAKING CONTACT WITH SUICIDAL PATIENTS 233

properties of health status questionnaires. Journal of
Clinical Epidemiology, 60, 34–42.

Vandewalle, J., Beeckman, D., Van Hecke, A., Debyser, B.,
Deproost, E. & Verhaeghe, S. (2019). Contact and
communication with patients experiencing suicidal
ideation: A qualitative study of nurses’ perspectives.
Journal of Advanced Nursing, 75, 2867–2877.

Vandewalle, J., Van Bos, L., Goossens, P. et al. (2020). The
perspectives of adults with suicidal ideation and behaviour

regarding their interactions with nurses in mental health
and emergency services: a systematic review. International
Journal of Nursing Studies, accepted for publication 110,
103692, https://doi.org/10.1016/j.ijnurstu.2020.103692.

World Health Organization (WHO) (2018). National Suicide
Prevention Strategies: Progress, Examples and Indicators.
https://www.who.int/mental_health/suicide-prevention/
national_strategies_2019/en/Accessed: 9 August 2019.

© 2020 Australian College of Mental Health Nurses Inc.

234 J. VANDEWALLE ET AL.

Copyright of International Journal of Mental Health Nursing is the property of Wiley-
Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv
without the copyright holder’s express written permission. However, users may print,
download, or email articles for individual use.

Development of the Uni Virtual Clinic: an online programme for
improving the mental health of university students
Louise M. Farrer , Amelia Gulliver, Natasha Katruss, Kylie Bennett *, Anthony Bennett*,
Kathina Ali† and Kathleen M. Griffiths ‡

Centre for Mental Health Research, The Australian National University, Canberra, Australia

ABSTRACT
There is growing recognition of the importance of addressing the mental
health needs of young people attending university. Anonymous, scalable,
and evidence-based online interventions can help to reduce burden on
university services and increase access to care for marginalised or
disconnected students. This paper reports the participatory design
methods used to develop the Uni Virtual Clinic (UVC), a comprehensive
online programme that was designed to prevent and treat mental
health problems and related issues in university students. Data
evaluating the participatory design process is also presented. The
potential for the implementation of the UVC within university-based
counselling services is strong, and has potential to reduce the
prevalence of mental disorders in a high-risk group of young people.

ARTICLE HISTORY
Received 19 December 2018
Revised 17 August 2019
Accepted 9 February 2020

KEYWORDS
University; student; mental
health; online; technology

Introduction

Young adults attending university face high age and life-stage related risks for developing mental
health problems, with approximately 30-50% of university students meeting criteria for a mental dis-
order (Auerbach et al., 2018; Eisenberg, Hunt, Speer, & Zivin, 2011; Said, Kypri, & Bowman, 2013). An
Australian study demonstrated a prevalence rate of 33.6% (Stallman, 2010), and in a large scale, epi-
demiological study of over 5000 college students in the US, the 12-month prevalence of any mental
disorder was almost 50%, with the highest rates of disorder being substance use disorders (29%),
anxiety disorders (11.9%) and mood disorders (10.6%) (Blanco et al., 2008). University students
may be at greater risk of mental health problems than their community-based counterparts, with
research demonstrating that the prevalence of severe psychological distress is significantly higher
in tertiary students (19-48%) compared to their age-matched peers not undertaking tertiary edu-
cation (3-11%) (Leahy et al., 2010; Stallman, 2010).

Despite the high prevalence of mental disorders among students, very few receive appropriate
mental health care (Eisenberg et al., 2011). Similar to young people in the general community (Rick-
wood & Braithwaite, 1994; Rickwood, Deane, & Wilson, 2007), when university students seek help for
mental health problems, they typically approach friends, family, or other non-clinical sources of
support (Eisenberg et al., 2011). Although these informal sources may be a useful starting point (Rick-
wood & Braithwaite, 1994), it is critical for those with mental health problems to seek help from evi-
dence-based sources. Less than half of students with a mental health problem seek professional help

© 2020 Informa UK Limited, trading as Taylor & Francis Group

CONTACT Louise M Farrer [email protected] Centre for Mental Health Research, The Australian National University,
Building 63, Eggleston Road, Canberra, ACT, 2061, Australia
*Current affiliation: e-hub Health Pty Ltd (https://ehubhealth.com/).
†Current affiliation: Research School of Psychology, The Australian National University, Canberra ACT 2601, Australia.
‡Current affiliation: College of Health and Medicine, The Australian National University, Canberra ACT 2601, Australia.

BRITISH JOURNAL OF GUIDANCE & COUNSELLING
2020, VOL. 48, NO. 3, 333–346
https://doi.org/10.1080/03069885.2020.1729341

for their symptoms, with help seeking rates as low as 5% for drug and alcohol use disorders (Blanco
et al., 2008; Wynaden, Wichmann, & Murray, 2013). The most frequently reported barriers to help
seeking by university students include lack of time, high treatment costs, and concerns about confi-
dentiality and stigma (Givens & Tjia, 2002; Mowbray et al., 2006).

Online mental health interventions are easily accessible, can be utilised in private, are cost-
effective, and typically require less time than face-to-face appointments (Griffiths & Christensen,
2007). Hence, they may be highly suited to the university student population. Young people fre-
quently report using the internet for help with their mental health (Burns, Davenport, Durkin, Lus-
combe, & Hickie, 2010). One study reported that 72% of young people aged 18–25 years believe
that websites are helpful when managing mental health problems (Oh, Jorm, & Wright, 2009), and
almost half of the young people in another study stated that they would use the internet to
access information about mental disorders (Burns et al., 2010). The internet is also commonly used
by young people to confide in peers about problems and to access forums for “question and
answer” advice (Burns et al., 2010). Additionally, online interventions targeting mental health pro-
blems have been found to be effective for university students. Several systematic reviews have
found that computer, internet, and telephone-based mental health interventions are promising in
university populations, particularly those aimed at improving anxiety symptoms (Conley, Durlak,
Shapiro, Kirsch, & Zahniser, 2016; Davies, Morriss, & Glazebrook, 2014; Farrer et al., 2013; Harrer
et al., 2018). A study investigating student attitudes towards online mental health resources reported
that almost half (47%) of university students indicated that they would use an online programme for
student wellbeing if one existed, and highly distressed students were significantly more likely to indi-
cate intent to use such a programme (Ryan, Shochet, & Stallman, 2010).

Virtual clinics provide streamlined, continuity of care for chronic physical and mental health con-
ditions (Jennings, Powell, Armstrong, Sturt, & Dale, 2009; Kethers et al., 2006). Virtual clinic models
typically focus on self-management in care, and incorporate multiple tools to manage health pro-
blems including information, symptom screening and monitoring tools, and other therapeutic
content. They are capable of providing evidence-based resources that are tailored to the specific clini-
cal needs and preferences of the user.

In its role as an Essential Participant organisation in the Australian Government funded Young and
Well Cooperative Research Centre, the Australian National University (ANU) developed the Uni Virtual
Clinic (UVC), which is a comprehensive, online service targeting mental disorders and mental health
problems in tertiary students. The UVC provides mental health support to university students and
targets all aspects of the mental health intervention spectrum including promotion, prevention,
early intervention, treatment, relapse prevention and recovery. Using participatory design
methods, the content and functionality of the UVC were developed in consultation with young
people (end-users), as well as university stakeholders and other service providers.

Participatory design is a user-centred process that promotes the active involvement of the event-
ual users of a product or service in its design and development (Schuler & Namioka, 1993). Using par-
ticipatory design methods ensures that the service can be tailored to the target population, creating
strong potential to foster a sense of empowerment and ownership, which, in turn, may improve
uptake of and engagement with the service (Bovaird, 2007). Participatory design methods also
involve other stakeholders such as service administrators or employees in coproducing the service
(Bovaird, 2007). One of the most powerful features of participatory design is that these stakeholders
are involved at each step of the design process, so that there is a focus on evaluation, maintenance
and ongoing progression, as well as an ability to design iteratively, making changes and improve-
ments to the service along the way (Gregory, 2003). This is particularly important when creating tech-
nology-based interventions that are subject to rapid technological evolution.

Our aims in the current paper are (a) to describe the participatory research and development
activities that were undertaken at the ANU to develop the UVC, (b) to present the resulting
content and functionality of the UVC intervention, and (c) to present data evaluating the participatory
design process with students.

334 L. M. FARRER ET AL.

Methods

The UVC was developed through multiple interlinked research and development stages in consul-
tation with young people, university stakeholders, and other service providers. These stages and
their results are outlined below.

Stage 1: initial scoping

The initial scoping stage was undertaken to inform the development of a service model (Stage 2) for
the UVC within a university setting. This stage comprised three systematic reviews, two surveys, and
four focus groups. Each method in this stage was designed to capture a different component to
inform the development of the UVC.

Systematic reviews
Three systematic reviews were conducted to examine the current evidence for online and mobile
interventions designed to support university student mental health.

The first review (Farrer et al., 2013) found that technology-based (e.g. mobile phone, internet)
interventions targeting mental health and related problems offered promise for students in university
settings. The data specifically suggested that technology-based cognitive behavioural therapy (CBT)
was likely to be particularly useful in targeting anxiety.

The second review (Gulliver, Farrer, et al., 2015) examined interventions targeting tobacco and
other substance use in students (excluding interventions targeting alcohol, which had previous
demonstrated efficacy in students [Carey, Scott-Sheldon, Elliott, Bolles, & Carey, 2009]) and concluded
that technology-based interventions increased abstinence from tobacco in students. However, there
were too few studies to draw conclusions about the efficacy of technology-based programmes for
other drugs. Together, these reviews demonstrated the high potential for the utility of a technol-
ogy-based intervention for mental health problems and substance use in university students.

The final review (Ali, Farrer, Gulliver, & Griffiths, 2015) examined peer-to-peer support interven-
tions for mental ill-health in young people, and demonstrated a lack of high-quality studies examin-
ing peer support interventions, despite their common use as an adjunct to internet interventions for
mental health problems. This finding emphasised the need to consider the addition and investigation
of the effectiveness of a peer-to-peer component in the UVC.

Stakeholder perspectives
Multiple cycles of obtaining stakeholder input and feedback were conducted using both qualitative
and quantitative methods. This development stage focused on examining the acceptability of a
virtual clinic in the university system. This work aimed to identify the optimal components and pro-
cesses for the UVC, including feedback on the types of services the clinic should offer (e.g. infor-
mation, prevention, recovery), the delivery modalities (e.g. web-based, smart phone), the role and
acceptability of symptom screening and monitoring, the level of intensity and autonomy of
content delivery (guided versus unguided), options for stepped care, and potential barriers and facil-
itators of engagement and uptake.

This data was collected initially through qualitative studies (focus groups with students at the ANU
conducted in December 2012) and quantitative studies (two large scale surveys – one conducted with
ANU university teaching staff in November-December 2013 and one conducted with ANU students in
October-December 2014). Analyses were undertaken to identify factors that could be used as a basis
for tailoring the UVC user experience. In addition, the university teaching staff survey was undertaken
due to the potential influence of staff on student mental health and help seeking pathways.

Qualitative studies. The qualitative methods primarily included four focus groups conducted with
university students (n = 5, 5, 4, 5) at the ANU. The focus groups (n = 19) demonstrated that students
viewed the concept of a university-specific virtual clinic favourably (Farrer, Gulliver, Chan, Bennett, &
Griffiths, 2015), despite expressing concerns about the privacy and security of their personal

BRITISH JOURNAL OF GUIDANCE & COUNSELLING 335

information (Gulliver, Bennett, et al., 2015). Conversely, some students also believed that an online
service would generate increased feelings of confidentiality when seeking help (Chan, Farrer, Gulliver,
Bennett, & Griffiths, 2016). Further, students indicated that they wished to connect with professionals
via the virtual clinic (e.g. make appointments with the university counselling centre), for the clinic to
provide information tailored to issues faced by students, and for the clinic to enable peer-to-peer
interaction. Overall, the results of these studies demonstrated a high level of support for a univer-
sity-specific virtual mental health clinic among students. In addition, a roundtable conducted with
ANU counselling centre staff demonstrated the need for and acceptability of a virtual clinic within
this service. However, constraints around current methods for booking appointments via telephone
and the required system changes needed for integration of the UVC into the counselling service,
meant that the ability for students to book face to face counselling appointments through the
virtual clinic was slated for a future iteration of the UVC.

Quantitative studies. The quantitative methods comprised two surveys, one with university staff
involved in teaching and supervising students, and the other with students, both at the ANU.

The student survey (Farrer, Gulliver, Bennett, Fassnacht, & Griffiths, 2016) was designed to examine
prevalence rates of mental disorders, risk factors, and feedback on UVC content and functionality
among a sample of Australian university students. A total of 611 ANU students were recruited to
the study (response rate 11.6%). The survey yielded a point prevalence rate of 7.9% for major
depression, and 17.5% for generalised anxiety disorder (GAD) (Farrer et al., 2016). The results indicated
that students in their first year of undergraduate study were at greatest risk of depression, whereas
female students, those who moved to attend university, and students experiencing financial stress
were at greatest risk of GAD. Students with experience of body image issues and lack of confidence
were at significantly greater risk of major depression. Finally, factors such as feeling too much
pressure to succeed, having a lack of confidence, and having difficulty coping with study were signifi-
cantly associated with risk of GAD among respondents. This survey confirmed the high rates of
common mental health problems in students, and also suggested that certain groups of students
may particularly benefit from accessing the UVC and other mental health services.

Although student feedback was critical to the intervention development process, gaining the per-
spectives of other relevant stakeholders such as academic staffwas also essential. University staffwho
have regular contact with students (lecturers, tutors, and research supervisors) are often a first point
of contact for academic accommodations (e.g. extensions of due dates) and referral to mental health
support. Thus, it is important to understand the attitudes of teaching staff about offering support to
students, and in particular, the types of support they are likely to recommend. Accordingly, we under-
took a survey examining the attitudes and mental health literacy of teaching staff at the ANU (n =
224). This survey demonstrated that many university teaching staff were highly sceptical about the
quality and effectiveness of online mental health interventions, with only 22.3% believing they
were a credible treatment option (Farrer, Gulliver, Bennett, & Griffiths, 2015). Further, a large pro-
portion of staff did not believe that it was part of their role to intervene in student mental health.
These findings were important as the attitudes of academic teaching staffmay affect their willingness
to recommend the UVC to students as a source of help. It was concluded that in order to maximise
uptake of the UVC, it is critical to increase positive attitudes towards online mental health interven-
tions among university teaching staff.

Stage 2: service model development

The second stage of the development of the UVC involved the iterative development and refinement
of the service model for the clinic. Based on the research conducted in Stage 1, the project team
developed a preliminary model which incorporated: (a) concepts of how a user might experience
or interact with the UVC, including over time; (b) possible components/features of the UVC; (c)
how the UVC might best communicate with other university service providers and other stake-
holders; (d) functionality and content of the clinic that may require tailoring for different sub-

336 L. M. FARRER ET AL.

groups of users; and (e) how new or existing users might be screened or directed to services as part of
their use of the clinic (e.g. stepped care).

Feedback on the service model was obtained through iterative cycles of prototype development
and user testing. The suggestions raised by users in a feedback session were used to re-develop the
existing prototype, which was then subjected to further stakeholder input during the next prototype
session. Three 1-hour prototype testing sessions were conducted with university students (n = 6; 3
male, 3 female, mean age = 21.2 years) (Gulliver, Bennett, et al., 2015).

In addition to this, the project team conducted several face-to-face discussions with university sta-
keholders to assess the information and service needs of different sub-groups of students (e.g. stu-
dents with a disability, postgraduates, international students). This was to ensure that the UVC
content and functionality was as relevant as possible to all potential user groups across the university.
The refinement of the service model was also informed by IT and clinical perspectives, which ensured
that the model was feasible, that user privacy and confidentiality could be adequately protected, and
that legal and duty of care obligations were considered.

Finally, a Student Leadership Group was established in 2014 and comprised 10–15 students each
year (2014–2016) from theANU to assist with the development and testing of the clinic. These students
provided input on all aspects of UVCdesign anddevelopment including graphic design, aswell as tools
and functionality. A number of students also contributed to content development by agreeing to be
interviewed about their experiences with mental health issues. These interviews were recorded, tran-
scribed, and used to inform the creation of real (de-identified) stories of student experiences of mental
health issues for the UVC. In addition to the input provided by the students in this group, a web
designer with extensive experience in designing mental health focused websites partnered with
the project to develop the logo, graphic design, and assist with the web functionality of the UVC.

Stage 3: software implementation

Stage 3 involved software implementation, which was undertaken by an in-house technical staff
team within the ANU, and was managed by senior staff with extensive multi-disciplinary experience
in e-health intervention research. The Student Leadership group of ANU students were also involved
in providing feedback into the design of the UVC at this stage.

Evaluation of participatory design

To assess the impact of participating in a user-centred design process, students who participated in
workshops and prototype testing sessions were asked to complete a brief survey at the end of the
session. Data were collected from 20 participants (male = 7, female = 12, no answer = 1) from three
prototype testing sessions, a student leadership group workshop, and the final evaluation of the
pilot UVC by the student leadership group.

Participants were asked to indicate their level of agreement with 32 adjectives, presented after a
statement (e.g. “After participating in today’s session, I feel… .safe/empowered/interrogated/
valued”). Of these 32 adjectives, 19 were positive (e.g. safe, empowered, respected) and 13 were
negative (e.g. bored, useless, interrogated). A full list of the adjectives is provided in Appendix.
Ratings of agreement were made on a 5-point scale ranging from strongly disagree (−2) to strongly
agree (2). A neutral response of “neither agree nor disagree” was coded as 0.

Results

The UVC intervention

The resulting intervention that was developed, the UVC, is an online programme for university stu-
dents to support and manage their mental health. The goal of the UVC is to connect students

BRITISH JOURNAL OF GUIDANCE & COUNSELLING 337

with the mental health information and support they need. This may include learning more about
mental disorders, completing a symptom quiz, trying a self-help therapy programme, or finding
out how to make an appointment with a face-to-face treatment provider.

The UVC is also designed to assist students with varying needs, including those who want to
manage their mental health, those who want to help their friends through a crisis, and those who
experience difficulties in times of stress. Figure 1 shows the original development model for the
UVC, demonstrating the interaction between the UVC and the student, as well as potential interaction
between other groups on or off campus including clinicians, researchers, service providers, and
student administration. However, the current pilot version of the clinic was modelled to accommo-
date student users only. In this version, students are not able to book appointments directly via
the website and there is no facility for counsellors to access any of the UVC quiz results or workbooks.
This departure from the planned model was partly a consequence of the high level of administrative
system change required to integrate the clinic within the existing university health system (which is
somewhat dependent on non-UVC systems), and partly because students were highly concerned
about the potential for other staff from the university being able to access their personal information
(e.g. staff that are not involved in health service delivery) (Gulliver, Bennett, et al., 2015).

The UVC targets four major groups of mental disorders that commonly affect university students:
mood disorders (major depression, bipolar disorder); anxiety, obsessive compulsive, and trauma-
related disorders (generalised anxiety disorder, social anxiety disorder, obsessive compulsive dis-
order, post traumatic stress disorder, specific phobia, panic disorder, agoraphobia); substance use dis-
orders (alcohol, smoking, and other drugs); and eating disorders (anorexia, bulimia, binge eating
disorder).

The UVC also contains resources targeting common issues experienced by university students that
impact on their mental health including: insomnia; suicide and self-harm; financial issues; loneliness/
social isolation; relationship issues; homesickness; adjustment to university (specific focus on inter-
national students); grief and loss; career and life after university; perfectionism; stress; physical
health (nutrition, exercise), time management and procrastination, disability, living arrangements;
sexual and gender identity; and exam anxiety.

Figure 1. Original service model of the UVC.

338 L. M. FARRER ET AL.

What does the UVC do?
The primary features of the UVC are:

(1) Information (factsheets tailored to the university student population, services, student stories
about mental health and other problems)

(2) Screening (quizzes, feedback and recommendations)
(3) Self-help tools (tailored guided and unguided pathways to care, access to online treatment

programmes)

These features are described in more detail below. Features intended to be added to future ver-
sions of the UVC are also described (e.g. providing access to professionals via an online booking
system and a peer-to-peer support forum/chat sessions).

Information
There is evidence that online information can be an effective intervention for mental health problems
(Donker, Griffiths, Cuijpers, & Christensen, 2009; Griffiths, Christensen, Jorm, Evans, & Groves, 2004).
The UVC comprises factsheets targeting all major mental disorders and related issues (see Figure 2),
and information about online and face-to-face services within the university setting and in the commu-
nity. A critical element of the UVC is that all the information provided is tailored to university students, a
feature not available on more generic online mental health websites or portals. The UVC contains infor-
mation about the prevalence of mental health problems in university students, signs and symptoms, risk
factors, impact on university life, help seeking, prevention, and treatment. Additional information is pre-
sented in the form of real (de-identified) stories of students’ experiences with mental health problems.

Figure 2. UVC factsheets.

BRITISH JOURNAL OF GUIDANCE & COUNSELLING 339

These stories are designed to give users an insight into the lived experiences of other students, which
has been shown previously to reduce stigma (Pinfold, Thornicroft, Huxley, & Farmer, 2005).

Screening
The UVC contains brief, validated screening measures for all mental disorders and some related
issues. Screeners provide feedback about symptoms, normative data that allows students to
compare their symptom levels with other young people their age, feedback about how their symp-
toms have changed since they last completed the screener, and recommendations for help based on
their scores.

Self-help tools
The UVC offers a range of self-help tools developed by the Centre for Mental Health Research (CMHR)
and other leading e-health researchers. These tools are based on evidence-based psychotherapeutic
principles such as cognitive behaviour therapy (CBT), interpersonal therapy (IPT), exposure therapy,
relaxation, problem solving therapy, and mindfulness. Users can complete these tools at their own
pace. Some tools are combined into treatment packages that guide the user through screening, psy-
choeducation, and treatment for a particular disorder.

Access to professionals
Future iterations of the UVC will explore the capacity for integration with existing mental health and
other services within and external to the university environment to facilitate access to professionals.
This may include an integrated booking system that allows students to make appointments with uni-
versity counselling and health centres, and online therapy services delivered by clinical psychology
interns through chat, e-mail or Skype.

Peer support
Another proposed function of the UVC will be to enable users to connect with other students through
an asynchronous forum and/or synchronous chat sessions. The forum and chat sessions will be mod-
erated to ensure user safety.

Problem-solving tool
A key priority of the UVC is to enable young people to find right help as quickly and easily as possible.
Young people may differ in their ability to recognise and label mental health problems in themselves or
their friends. To accommodate this, users have several options for seeking help from the UVC homepage.
Resources specific to mental disorders and related issues are clearly accessible from links on the home-
page. For users who are not sure what they are looking for, the problem-solving tool (see Figure 3) on the
homepage (see Figure 4) allows them to select from options that reflect their experience from a range of
statements (e.g. I’m feeling stressed, I can’t concentrate, I’m not coping). These statements link to path-
ways that direct the user to the most relevant content based on their current experience.

Guided and unguided pathways
Unguided pathways allow the user complete autonomy to browse all of the available UVC resources
at once, whereas guided pathways using the problem-solving tool as described above, guide the user
through available resources and make recommendations that best fit the user’s situation. Users can
choose between guided and unguided pathways at all times within the UVC.

User control of privacy
The user can choose how they wish to interact with the UVC. This is achieved by enabling the user to
specify how personal information is collected and stored by the clinic over time, and to configure
how the clinic interacts with them through personal reminders and updates.

340 L. M. FARRER ET AL.

Evaluation of user-centred design

Demographics
The age of participants ranged from 18 to 23 years (M = 20.8), and they were evenly distributed from
first year (n = 3; 15%) to fifth year students (n = 3; 15%). Just over half of participants had previously
experienced a mental health problem themselves (n = 11; 55%), and a majority had someone close to
them experience a mental health problem (n = 16; 80%). There were two international students
(10%), half lived on campus (n = 10; 50%), and they were from a wide range of disciplines including

Figure 4. UVC homepage.

Figure 3. Problem-solving tool.

BRITISH JOURNAL OF GUIDANCE & COUNSELLING 341

combined and single degrees in Psychology, Engineering, Law, International Relations, Arts, Science,
Commerce, Business Administration, and Asia Pacific Studies.

Evaluation data
Overall, attitudes towards participation were positive, with each of the 19 positive items receiving
average ratings above 0, indicating agreement (M = 0.96, SD = 0.24; range from 0.50 for “energised”
to 1.35 for “valued”). Each of the 13 negative items received average ratings below 0, indicating dis-
agreement (M =−1.50, SD = 0.46; ranging from −1.05 for “confused” to −1.70 for “useless”). Adjec-
tives that received the highest average scores (indicating strongest agreement) were valued (M =
1.35, SD = 0.49), safe (M = 1.30, SD = 0.57), helpful (M = 1.20, SD = 0.41), involved (M = 1.20, SD = 0.41),
respected (M = 1.15, SD = 0.37), and glad (M = 1.15, SD = 0.67). Those that received the lowest
average scores (indicating strongest disagreement) were useless (M =−1.70, SD = 0.47), powerless
(M =−1.65, SD = 0.49), interrogated (M =−1.65, SD = 0.49), uncomfortable (M =−1.65, SD = 0.49), and
distressed (M =−1.65, SD = 0.49).

Discussion

In this paper we describe the development of a complex online intervention for a university student
population. The UVC is an innovative, scalable, and flexibly built tool that has strong potential for dis-
semination in universities nationally and internationally. The UVC also has potential to serve as a
model for developing such clinics for young people in other settings such as schools.

Crucially, given the high rates of mental health problems in young people at university, the UVC
has the potential to create mentally healthy universities by reducing the prevalence of mental dis-
orders in a high risk group of young people and averting the associated distress, poor academic out-
comes, dropout, career loss, and impaired social functioning. University graduates can take their
learnings from the clinic with them into the workplace and other settings. However, future invest-
ment in this programme, as well as increased recognition and funding for the implementation and
evaluation of the UVC is required.

There were several benefits and challenges associated with using a participatory design approach
in the development of the UVC. Participatory design was fundamental to the UVC development
process and allowed for the UVC to be designed according to the priorities and preferences of
end users. Most students who had participated in the design process for the UVC reported that
they felt “valued”, “helpful”, “involved”, and “respected”, suggesting that being included in the
process was an empowering experience. Regardless of its potential to improve effectiveness, and
encourage user acceptance and adoption of a programme, it has been argued that a moral impera-
tive underlies participatory design to enable people to gain more control over their health care and
quality of life (Frauenberger, Good, Fitzpatrick, & Iversen, 2015).

Engaging students in the design process allowed the development team to consider the changing
needs of students and university environments, as well as technological advances in web design and
development strategies for e-mental health services. Conducting student surveys, engaging in
student consultations, and the creation of the Student Leadership Group resulted in wide promotion
of the UVC across the ANU campus. This improved the visibility of the UVC within the student com-
munity, and cultivated a positive attitude towards the UVC as an e-mental health tool for students,
particularly due to the university community’s engagement and consultation throughout the UVC
design and development process.

There were also challenges associated with using a participatory design approach. Engaging end-
users in design and development is time-consuming and resource intensive, and this impacted on
the time required for content creation and web development. It was also difficult to determine
when user engagement in a particular development phase was considered “complete”. In addition,
feasibility issues in the ability of the project team to implement all of the recommendations of the
end users, either due to technical difficulty, or budgetary constraints. Further, there was often

342 L. M. FARRER ET AL.

disagreement within and between end-user groups and other stakeholders regarding UVC content
and functionality, which was challenging to harmonise in the resulting programme. Additional
difficulties arose in engaging students in the participatory design process itself, as students often
had limited availability throughout the semester, and many were unavailable outside of semester
dates. Therefore, much of the design and development work for the UVC was constrained by the aca-
demic calendar.

The UVC has the potential to assist universities to increase their capacity to provide support to
students with mental health issues. Future directions for research include a proof-of-concept trial
to assess engagement and uptake of the programme, satisfaction, and mental health symptom
reduction, followed by a fully-powered, multi-site randomised controlled trial (RCT). If shown to be
effective, the programme can be implemented in several ways within a university setting. Given
that the programme targets all parts of the intervention spectrum, it has the potential to be used
not only as a treatment tool, but for large scale prevention/early intervention, or even relapse preven-
tion for students recovering from an acute episode of a mental health problem. Within a university
counselling environment, the UVC could be used to screen and triage students entering the service,
to provide resources to students while they are waiting for an appointment, to augment face-to-face
counselling with homework exercises that students can complete between sessions, or as a
step-down service to support students following counselling. It could also have utility for groups
of students who find it difficult to engage with on-campus services (e.g. remote students who
study predominantly off-campus).

University students experience high rates of mental health problems, and face significant chal-
lenges in terms of help seeking barriers and treatment accessibility. The UVC is a scalable, evi-
dence-informed intervention that has the potential to reduce burden on campus mental health
services, and treat or prevent the development of mental health problems in a high-risk group of
young people.

Acknowledgements

The authors would like to thank the following people for their contributions to the project: Rebekah Anne-Smith, Annika
Hellsing, Ada Tam, Gurbhej Singh Sandhu, David Berriman, Nhi Ha, John Cunningham, John Gosling, Jade Chan, Dimity
Crisp, Brad Carron-Arthur, Edwina Wright, Julia Reynolds, Rebecca Randall, Robert Tait, Alison Calear, Philip Batterham,
Helen Christensen, Mick Carney, and finally the staff and students from the Australian National University and the
Student Leadership Group who participated in numerous workshops, surveys and qualitative research activities for
the UVC.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Funding

This project was resourced by the Young and Well CRC (youngandwellcrc.org.au). The Young and Well CRC was estab-
lished under the Australian Government’s Cooperative Research Centres Programme.

Notes on contributors

Dr Louise Farrer is a Senior Research Fellow and Registered Psychologist at the Centre for Mental Health Research at the
Australian National University, Australia. Dr Farrer’s primary research interests are the development, evaluation, and
implementation of technology-based mental health prevention and treatment programmes.

Dr Amelia Gulliver is a Research Fellow with the ACACIA team at the Centre for Mental Health Research at the Australian
National University, Australia. She works on various projects relevant to mental health consumers and carers, technology-
based mental health interventions, and help-seeking.

BRITISH JOURNAL OF GUIDANCE & COUNSELLING 343

Natasha Katruss is an experienced research assistant who has worked on a variety of research projects relating to mental
health. She also has expertise in data analysis and web development.

Kylie Bennett is experienced in managing all aspects of complex research projects, particularly in the field of e-mental
health. She is responsible for the project development and management of several online mental health programmes.

Anthony Bennett is experienced in the software development of database driven web applications, and has a strong
understanding of the privacy and other issues associated with the implementation of internet interventions and associ-
ated research projects. He is responsible for the development of the software architecture for several online mental
health programmes.

Kathina Ali is a PhD research student working in the area of eating disorders, body image and barriers towards seeking
help. Her PhD research examines online peer-to-peer support for young people with eating, weight and shape concerns.

Emeritus Professor Katheen Griffiths’ research interests and experience focus on consumer informatics, the development
and evaluation of online psycho-educational, early intervention and preventive interventions, the evaluation of online
self-help communities, and the use of the internet in reducing stigma.

ORCID

Louise M. Farrer http://orcid.org/0000-0001-5694-4631
Kylie Bennett http://orcid.org/0000-0003-4160-492X
Kathleen M. Griffiths http://orcid.org/0000-0003-1556-0708

References

Ali, K., Farrer, L., Gulliver, A., & Griffiths, K. M. (2015). Online peer-to-peer support for young people with mental health
problems: A systematic review. JMIR Mental Health, 2(2), e19.

Auerbach, R. P., Mortier, P., Bruffaerts, R., Alonso, J., Benjet, C., Cuijpers, P.,… Collaborators, W. W.-I. (2018). WHO world
mental health surveys international college student project: Prevalence and distribution of mental disorders.
Journal of Abnormal Psychology, 127, 623–638.

Blanco, C., Okuda, M., Wright, C., Hasin, D. S., Grant, B. F., Liu, S.-M., & Olfson, M. (2008). Mental health of college students
and their non-college-attending peers: Results from the national epidemiologic study on alcohol and related con-
ditions. Archives of General Psychiatry, 65(12), 1429–1437.

Bovaird, T. (2007). Beyond engagement and participation: User and community coproduction of public services. Public
Administration Review, 67(5), 846–860.

Burns, J. M., Davenport, T. A., Durkin, L. A., Luscombe, G. M., & Hickie, I. B. (2010). The internet as a setting for mental health
service utilisation by young people. Medical Journal of Australia, 192(11), S22.

Carey, K. B., Scott-Sheldon, L. A., Elliott, J. C., Bolles, J. R., & Carey, M. P. (2009). Computer-delivered interventions to reduce
college student drinking: A meta-analysis. Addiction, 104(11), 1807–1819.

Chan, J. K., Farrer, L. M., Gulliver, A., Bennett, K., & Griffiths, K. M. (2016). University students’ views on the perceived
benefits and drawbacks of seeking help for mental health problems on the internet: A qualitative study. JMIR
Human Factors, 3(1), e3.

Conley, C. S., Durlak, J. A., Shapiro, J. B., Kirsch, A. C., & Zahniser, E. (2016). A meta-analysis of the impact of universal and
indicated preventive technology-delivered interventions for higher education students. Prevention Science, 17,
659–678.

Davies, E. B., Morriss, R., & Glazebrook, C. (2014). Computer-delivered and web-based interventions to improve
depression, anxiety, and psychological well-being of university students: A systematic review and meta-analysis.
Journal of Medical Internet Research, 16, e130.

Donker, T., Griffiths, K. M., Cuijpers, P., & Christensen, H. (2009). Psychoeducation for depression, anxiety and psychologi-
cal distress: A meta-analysis. BMC Medicine, 7, 79.

Eisenberg, D., Hunt, J., Speer, N., & Zivin, K. (2011). Mental health service utilization among college students in the United
States. Journal of Nervous and Mental Disease, 199(5), 301–308.

Farrer, L., Gulliver, A., Bennett, K., Fassnacht, D. B., & Griffiths, K. M. (2016). Demographic and psychosocial predictors of
major depression and generalised anxiety disorder in Australian university students. BMC Psychiatry, 16, 241.

Farrer, L., Gulliver, A., Bennett, K., & Griffiths, K. M. (2015). Exploring the acceptability of online mental health interventions
among university teaching staff: Implications for intervention dissemination and uptake. Internet Interventions, 2,
359–365.

Farrer, L., Gulliver, A., Chan, J. K., Bennett, K., & Griffiths, K. M. (2015). A virtual mental health clinic for university students: A
qualitative study of end-user service needs and priorities. JMIR Mental Health, 2(1), e2.

Farrer, L., Gulliver, A., Chan, J. K. Y., Batterham, P. J., Reynolds, J., Calear, A. L.,… Griffiths, K. M. (2013). Technology-based
interventions for mental health in tertiary students: Systematic review. Journal of Medical Internet Research, 15(5), e101.

344 L. M. FARRER ET AL.

Frauenberger, C., Good, J., Fitzpatrick, G., & Iversen, O. S. (2015). In pursuit of rigour and accountability in participatory
design. International Journal of Human-Computer Studies, 74, 93–106.

Givens, J. L., & Tjia, J. (2002). Depressed medical students’ use of mental health services and barriers to use. Academic
Medicine, 77(9), 918–921.

Gregory, J. (2003). Scandinavian approaches to participatory design. International Journal of Engineering Education, 19(1),
62–74.

Griffiths, K. M., & Christensen, H. (2007). Internet-based mental health programs: A powerful tool in the rural medical kit.
Australian Journal of Rural Health, 15(2), 81–87.

Griffiths, K. M., Christensen, H., Jorm, A. F., Evans, K., & Groves, C. (2004). Effect of web-based depression literacy and cog-
nitive-behavioural therapy interventions on stigmatising attitudes to depression: Randomised controlled trial. British
Journal of Psychiatry, 185, 342–349.

Gulliver, A., Bennett, K., Bennett, A., Farrer, L. M., Reynolds, J., & Griffiths, K. M. (2015). Privacy issues in the development of
a virtual mental health clinic for university students: A qualitative study. JMIR Mental Health, 2(1), e9.

Gulliver, A., Farrer, L., Chan, J. K., Tait, R. J., Bennett, K., Calear, A. L., & Griffiths, K. M. (2015). Technology-based interven-
tions for tobacco and other drug use in university and college students: A systematic review and meta-analysis.
Addiction Science & Clinical Practice, 10, 5.

Harrer, M., Adam, S. H., Baumeister, H., Cuijpers, P., Karyotaki, E., Auerbach, R. P.,… Ebert, D. D. (2018). Internet interven-
tions for mental health in university students: A systematic review and meta-analysis. International Journal of Methods
in Psychiatric Research, 28(2), e1759.

Jennings, A., Powell, J., Armstrong, N., Sturt, J., & Dale, J. (2009). A virtual clinic for diabetes self-management: Pilot study.
Journal of Medical Internet Research, 11(1), e10.

Kethers, S., Lamb, P., Lu, S., Paris, C., Wilkinson, R., & Griffiths, K. M. (2006). Combining personalization and privacy to deliver
remote care to people with depressive illnesses. Paper presented at the CHI-2006 Workshop on Privacy-Enhanced
Personalization, Montreal, Canada.

Leahy, C. M., Peterson, R. F., Wilson, I. G., Newbury, J. W., Tonkin, A. L., & Turnbull, D. (2010). Distress levels and self-
reported treatment rates for medicine, law, psychology and mechanical engineering tertiary students: Cross-sectional
study. Australian and New Zealand Journal of Psychiatry, 44(7), 608–615.

Mowbray, C. T., Mandiberg, J. M., Stein, C. H., Kopels, S., Curlin, C., Megivern, D.,… Lett, R. (2006). Campus mental health
services: Recommendations for change. American Journal of Orthopsychiatry, 76(2), 226–237.

Oh, E., Jorm, A., & Wright, A. (2009). Perceived helpfulness of websites for mental health information. Social Psychiatry and
Psychiatric Epidemiology, 44(4), 293–299.

Pinfold, V., Thornicroft, G., Huxley, P., & Farmer, P. (2005). Active ingredients in anti-stigma programmes in mental health.
International Review of Psychiatry, 17(2), 123–131.

Rickwood, D. J., & Braithwaite, V. A. (1994, August). Social-psychological factors affecting help-seeking for emotional pro-
blems. Social Science & Medicine, 39(4), 563–572.

Rickwood, D. J., Deane, F. P., & Wilson, C. J. (2007). When and how do young people seek professional help for mental
health problems? Medical Journal of Australia, 187(Suppl. 7), S35–S39.

Ryan, M. L., Shochet, I. M., & Stallman, H. M. (2010). Universal online interventions might engage psychologically dis-
tressed university students who are unlikely to seek formal help. Advances in Mental Health, 9(1), 73–83.

Said, D., Kypri, K., & Bowman, J. (2013). Risk factors for mental disorder among university students in Australia: Findings
from a web-based cross-sectional survey. Social Psychiatry and Psychiatric Epidemiology, 48(6), 935–944.

Schuler, D., & Namioka, A. (1993). Participatory design: Principles and practices. Hillsdale, NJ: Lawrence Erlbaum.
Stallman, H. M. (2010). Psychological distress in university students: A comparison with general population data.

Australian Psychologist, 45(4), 249–257.
Wynaden, D., Wichmann, H., & Murray, S. (2013). A synopsis of the mental health concerns of university students: Results

of a text-based online survey from one Australian university. Higher Education Research & Development, 32(5), 846–860.

BRITISH JOURNAL OF GUIDANCE & COUNSELLING 345

Appendix. Participation measure

Thank you for your participation in today’s activity. We value your feedback and welcome your responses to the ques-
tions below.
(1) After participating in today’s session, I feel…

Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree
Empowered
Valued
Helpful
Scrutinised
Connected
Satisfied
Bored
Useless
Caring
Involved
Cooperative
Respected
Frustrated
Powerless
Important
Understood
Overwhelmed
Excited
Stressed
Creative
Confronted
Energised
Consulted
Interrogated
Safe
Confused
Insightful
Supported
Distressed
Inconvenienced
Uncomfortable
Glad

(2) Did you feel any other emotions besides the ones listed above?
______________________________________________________________________________________ (please specify)
(3) Is there anything that could be improved about the consultation process you’ve participated in today?
___________________________________________________________________________________________________
___________________________________________________________________________________________________
___________________________________________________________________________________________________
___________________________________________________________________________________________________
Age _____
Gender (Please circle) _____M/_____F
Year of degree (e.g. 1st) _________________
Area of study (e.g. Economics) ___________________
Are you an International student? _____Y/_____N
Do you live in an ANU residential hall or college? _____Y/_____N
Have you experienced a mental health problem before? _____Y/_____N
Has someone close to you experienced a mental health problem before? _____Y/_____N
Have you participated in a Virtual Clinic research activity before? _____Y/_____N
If so, what have you participated in?___________________________________________

346 L. M. FARRER ET AL.

Copyright of British Journal of Guidance & Counselling is the property of Routledge and its
content may not be copied or emailed to multiple sites or posted to a listserv without the
copyright holder’s express written permission. However, users may print, download, or email
articles for individual use.

  • Abstract
  • Introduction
  • Methods
    • Stage 1: initial scoping
      • Systematic reviews
      • Stakeholder perspectives
    • Stage 2: service model development
    • Stage 3: software implementation
    • Evaluation of participatory design
  • Results
    • The UVC intervention
      • What does the UVC do?
      • Information
      • Screening
      • Self-help tools
      • Access to professionals
      • Peer support
      • Problem-solving tool
      • Guided and unguided pathways
      • User control of privacy
    • Evaluation of user-centred design
      • Demographics
      • Evaluation data
  • Discussion
  • Acknowledgements
  • Disclosure statement
  • Notes on contributors
  • ORCID
  • References
  • Appendix. Participation measure

JDNP-D-20-00049 Page 122 03/07/21 5:32 AM

122 Journal of Doctoral Nursing Practice, Volume 14, Number 2, 2021 © 2021 Springer Publishing Company
http:// dx. doi. org/ 10. 1891/JDNP-D-20-00049

Improving Suicidal Ideation
Screening and Suicide

Prevention Strategies on Adult
Nonbehavioral Health Units

Anne C. Lindstrom, DNP, APRN
Rush University, Northwestern Medicine Central DuPage Hospital,

Winfield, IL
Melinda Earle, DNP, RN

Rush University, Chicago, IL

Background: Suicide is a risk in hospitalized patients within and outside of behavioral health units.
Williams et al. (2018). Incidence and method of suicide in hospitals in the United States. Williams
et al. (2018), suicides occur annually in hospitals, 14%–26% of occurrences outside of a behavioral
health unit. Objectives: The purpose of this project was to improve compliance with universal
patient suicide ideation screening and targeted actions for patients screening positive admitted
to nonbehavioral health units. Methods: Electronic medical record (EMR) changes to support
screening and targeted patient safety measure documentation were implemented. Nursing educa-
tion was provided to support these changes. Pre- and postassessments were used to measure knowl-
edge gained from the education. A compliance report was generated from the EMR to measure
compliance with universal screening and patient safety measures. Results: In a 4-month period,
screening compliance improved by 20.6%, bedside safety companion use was 100%, suicide precau-
tions, documentation 82.5%, and a behavioral health social work note documented 76% of the time.
Conclusions: Universal screening improvement is feasible and has the potential to improve patient
safety. Implementation should be considered across health care organizations. Implications for
Nursing: Providing nursing education and easier EMR workflows for nursing documentation of
suicide screening and prevention strategies improves patient safety.

Keywords: suicide; screening; interventions; behavioral health

Background

Suicide is the 10th leading cause of death in the United
States, furthermore the rate is increasing in almost all
states despite a decrease worldwide (Grumet et al.,
2019). Currently, suicide claims one life every 40 sec-
onds in the United States (World Health Organization
[WHO], 2014). The majority of suicides are preceded

by warning signs, either verbal or behavioral (WHO,
2014).

Healthcare organizations have been identified by
the U.S Surgeon General as a setting that can impact
suicide rates (U.S. Department of Health and Human
Services, 2012). Many people who die by suicide present
to healthcare organizations for nonbehavioral health

ID:ti0005ID:p0050ID:p0055ID:p0060ID:p0065ID:p0070ID:ti0010ID:p0075ID:ti0015ID:p0080ID:p0085 ID:p0090ID:p0095ID:p0100ID:p0105ID:p0110ID:p0115ID:ti0005ID:p0050ID:p0055ID:p0060ID:p0065ID:p0070ID:ti0010ID:p0075ID:ti0015ID:p0080ID:p0085 ID:p0090ID:p0095ID:p0100ID:p0105ID:p0110ID:p0115ID:ti0005ID:p0050ID:p0055ID:p0060ID:p0065ID:p0070ID:ti0010ID:p0075ID:ti0015ID:p0080ID:p0085 ID:p0090ID:p0095ID:p0100ID:p0105ID:p0110ID:p0115ID:ti0005ID:p0050ID:p0055ID:p0060ID:p0065ID:p0070ID:ti0010ID:p0075ID:ti0015ID:p0080ID:p0085 ID:p0090ID:p0095ID:p0100ID:p0105ID:p0110ID:p0115

JDNP-D-20-00049 Page 123 03/07/21 5:32 AM

123Improving Suicidal Ideation Screening

complaints prior to committing suicide (Grumet et al.,
2019).

Suicide is a risk in hospitalized patients and occurs
both inside and outside of behavioral health units.
Williams et al. (2018) cite 48.5 to 64.9 suicides occur
annually in hospitals and 14%–26% of occurrences
outside of a behavioral health unit. Therefore, univer-
sal screening for suicide ideation in nonbehavioral
healthcare settings is considered to be an important
and effective piece of suicide prevention (Horowitz et
al., 2017). Systematic identification and assessment of
patients at risk for suicide is crucial to decreasing the
incidence (Grumet et al., 2019).

Rapid identification of patients at risk for suicide
accelerates provision of a safe environment for care
(Grumet et al., 2019). Rapid identification of patients
at risk poses a challenge. Targeted screening has the
potential to miss patients who may be at risk, however,
universal screening has the potential to identify patients
not truly at-risk increasing resource use (Grimley-
Baker, 2018). The U.S Preventative Services Task Force
in 2014 stated the current evidence supporting universal
screening is insufficient to assess the benefit versus harm
(LeFevre, 2014). Nonetheless, The Joint Commission
(2018) has encouraged organizations to develop univer-
sal screening policies and requires that all patients who
present with a primary behavioral health complaint be
screened. The discrepancy in recommendations has left
healthcare organization to develop individual policies
and procedures based on local level resource availabil-
ity and expert opinion (Grimley-Baker, 2018). Universal
screening has been adopted in many organizations and
feasibility has been established with a compliance rate
of 84%–92% (Boudreaux et al., 2016; Stuck et al., 2017).
However, barriers to screening exist and mostly focus on
health care provider concerns. Identified barriers include
concern screening may increase suicidal thoughts, lack
of time and privacy to complete screening, lack of
patient willingness to complete, communication chal-
lenges such as language barriers, lack of training, and
lack of standard protocol (Bolton et al., 2015; Giacchero
et al., 2017; Petrik et al., 2015).

Screening is not enough to keep suicidal patients
safe. A safe care environment must be provided while
further assessment is completed. Root cause of suicide
events in nonbehavioral health inpatient units include
lack of risk assessment, poor handoff communication,
lack of staff training, lack of repeated assessments, and
lack of supervision (Grimley-Baker, 2018; Mills  et  al.,
2014). The stated root causes highlight the importance
of providing a safe care environment. Staff knowledge,
the involvement of clinical experts, establishment of
clinical competence, and a team approach to patient
care can mitigate risks (Grimley-Baker, 2018; Mills

et al., 2014). Additionally, environmental resources offer
means to commit suicide and risk must be mitigated.
The most common methods noted were hanging, over-
dose, and cutting (Mills et al., 2014). Access to means
should be evaluated and any potential risks removed
and the patient supervised at all times including while
in the bathroom (Grimley-Baker, 2018).

At a Midwestern suburban medical center, a policy
requires universal suicidal ideation screening as part of
the nursing admission assessment. Patients are asked
two questions: (a) are you feeling hopeless about the
present/future and (b) have you had any thoughts about
taking your life? A patient who answers yes to the first
question has a behavioral health social work consult
placed. Subsequently, a patient who answers yes to the
second question requires orders placed for a bedside
safety companion, implementation of a prescribed set
of suicide precautions and a consult to the behavioral
health services social worker. The behavioral health
social worker completes the Columbia Suicide Severity
Rating Scale (C-SSRS) (Posner et al., 2011). A com-
pliance report, generated from the electronic medical
record  (EMR), 1 year after the policy implementation
indicated a complete screen was documented 79% of
the time. When a positive screen was noted the safety
companion order was placed 35% of the time, suicide
precautions were present 51% of the time, and an order
for a behavioral health social work consult was present
81% of the time.

There is limited literature outlining how to improve
suicide screening in the inpatient setting. A literature
search was completed using CINAHL and PsycInfo
with a date range of 2012–2019. Search terms included
suicide, suicidal ideation, hospital, inpatient, and adult.
Six articles discuss implementation of suicide screen-
ing within the hospital but no literature was found on
how to improve screening compliance when a process is
already established (Adams, 2013; Horowitz et al., 2013;
Huh et al., 2012; Maclay, 2012; Roaten et al., 2018).

At the medical center in this study, there were sev-
eral causal factors, identified through chart reviews
and focus groups with nurses, influencing compli-
ance. Documentation of the screening was optional
in the electronic medical record, the nurse completing
the admission screen could chose not to answer the
question. If the question was answered positively there
were no functions in the electronic medical record
to prompt the nurse to enter the subsequent orders.
Additionally, there was no targeted education pro-
vided to the nurses when universal screening was ini-
tiated. Recurring education around suicidal ideation,
safety of suicidal patients, and the policy was not pro-
vided to nurses. Targeted intervention was needed to
improve  identification and safety of suicidal patients,

ID:p0090ID:p0095ID:p0100ID:p0105ID:p0110ID:p0115ID:p0090ID:p0095ID:p0100ID:p0105ID:p0110ID:p0115ID:p0090ID:p0095ID:p0100ID:p0105ID:p0110ID:p0115ID:p0090ID:p0095ID:p0100ID:p0105ID:p0110ID:p0115

JDNP-D-20-00049 Page 124 03/07/21 5:32 AM

124 Lindstrom et al.

reduce risk, and potentially save lives. (Grumet et al.,
2019).

Objective

The primary aim of the project was to improve suicide
screening of patients admitted to nonbehavioral health
units to 100%. The secondary aims were to improve
timeliness of a behavioral health social work consult
note entered into the electronic medical record, with a
goal of 100% entered within 24 hours. Additionally, the
secondary aims were to ensure a safety companion is
provided to 100% of patients and ensure suicide precau-
tions are initiated for 100% of patients who screened
positive for suicidal ideation

Methods

The project was deemed nonhuman subjects research
by the institutional review board. The define, measure,
analyze, improve and control (DMAIC) methodol-
ogy was used as a process improvement framework
for this project. The healthcare organization uses the
DMAIC methodology for quality improvement. The
chosen methodology allowed a logical and standardized
approach to improving clinical care. The problem was
clearly defined and measured. Possible solutions were
analyzed and implemented. Finally, a control plan was
developed to ensure sustainability (American Society
for Quality, 2019).

The interventions were developed based on a review
of the literature discussing initial implementation of
suicide screening (Adams, 2013; Horowitz et al., 2013;
Huh et al., 2012; Maclay, 2012; Roaten et al., 2018) and
focus group findings from key stakeholders, including
bedside nurses. The intervention included a change to
the electronic medical record, education for nurses, and
compliance monitoring.

Electronic Medical Record
Improvements

Roaten et al. (2018) discuss changes to the electronic
medical record as an integral piece to ensuring suicide
screening is complete. This was also an area identified
in the focus group discussions. The electronic medical
record change included adding an additional answer
option to the suicide screening question and making the
question required documentation within the electronic
medical record. Previously, when screening a patient for
suicide on admission the answer options were yes or
no. Unable to assess was added as an additional option.
The addition of this option allows nurses to complete
the question even when a patient is unable to answer.
The answer is used for a patient who is intubated or

otherwise unable to speak. If unable to assess is used as
an answer the nurse is prompted periodically through-
out the patients stay to reassess and answer yes or no.

Nursing Protocol Development

A nursing protocol was developed with the goal of
improving compliance of orders for suicide precau-
tions, safety companion, and behavioral health social
worker consult. The protocol allows nurses to enter
these orders per protocol for patients screening posi-
tive for suicidal ideation. An order for suicide precau-
tions includes an environmental assessment to ensure
no hazardous objects are available to the patient. The
protocol ensures the appropriate orders are entered and
the patient remains in a safe environment under con-
stant 1:1 supervision. The provider is then notified after
the patient’s safety has been assured. The protocol was
approved by the hospital’s Chief Nurse Executive and
the Medical Executive Committee.

A best practice alert for appropriate order entry was
also added. The best practice alert prompts the nurse to
take action in the case of a patient who screens posi-
tive for current suicidal ideation. The best practice alert
includes a reminder to place an order for suicide precau-
tions and a behavioral health social work consult.

Compliance Monitoring

Data on screening compliance and subsequent actions
was collected using a report developed by the analyt-
ics team. The report pulled information directly from
the electronic medical record and included individual
patient level data. Data fields included admission data,
response to screening question and if required date and
time policy actions were taken.

Nursing Education

Compliance with screening and the bundled actions
required by the policy was supported by providing edu-
cation to the nursing staff (Manister et al., 2017; Roaten
et al., 2018). Education was developed in an electronic
learning module and was required of all adult inpa-
tient nurses. The content was developed and evaluated
by behavioral health experts, professional practice  and
nursing education, and accreditation experts within the
organization. The education included content detail-
ing the care of a suicidal patient. The screening process
was reviewed in addition to actions required by policy
and the nursing protocol (see Figure 1). The education
included an interactive portion where the nurses were
asked to hover over a picture of a patient room to iden-
tify any objects that could be considered a safety hazard.

ID:ti0020ID:p0120ID:ti0025ID:p0125ID:p0130ID:ti0030ID:p0135ID:ti0035ID:p0140ID:p0145ID:ti0040ID:p0150ID:ti0045ID:p0155 ID:p0160ID:p0270ID:ti0050ID:p0275ID:p0285ID:p0295ID:p0305ID:ti0020ID:p0120ID:ti0025ID:p0125ID:p0130ID:ti0030ID:p0135ID:ti0035ID:p0140ID:p0145ID:ti0040ID:p0150ID:ti0045ID:p0155 ID:p0270ID:ti0050ID:p0275ID:p0285ID:p0295ID:p0305ID:ti0055ID:p0310ID:p0315ID:p0320ID:p0325ID:p0330ID:ti0060ID:p0335ID:ti0065ID:p0340ID:ti0020ID:p0120ID:ti0025ID:p0125ID:p0130ID:ti0030ID:p0135ID:ti0035ID:p0140ID:p0145ID:ti0040ID:p0150ID:ti0045ID:p0155 ID:p0160ID:p0270ID:ti0050ID:p0275ID:p0285ID:p0295ID:p0305ID:ti0020ID:p0120ID:ti0025ID:p0125ID:p0130ID:ti0030ID:p0135ID:ti0035ID:p0140ID:p0145ID:ti0040ID:p0150ID:ti0045ID:p0155 ID:p0160ID:p0270ID:ti0050ID:p0275ID:p0285ID:p0295ID:p0305

JDNP-D-20-00049 Page 125 18/07/21 11:50 PM

125Improving Suicidal Ideation Screening

A pre- and postassessment was included to determine
the knowledge gain (see Figure 2 ).

Th e staffi ng coordinator, who oversees the safety
companions for the hospital, sent a daily report of
patients admitted with suicidal ideation. Observation
and chart audits were completed to ensure suicide pre-
cautions were in place and safety companion was at
bedside. Chart audits were completed to ensure orders
and screening were completed appropriately.

Results

Th e nursing education module was completed by 91.5%
( N = 344) of eligible nurses. Th e pretest baseline knowl-
edge level was noted to be high with 89.2% of the ques-
tions on the pretest answered correctly. Th e posttest
assessment data demonstrated 92.1% of the questions
were answered correctly, an improvement of 2.9% (See
Figure 3 ).

Data was collected from January 1, 2020, through
May 31, 2020. Goals of 100% were identifi ed for each
metric, the organization is a high reliability organiza-
tion and strives for 100% compliance in any metric
related to patient safety. In the data collection period

6,229 patients were admitted to adult nonbehavioral
health units within the hospital. Of these patients
99.6% ( N = 6210) were screened for suicidal ideation.
Of the patients who were screened, 95.8% ( N = 5947)
screened negative, 1% ( N = 63) screened positive, and
3.2% ( N = 200) were unable to be assessed. Th is demon-
strated a 20.6% improvement in screening compliance
(see Figure 4 ).

Patients who screened positive for suicidal ideation
were evaluated for adherence to policy required actions
including an order placed for a consult to the behavioral
health social worker, an order entered for suicide pre-
cautions and an order placed for a safety companion. Of
these patients, 84% ( N = 53) had an order for a consult
to the behavioral health social worker, 57% ( N = 36) had
an order entered for suicide precautions and 47.6% ( N =
30) has an order for a safety companion. Th is represents
a 3%, a 6% and a 12.6% improvement, respectively (See
Figure 5 ).

Observation and chart audits were also performed.
Based on audit data, 100% ( N = 63) of patients had a
safety companion at bedside, 82.5% had safety precau-
tions documented ( N = 52) and 76% ( N = 48) had a
behavioral health social work note in the chart within
24 hours of admission.

ID:p0160ID:p0270ID:ti0050ID:p0275ID:p0285ID:p0295ID:p0305ID:p0270ID:ti0050ID:p0275ID:p0285ID:p0295ID:p0305ID:ti0055ID:p0310ID:p0315ID:p0320ID:p0325ID:p0330ID:ti0060ID:p0335ID:ti0065ID:p0340

Figure 1. Project intervention overview.

ID:p0160ID:p0270ID:ti0050ID:p0275ID:p0285ID:p0295ID:p0305ID:p0160ID:p0270ID:ti0050ID:p0275ID:p0285ID:p0295ID:p0305

JDNP-D-20-00049 Page 126 18/07/21 11:50 PM

126 Lindstrom et al.

Figure 2. Nursing education content overview.

Figure 3. Nursing education pre- and posttest assessment.

• Suicide Screening
• Review of assessment questions
• Completion of Columbia Suicide Severity Rating Scale for those who screen positive or present with a primary behavioral

health complaint
• Electronic medical record workfl ow for screening completion

• Action to be taken if a patient screens positive.
• Patients who screen positive need to be kept safe immediately
• Patients should be kept under 1:1 observation
• Institute the Nursing Suicide Protocol

• Nursing Suicide Protocol
• RN ensures 1:1 patient observation initiated
• Protocol orders entered for Suicide Precautions and Behavioral Health Social Work Consult

• Environmental Mitigations
• Patient and belonging search and removal of possible hazards
• Room check and removal of possible hazards
• Dietary safety tray

• Patient Monitoring Requirements
• 1:1 observation at all times
• Review of transport and patient ambulation process

• Documentation Requirements

ID:p0165ID:p0280 ID:p0290ID:p0300ID:ti0055ID:p0310ID:p0315ID:p0320ID:p0160ID:p0165ID:p0280ID:ti0070 ID:p0345ID:p0347ID:p0348ID:p0165ID:p0280 ID:p0290ID:p0300ID:ti0055ID:p0310ID:p0315ID:p0320ID:p0165ID:p0280 ID:p0290ID:p0300ID:ti0055ID:p0310ID:p0315ID:p0320

JDNP-D-20-00049 Page 127 18/07/21 11:50 PM

127Improving Suicidal Ideation Screening

Conclusion

Th e results are overall positive. An increase in screening
compliance was achieved. Literature suggested a com-
pliance rate benchmark of 84%–92%. ( Boudreaux et al.,
2016 ; Stuck et al., 2017 ). Th e project achieved a compli-
ance rate of 99.6%.

A safe care environment must be provided to patients
identifi ed as at risk for suicide. Th e project aimed to
address the root causes of suicide events discussed in the

literature including lack of risk assessment, lack of staff
training, and lack of repeat assessment and supervision
( Grimley-Baker, 2018 ; Mills et al., 2014 ).

A lack of risk assessment was addressed with uni-
versal screening. Patients who screen positive are kept
under direct supervision, addressing the risk of lack of
supervision. Consistency was obtained in placing orders
for suicide precautions, safety companion, and consult
to the behavioral health social worker. Additionally,
the audits confi rmed the correct actions were taken. A

ID:p0290ID:p0300ID:ti0055ID:p0310ID:p0315ID:p0320ID:p0345ID:p0347ID:p0348

Figure 4. Suicide screening.

Figure 5. Order compliance.

ID:p0290ID:p0300ID:ti0055ID:p0310ID:p0315ID:p0320ID:p0290ID:p0300ID:ti0055ID:p0310ID:p0315ID:p0320

JDNP-D-20-00049 Page 128 03/07/21 5:32 AM

128 Lindstrom et al.

safety companion was providing direct supervision and
a behavioral health social worker evaluated the patient
within 24 hours.

Staff knowledge and training was also identified as an
area that can mitigate patient risk (Grimley-Baker, 2018;
Mills et al., 2014). The education provided to the nurses
was comprehensive and included care of the patient from
the time of the screening to discharge. A large percent-
age of nurses attended the training during the defined
period, 91.5%. The majority of nurses who did not attend
the training were on a leave of absence during the train-
ing period. Clinical leaders received notification of the
training needs and followed up with individual nurses as
needed. New hire training will be addressed during ori-
entation, the electronic learning module will be assigned
as mandatory education to all nurses hired going forward.

In conclusion, the project demonstrated improve-
ment in universal suicide screening and implementation
of appropriate safety measures for patients in adult non-
behavioral health units.

Limitations

The results are limited in generalizability as this project
was completed at one site. Additionally, the project was
implemented two months before the COVID-19 pan-
demic impacted the hospital. Disaster state charting was
implemented which may have impacted documentation
compliance.

Implications for Nursing

Universal suicide ideation screening can be successfully
implemented when nurses are provided clear policies,
targeted education, and nurse-driven protocol order sets
and best practice alerts embedded in the electronic medi-
cal record. Especially when order sets and documentation
mirror established nursing workflow. This allows nurses
to be more efficient and improves screening and interven-
tion compliance. These measures increased nurse auton-
omy and efficiency and avoided multiple phone calls to
providers.

References

Adams, N. (2013). Developing a suicide precaution procedure. Medsurg
Nursing, 22(6), 383–386.

American Society for Quality. (2019). The define, measure, ana-
lyze, improve, control (DMAIC) process. https://asq.org/
quality-resources/dmaic

Bolton, J., Gunnell, D., & Turecki, G. (2015). Suicide risk assessment
and intervention in people with mental illness. BMJ, 351(nov091),
h4978–h4978. https://doi.org/10.1136/bmj.h4978

Boudreaux, E., Carmargo, C., Arias, S., Sullivan, A., Allen, M.,
Goldstein, A., Manton, A. P., Espinola, J. A., & Miller, I. W.

(2016). Improving suicide risk screening and detection in the
emergency department. American Journal of Preventive Medicine,
50(4), 445–453. https://doi.org/10.1016/j.amepre.2015.09.029

Giacchero Vedana, K., Magrini, D., Zanetti, A., Miasso, A., Borges,
T., & dos Santos, M. (2017). Attitudes towards suicidal behavior
and associated factors among nursing professionals: A quantitative
study. Journal of Psychiatric and Mental Health Nursing, 24(9–10),
651–659. https://doi.org/10.1111/jpm.12413

Grimley-Baker, K. (2018). Preventing suicide beyond psychiat-
ric units. Nursing, 48(3), 59–61. https://doi.org/10.1097/01.
nurse.0000529816.67148.e9

Grumet, J., Hogan, M., Chu, A., Covington, D., & Johnson, K. (2019).
Compliance standards pave the way for reducing suicide in health
care systems. Journal of Health Care Compliance, 21, 17–26.

Horowitz, L., Bourdreaux, E., Schoenbaum, M., Pao, M., & Bridge,
J. (2017). Universal suicide risk screening in the hospital set-
ting: Still a Pandora’s box? The Joint Commission Journal on
Quality and Patient Safety, 44(1), 1–3. https://doi.org/10.1016/j.
jcjq.2017.11.001

Horowitz, L., Snyder, D., Ludi, E., Rosenstein, D., Kohn-Godbout, J.,
Lee, L., Cartledge, T., Farrar, A., & Pao, M. (2013). Ask suicide-
screening questions to everyone in medical settings: The asQ’em
quality improvement project. Psychosomatics, 54(3), 239–247.
https://doi.org/10.1016/j.psym.2013.01.002

Huh, J., Weaver, C., Martin, J., Caskey, N., O’Riley, A., & Kramer,
B. (2012). Effects of a late-life suicide risk-assessment train-
ing on multidisciplinary healthcare providers. Journal of
the American Geriatrics Society, 60(4), 775–780. https://doi.
org/10.1111/j.1532-5415.2011.03843.x

Joint Commission Suicide Prevention Working Group. (2018).
Suicide prevention resources to support Joint Commission
accredited organizations implementation of NPSG 15.01.01,
revised November 2018. https://www.jointcommission.org/
npsg_150101_suicide_prevention_resources/

LeFevre, M. (2014). Screening for suicide risk in adolescents, adults,
and older adults in primary care: recommendations from the
U.S. Preventive Services Task Force. Annals of Internal Medicine,
160(10), 1–22. https://doi.org/10.7326/p14-9016

Maclay, T. (2012). How to save a life: A suicide prevention protocol
for critical care. Nursing Critical Care, 7(4), 17–21. https://doi.
org/10.1097/01.ccn.0000415623.56868.c3

Manister, N., Murray, S., Burke, J., Finegan, M., & McKiernan, M.
(2017). Effectiveness of nursing education to prevent inpatient
suicide. The Journal of Continuing Education in Nursing, 48(9),
413–419. https://doi.org/10.3928/00220124-20170816-07

Mills, P., Watts, B., & Hemphill, R. (2014). Suicide attempts and
completions on medical-surgical and intensive care units. Journal
of Hospital Medicine, 9(3), 182–185. https://doi.org/10.1002/
jhm.2141

Petrik, M., Gutierrez, P., Berlin, J., & Saunders, S. (2015). Barriers
and facilitators of suicide risk assessment in emergency depart-
ments: A qualitative study of provider perspectives. General
Hospital Psychiatry, 37(6), 581–586. https://doi.org/10.1016/j.
genhosppsych.2015.06.018

Posner, K., Brown, G., Stanley, B., Brent, D., Yershova, K., Oquendo,
MA., Currier, G. W., Melvin, G. A., Greenhill, L., Shen, S., &
Mann, J. J. (2011). The Columbia–suicide severity rating scale:
Initial validity and internal consistency findings from three

ID:p0325ID:p0330ID:ti0060ID:p0335ID:ti0065ID:p0340ID:ti0070 ID:p0345ID:p0347ID:p0348ID:p0290ID:p0300ID:p0325ID:p0330ID:ti0060ID:p0335ID:ti0065ID:p0340ID:ti0070 ID:p0345ID:p0347ID:p0348ID:p0325ID:p0330ID:ti0060ID:p0335ID:ti0065ID:p0340ID:ti0070 ID:p0345ID:p0347ID:p0348

JDNP-D-20-00049 Page 129 03/07/21 5:32 AM

129Improving Suicidal Ideation Screening

multisite studies with adolescents and adults. American Journal
of Psychiatry, 168(12), 1266–1277. https://doi.org/10.1176/appi.
ajp.2011.10111704

Roaten, K., Johnson, C., Genzel, R., Khan, F., & North, C. (2018).
Development and implementation of a universal suicide risk
screening program in a safety-net hospital system. The Joint
Commission Journal on Quality and Patient Safety, 44(1), 4–11.
https://doi.org/10.1016/j.jcjq.2017.07.006

Stuck, A., Wilson, M., Chalmers, C., Lucas, J., Sarkin, A., Choi,
K., & Center, K. (2017). Health care usage and suicide
risk screening within 1 year of suicide death. The Journal of
Emergency Medicine, 53(6), 871–879. https://doi.org/10.1016/j.
jemermed.2017.06.033

U.S. Department of Health and Human Services (HHS) Office of
the Surgeon General and National Action Alliance for Suicide
Prevention. (2012). 2012 national strategy for suicide prevention:
goals and objectives for action.

Williams, S., Schmaltz, S., Castro, G., & Baker, D. (2018). Incidence
and method of suicide in hospitals in the United States. The

Joint Commission Journal on Quality and Patient Safety, 44(11),
643–650. https://doi.org/10.1016/j.jcjq.2018.08.002

World Health Organization. (2014). Preventing suicide—a
global imperative. https://apps.who.int/iris/bitstream/han-
dle/10665/131056/9789241564878_eng.pdf?sequence

Disclosure. The authors have no relevant financial inter-
est or affiliations with any commercial interests related
to the subjects discussed within this article.

Funding. The author(s) received no specific grant or
financial support for the research, authorship, and/or
publication of this article.

Correspondence regarding this article should be directed
to Anne Lindstrom, DNP, APRN, Rush University,
Northwestern Medicine Central DuPage Hospital, 25
N. Winfield Rd, Winfield, IL 60190. E-mail: anne.
[email protected]

ID:p0345ID:p0347ID:p0348ID:p0345ID:p0347ID:p0348ID:p0345ID:p0347ID:p0348

Copyright of Journal of Doctoral Nursing Practice is the property of Springer Publishing
Company, Inc. and its content may not be copied or emailed to multiple sites or posted to a
listserv without the copyright holder’s express written permission. However, users may print,
download, or email articles for individual use.

  • Improving Suicidal Ideation Screening and Suicide Prevention Strategies on Adult Nonbehavioral Health Units
    • Background
    • Objective
    • Methods
    • Results
    • Conclusion
    • Implications for Nursing
    • References

Screening Youth for Suicide Risk in
Medical Settings
Time to Ask Questions

Lisa M. Horowitz, PhD, MPH, Jeffrey A. Bridge, PhD, Maryland Pao, MD, Edwin D. Boudreaux, PhD

From the Intra
of Mental He
Pediatric Prac
dren’s Hospit
Columbus, Oh
ment of Psyc
(Boudreaux),
Massachusetts

Address co
Institute of Me
5362, Bethesda

0749-3797/
http://dx.do

S170 Am J

This paper focuses on the National Action Alliance for Suicide Prevention’s Research Prioritization
Task Force’s Aspirational Goal 2 (screening for suicide risk) as it pertains specifically to children,
adolescents, and young adults. Two assumptions are forwarded: (1) strategies for screening youth for
suicide risk need to be tailored developmentally; and (2) we must use instruments that were created
and tested specifically for suicide risk detection and developed specifically for youth. Recommen-
dations for shifting the current paradigm include universal suicide screening for youth in medical
settings with validated instruments.
(Am J Prev Med 2014;47(3S2):S170–S175) Published by Elsevier Inc. on behalf of American Journal of

Preventive Medicine

Introduction

Suicide remains a leading cause of death for youth
worldwide.1 Screening for risk of suicide and
suicidal behavior is an important and necessary

first step toward suicide prevention in young people.
Implementing effective screening programs involves
targeting high-risk populations in favorable settings.2

Medical settings have been designated as key venues to
screen for suicide risk and are therefore the focus of this
article.
The National Action Alliance for Suicide Prevention

(Action Alliance) developed 12 Aspirational Goals as a
way of structuring a suicide prevention research agenda
aimed at decreasing suicides in the U.S. by 40% over the
next decade. Aspirational Goal 2 pertains to screening for
suicide risk: “to determine the degree of suicide risk
among individuals in diverse populations and in diverse
settings through feasible and effective screening and
assessment approaches.”3

As an adjunct to a separate article in this supplement
that proposes a paradigm shift for suicide screening

mural Research Program (Horowitz, Pao), National Institute
alth, NIH, Bethesda, Maryland; Center for Innovation in
tice (Bridge), The Research Institute at Nationwide Chil-
al and The Ohio State University College of Medicine,
io; and the Department of Emergency Medicine, Depart-
hiatry, and Department of Quantitative Health Sciences
University of Massachusetts Medical School, Worcester,

rrespondence to: Lisa M. Horowitz, PhD, MPH, National
ntal Health, Clinical Research Center, Building 10, Room 6-
MD 20892. E-mail: [email protected]

$36.00
i.org/10.1016/j.amepre.2014.06.002

Prev Med 2014;47(3S2):S170–S175 Published by E

instrument development and research aligned with this
Aspirational Goal,4 this paper focuses on suicide screen-
ing as it pertains specifically to children, adolescents, and
young adults. The aims of this paper are to describe how
youth suicide prevention strategies need to be considered
independently of adult suicide prevention strategies,
underscore the need for universal screening with vali-
dated suicide screening instruments for youths in all
medical settings, and describe paradigm shifts that would
need to occur to achieve reductions in youth suicide/
suicidal behavior.

Assumptions of Screening for Suicide Risk
Assumption 1: Strategies for Screening Youth
for Suicide Risk Need to be Tailored
Developmentally
In the field of pediatrics, there is a well-known maxim:
“Children are not just small adults.” This tenet is
applicable to suicide prevention strategies. As with many
types of public health threats, a one-size-fits-all approach
will not be effective. Suicide risk changes at each
developmental stage of a young person’s life, increasing
with age throughout adolescence and early adulthood.5

Although death by suicide does occur in children under
12 years,6 suicide and suicidal behavior are rare prior to
puberty, in part because mood disorders, for example, are
less common in younger children. Risk of suicide
increases in the late teen years, coinciding with increased
risk of mood disorder onset. Nevertheless, half of all
mental illness onset begins in childhood, making it a
critical period of time to intervene.7

lsevier Inc. on behalf of American Journal of Preventive Medicine

Horowitz et al / Am J Prev Med 2014;47(3S2):S170–S175 S171

Developmental trajectories are the main character-
istics that set children apart from adults (Figure 1),
considering factors such as variable physical growth;
differences in cognition (ability to think abstractly);
language (ability to communicate needs); and social
competence (ability to make friends). These streams of
development are all happening at different times and
rates in children and adolescents. Converging upon these
trajectories are critical risk factors such as mental illness,
family history of mental illness, and history of suicidal
ideation or behavior. In addition, other psychiatric
comorbid conditions, such as substance abuse, may help
promote the transition from suicidal ideation to
behavior.
Some psychological traits can increase risk, such as

impulsive aggression in which a child may have a
tendency to react aggressively to frustrating situations or
have other maladaptive coping strategies. Environmental
factors such as psychosocial stressors, poverty, and “non-
intact” families may contribute to hopelessness. Many
youth have acute stressors that include interpersonal
conflict, loss, and problems with school.8 These factors
can all increase a young person’s risk for suicide. Ideally,
protective factors such as strong relationships with adults,
academic success, or religious beliefs can modify these risk
factors and reduce risk for suicidal behaviors—but even
these are not always sufficiently protective.
According to the most recent CDC data, 15.8% of all

high school students in the U.S. have seriously consid-
ered suicide.9 Some existential questioning is expected in
adolescence; however, when these thoughts become more
frequent or expand into plans to end one’s life, they

Figure 1. Developmental considerations in youth suicide
SI, suicidal ideation; SB, suicidal behavior; hx, history of; dx, diagnosis

September 2014

become clinically significant. Manifestations along the
continuum of suicide, from thoughts to behavior, are
important because they can all be predictive of death by
suicide. The hope is that screening and early detection
can have an impact and thwart the progression from
ideation to behavior.
Another important difference when evaluating and

treating youth as compared to adults is that most youth
are accompanied by parents or guardians when they visit
a medical setting. This has implications for the first
assumption noted above, as these adult caregivers can
provide useful collateral information that assists with
suicide risk assessment. In addition, having a parent/
guardian aware of elevated suicide risk in their child
affords them the opportunity to help with means
restriction and other important safeguards that can aid
in prevention of suicide. Currently, however, there is no
empirical evidence about whether including parental
questions in a suicide screening tool is more effective
than only screening the child, nor are there clinical
guidelines for how to proceed if parents and youth
disagree in their answers.

Assumption 2: We Must Use Instruments that
Were Developed and Tested Specifically for
Suicide Risk Detection and Developed
Specifically for Youth
This section emphasizes the importance of using instru-
ments that have been validated to detect the condition of
interest—suicide risk in youth. Sometimes, suicide risk
detection strategies are created for the general public and
are then utilized for children and adolescents, even if

Horowitz et al / Am J Prev Med 2014;47(3S2):S170–S175S172

age-specific validity has not been proven. Given all the
variables mentioned above, adult instruments may not
always be appropriate for screening youth for suicide risk.
The current paradigm is that screening occurs in a

non-standardized manner with patients who appear at
high risk to non–mental health clinicians, who may or
may not be knowledgeable about the risk factors. Screen-
ing items and suicide screening practices differ across
and within hospitals depending on knowledge and
training of staff, which varies greatly. The current
national practice for suicide screening in most hospitals
has not been assessed. For example, when the Joint
Commission issued Patient Safety Goal 15A in 2007
requiring all behavioral health patients to be screened for
suicide,10 nurses were asked to screen patients, but were
not given validated instruments for making such inqui-
ries. This would be akin to asking a nurse to guess a
patient’s body temperature without giving them a
thermometer.
Nurses reported a wide range of screening questions,

from indirect questions such as Are you safe? and How
will I know when you’re angry? to very specific questions
such as Have you had any thoughts of wanting to harm
yourself or others? (L. Horowitz, National Institute of
Mental Health, and J. Bridge, The Research Institute at
Nationwide Children’s Hospital and The Ohio State
University College of Medicine, personal communica-
tion, 2013). A national survey on what is being asked and
how to standardize the questions would be useful.
A proposed paradigm shift is to implement validated

tools and training staff to use clinical practice guidelines
developed for managing positive screens safely. Screening
would not be limited to patients with a known psychiatric
history; rather, it would occur universally in certain
settings. However, specific guidelines will need to be
established for setting up screening parameters for who
should administer the screening instrument, when dur-
ing the visit the patient should be screened, and, most
importantly, how positive screens will be managed.
If universal screening is to be implemented, the initial

screening tool will have to be brief, highly sensitive,
highly specific, and validated on the targeted population
for the condition under evaluation. Several measures
have been used to screen patients for suicide risk in
various medical settings: for specific use in the pediatric
emergency department (ED) population, the Risk of
Suicide Questionnaire (RSQ)11 and the Ask Suicide-
Screening Questions (ASQ);12 and in primary care (PC)
clinics, the Behavioral Health Screen (BHS),13 the
Columbia Suicide Severity Rating Scale (CSSRS),14 and
others.2,15 Validation studies should test for sensitivity,
specificity and negative and positive predictive values.
Prospective predictive validity of completed suicide and

suicidal behavior has yet to be established on the tools
mentioned above, and is greatly needed.
Because depression and suicide are frequently linked,

clinicians often use depression screens as suicide risk
detection instruments. Yet, depression screens are not
necessarily designed to be sensitive or specific enough
instruments for recognizing suicidal thoughts and behav-
iors, especially in medical patients.16

A widely used valid and reliable depression screening
instrument, the Patient Health Questionnaire (PHQ-9),17

provides an illustrative example. The ninth item on the
PHQ-9 asks the patient how often he or she is bothered
by the thought that you would be better off dead, or of
hurting yourself in some way and is widely used clinically
and in research studies to screen for suicide risk. This
item simultaneously and indistinguishably measures
both passive thoughts of death and suicide ideation, both
symptoms of depression. Because the question contains
an “or,” it has been found to be overly sensitive in that it
detects patients who have passive thoughts of death or
thoughts of hurting themselves.
In patients with serious medical illnesses, thoughts of

death are common and may be categorically unrelated to
suicide. Recent studies examining the use of Item 9 to
assess for suicide risk in medically ill patients suggest that
this question provides ambiguous, non-specific, and
difficult-to-interpret information that may overburden
already strained mental health resources.18 In addition,
inquiring about hurting and killing oneself, especially for
adolescents, may identify two different problems. In
settings where mental health resources are limited, asking
youth as directly as possible about suicide may be critical
for more accurate detection.

Recommendations
The public health import of utilizing universal screening
in medical settings as a way to identify youth at risk for
suicide and suicidal behavior is immense. Screening
positive on validated instruments may not only be
predictive of future suicidal behavior but also be a proxy
for other serious mental health concerns that require
further mental health attention and follow-up. For
example, it may not be feasible to screen for every
sociobehavioral risk factor in a busy ED setting.
However, once a young person screens positive for

suicide risk and receives a mental health evaluation, they
can be further assessed for serious mental illness, sub-
stance abuse, homicidal ideation, and history of physical
and sexual abuse. The proposed paradigm shift is that an
effective suicide screening instrument not only will detect
imminent risk but can also identify youth with significant
emotional distress warranting further mental health

www.ajpmonline.org

Horowitz et al / Am J Prev Med 2014;47(3S2):S170–S175 S173

attention, which if otherwise ignored can lead to serious
personal and societal consequences (e.g., school absen-
teeism, antisocial behavior, school dropout, and
increased use of healthcare services).
Any setting in which a healthcare provider delivers

medical care, such as PC clinics, EDs, inpatient medical
units, and school-based clinics, may be ideal venues to
identify youth at elevated risk. More than 80% of youths
visit their PC doctor each year, making the PC clinic well
situated to identify young people at risk. Wintersteen19

showed that there was a 4-fold increase in detection of
suicidal ideation by pediatricians when screening tools
were used in outpatient clinics (base rate¼0.8%, screen-
ing tools¼3.6%). The study, however, emphasized that
these data translated into one additional youth per week
requiring further mental health follow-up, which did not
overwhelm the pediatric care clinics.
Similar results have been found in pediatric emergency

care settings. For those who are not connected to a PC
clinic, estimated to be about 1.5 million youth, the ED is
their sole contact with the healthcare system,20 creating
not only an opportunity but a responsibility to screen for
suicide risk. A recent Canadian study revealed that 80%
of youth who died by suicide visited a PC provider, an
ED, or had an inpatient medical hospitalization within
3 months prior to their death.21 The obvious clinical
challenge is that these individuals do not walk into their
doctor’s office and say, “I want to kill myself”; rather, they
frequently present with somatic complaints (e.g., head-
aches, stomachaches), and may not talk about their
suicidal thoughts unless asked directly.
Pediatric ED studies show that screening for suicide

risk can reveal previously undetected thoughts of suicide
in youth presenting with medical/surgical chief com-
plaints.18 Moreover, screening was found to be acceptable
to clinicians, parents, and youth and was found to be
non-disruptive to ED workflow. Several studies reveal
that young patients embrace the notion of being screened
for suicide risk in medical settings.22,a

Larkin and Beautrais23 describe the ED as an impor-
tant nexus for suicide-related endophenotypes (e.g.,
alcohol and substance abuse, pain syndromes, medical
comorbidities). These high-risk groups include young
people who may be disenfranchised, may have dropped
out of school, are not employed, or are in the foster care
system. These young people are often isolated and do not
have a connection with someone who can recognize that
they need help. An ED visit can provide this opportunity.
A major barrier to screening for suicide risk is the

concern about how to safely manage patients who screen
positive. What does a positive screen on a validated

aContact corresponding author for additional references.

September 2014

instrument that was created to detect suicide risk actually
mean? Screening positive means a patient has a symptom
that requires further evaluation. To use a medical
analogy, this is akin to a pediatric patient who is found
to have high blood pressure during an ED visit. They are
not immediately administered an anti-hypertensive med-
ication; rather, a further assessment ensues to determine
what is causing the high blood pressure and what may
happen to the patient if the hypertension persists.
Screening positive on a suicide risk screen is similar;

something is amiss and further evaluation is necessary. A
patient who screens positive is in need of a psychiatric
evaluation by a trained mental health professional who
can examine related symptoms, judge risk of self-harm,
and, if necessary, guide the primary physician in appro-
priate disposition decisions and link the patient with
mental health treatment if needed. It does not necessarily
mean a constant observer is necessary or that the child
needs to be hospitalized on an inpatient psychiatric unit,
although these are potential outcomes.
Not inquiring about suicide risk would be akin to not

measuring blood pressure because the system did not
want to find out the child had hypertension. In addition,
taking into account developmental needs, a child-sized
blood pressure cuff would be needed to measure blood
pressure properly. The patient has the symptom whether
or not a healthcare provider asks about it. But if we do
not ask, chances are the patient will not tell us, and they
may not get the help they need.
Important research pathways will include validating

screening instruments with targeted populations in the
specific healthcare settings in which they will be used.
This effort would require conducting universal screening
and developing clinical practice guidelines tailored for
youth to manage positive screens safely and effectively in
each setting, with long-term follow-up for youth who
screen positive and negative to determine the validity and
full impact of screening.
Critical stakeholders in the screening process will need

to be identified, such as hospital administrators, whose
commitment to implementing effective screening pro-
grams and providing mental health resources for positive
screens will be essential. Importantly, we will need nurse
and physician champions to help with changing clinical
practice to include screening and reduce stigma associ-
ated with patients who screen positive. We will need to
educate families about what positive screens imply, the
need for mental health follow-up services for the patient,
and guidance sessions for the parents.
Screening for suicide risk can become part of core

performance improvement measures for hospitals and
clinics by adding screening to hospital scorecards and
Healthcare Effectiveness Data and Information Set

Horowitz et al / Am J Prev Med 2014;47(3S2):S170–S175S174

(HEDIS) measures. Currently, more than 90% of Amer-
ican health insurance plans use HEDIS as a tool to
measure performance on critical dimensions of health-
care delivery.24 The current metrics include “adolescent
well-visits” or “anti-depressant medication manage-
ment,” and “cervical cancer screening in adolescent
females,” but suicide screening is notably absent
Barriers to universal screening include strapped men-

tal health resources and limited patient care time. Other
roadblocks include myths of iatrogenic risk. Many,
including healthcare providers, still believe that we may
be putting ideas of suicide into a youth’s mind if we ask
them directly about suicide; however, there have been
several studies that refute this myth.25,a Another barrier is
the lack of mental health resources available in medical
settings to manage positive screens, especially providers
trained in child/adolescent mental health. Linkage rates
to mental health providers have been low with people
who have screened positive, partly due to few resources,
but also because the stigma of having mental health
concerns still plagues patients and prevents them from
initiating conversations about their mental suffering and
seeking help.
Opponents of universal screening may argue that

suicide is a low–base rate event, especially in young
people, so we cannot develop instruments that accurately
predict suicide. Although it is true that we do not
currently have tools that predict which youths will kill
themselves, we do have tools that can detect suicidal
ideation, which should not be minimized in young
people. Nock et al.26 found that approximately one third
of youth with suicidal ideation go on to develop a suicide
plan in adolescence, and about 60% of those with a plan
will attempt suicide. The hope is that intervening early,
during ideation, will lead to prevention.

Conclusions
Youth suicide prevention strategies will need to be
designed with developmental considerations in mind. It
is time for all youth in medical settings to be screened for
suicide risk, just as they are routinely screened for
hypertension, fever, and falls risk. We cannot rely solely
on depression screens or non-validated instruments to
identify young people at risk for suicide. We as research-
ers need to create and test developmentally sound tools
for healthcare providers to use.
Demonstration projects in pediatric medical settings

with these instruments will highlight strengths and
uncover future challenges to overcome. Importantly,
screening can only take us so far. We must turn our
research efforts toward developing more effective inter-
ventions. Lastly, we must hold ourselves, as clinicians and

researchers, accountable for lowering the youth suicide
rate within the next decade. Every healthcare provider
can have an impact.

Publication of this article was supported by the Centers for
Disease Control and Prevention, the National Institutes of
Health Office of Behavioral and Social Sciences, and the
National Institutes of Health Office of Disease Prevention.
This support was provided as part of the National Institute of
Mental Health-staffed Research Prioritization Task Force of
the National Action Alliance for Suicide Prevention.
Dr. Boudreaux receives consulting payment and owns stock

options in Polaris Health Directions, a private company that
creates and markets mental health assessment and intervention
software. This paper does not endorse any specific programs or
products that Dr. Boudreaux has developed.
No financial disclosures were reported by the other authors

of this paper.

References
1. Bridge JA, Goldstein TR, Brent DA. Adolescent suicide and suicidal

behavior. J Child Psychol Psychiatry 2006;47(3–4):372–94.
2. Peña JB, Caine ED. Screening as an approach for adolescent suicide

prevention. Suicide Life Threat Behav 2006;36(6):614–36.
3. Pearson J, Claassen C, Booth CL. Introduction to the Suicide

Prevention Research Prioritization Task Force special supplement:
the topic experts. Am J Prev Med 2014;47(3S2):S102–S105.

4. Boudreaux ED, Horowitz L. Suicide risk screening and assessment:
designing instruments with dissemination in mind. Am J PrevMed 2014.

5. Gould MS, Greenberg T, Velting DM, Shaffer D. Youth suicide risk
and preventive interventions: a review of the past 10 years. J Am Acad
Child Adolesc Psychiatry 2003;42(4):386–405.

6. Tishler CL, Reiss NS, Rhodes AR. Suicidal behavior in children
younger than twelve: a diagnostic challenge for emergency department
personnel. Acad Emerg Med 2007;14(9):810–8.

7. Kessler RC, Berglund P, Demler O, et al. Lifetime prevalence and age-
of-onset distributions of DSM-IV disorders in the National Comor-
bidity Survey Replication. Arch Gen Psychiatry 2005;62(6):593.

8. Karch DL, Logan J, McDaniel DD, Floyd CF, Vagi KJ. Precipitating
circumstances of suicide among youth aged 10–17 years by sex: data
from the national violent death reporting system, 16 states, 2005–2008.
J Adolesc Health 2013;53(1S):S51–S53.

9. CDC National Center for Injury Prevention and Control. Web-based
Injury Statistics Query and Reporting System (WISQARS). cdc.gov/
injury/wisqars/index.html.

10. Joint Commission on Accreditation of Healthcare Organizations
Patient Suicide: complying with National Patient Safety Goal 15A. Jt
Comm Perspect Patient Safety 2008;8:7–8, 11.

11. Horowitz LM, Wang PS, Koocher GP, et al. Detecting suicide risk in a
pediatric emergency department: development of a brief screening tool.
Pediatrics 2001;107(5):1133–7.

12. Horowitz LM, Bridge JA, Teach SJ, et al. Ask Suicide-Screening
Questions (ASQ): a brief instrument for the pediatric emergency
department. Arch Pediatr Adolesc Med 2012;166(12):1170–6.

13. Diamond G, Levy S, Bevans KB, et al. Development, validation, and
utility of Internet-based, behavioral health screen for adolescents.
Pediatrics 2010;126(1):163–70.

14. Posner K, Brown GK, Stanley B, et al. The Columbia-Suicide Severity
Rating Scale: initial validity and internal consistency findings from

www.ajpmonline.org

Horowitz et al / Am J Prev Med 2014;47(3S2):S170–S175 S175

three multisite studies with adolescents and adults. Am J Psychiatry
2011;168(12):1266–77.

15. Goldston D. Assessment of suicidal behaviors and risk among children
and adolescents. Bethesda MD: National Institute of Mental Health,
2000. Contract No. 263-MD-909995.

16. Recklitis CJ, Lockwood RA, Rothwell MA, Diller LR. Suicidal ideation
and attempts in adult survivors of childhood cancer. J Clin Oncol
2006;24(24):3852–7.

17. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief
depression severity measure. J Gen Intern Med 2001;16(9):606–13.

18. Walker J, Hansen CH, Hodges L, et al. Screening for suicidality in
cancer patients using item 9 of the nine-item patient health question-
naire; does the item score predict who requires further assessment?
Gen Hosp Psychiatry 2010;32(2):218–20.

19. Wintersteen MB. Standardized screening for suicidal adolescents in
primary care. Pediatrics 2010;125(5):938–44.

20. Wilson KM, Klein JD. Adolescents who use the emergency department as
their usual source of care. Arch Pediatr Adolesc Med 2000;154(4):361–5.

September 2014

21. Rhodes AE, Khan S, Boyle MH, et al. Sex differences in suicides among
children and youth: the potential impact of help-seeking behaviour.
Can J Psychiatry 2013;58(5):274–82.

22. Ballard ED, Bosk A, Snyder D, et al. Patients’ opinions about suicide
screening in a pediatric emergency department. Pediatr Emerg Care
2012;28(1):34–8.

23. Larkin GL, Beautrais AL. Emergency departments are underutilized
sites for suicide prevention. Crisis 2010;31(1):1–6.

24. Health Indicators Warehouse. Healthcare Effectiveness Data and
Information Set (HEDIS). healthindicators.gov/Resources/DataSour
ces/HEDIS_56/Profile.

25. Gould MS, Marrocco FA, Kleinman M, et al. Evaluating iatrogenic risk
of youth suicide screening programs: a randomized controlled trial.
JAMA 2005;293(13):1635–43.

26. Nock MK, Green JG, Hwang I, et al. Prevalence, correlates, and
treatment of lifetime suicidal behavior among adolescents results from
the National Comorbidity Survey Replication Adolescent Supplement.
JAMA Psychiatry 2013;70(3):300–10.

  • Screening Youth for Suicide Risk in Medical Settings
    • Introduction
    • Assumptions of Screening for Suicide Risk
      • Assumption 1: Strategies for Screening Youth for Suicide Risk Need to be Tailored Developmentally
      • Assumption 2: We Must Use Instruments that Were Developed and Tested Specifically for Suicide Risk Detection and…
    • Recommendations
      • Conclusions
    • References

DNP 618 Article Matrix and Analysis

Student Name:

PICO Question: In adolescent patients seen in primary care setting (P) what is the effect of utilization of Ask Suicide Screening Question toolkit (I) compared to those not screened (C) within 3 months

Search process: Database was searched utilizing Northern Kentucky University Library and the National Library of Medicine. The articles were peer reviewed and published within the last five years. The articles were narrowed down based on the criterial listed.

Search terms: Ask Suicide Screening Questions toolkit, suicide screening and prevention, suicide in patients, how to improve the mental health of patients or students

Databases: MEDLINE, CINHAL

Total number of articles obtained from search results: N = 1767

Number of articles initially excluded based on abstract reading: N= 30 (out of first 100 relevant)

Number of articles reviewed: N= 8

Number of articles excluded based on criteria: N= 1611

Inclusion criteria: Must be in English, full text only, published in 2017 or later, peer-reviewed articles, include adolescent, depression and suicide as the subject, be discoverable in a reputable database

Exclusion Criteria: Article published prior to 2017, language other than English, no full text link, obtained from a non-reputable source.

Number of systematic reviews or meta-analyses used in Matrix- N = 1

Repeat this table – one for each article you are review. DO NOT double space in the table

Author, year; Credentials Article #1


Men, V. Y., Emery, C. R., & Yip, P. S. F. (2021). All the authors have affiliation with Department of Social Work and Social Administration, The University of Hong Kong. Published in Psycho‐ Oncology Journal

Article Focus/Title

Characteristics of cancer patients who died by suicide

Research Design/Intervention (describe intervention)

Cross-sectional study : No intervention

Level of Evidence and model used to grade evidence

Level IV: Johns Hopkins (Dearholt et al., 2017).

Evaluation Tool (CASP or others- identify tool used)

JBI Critical Appraisal Checklist for Analytical Cross-Sectional Studies, 8/8.

Sample/# of subjects, how recruited, power analysis?

Using data from the past records, researchers were able to count 14,446 suicides that occurred between 2003 and 2017. Based on the reported medical histories, the cases were classified according to whether or not they had cancer. 1,461 (10.1%) of the 14,446 people who committed suicide had a prior diagnosis of cancer. Univariate analyses and overall and subgroup multiple logistic regressions were performed to compare characteristics between the two groups.

Data Collection Procedure

Past records were utilized. Using data from the Coroner’s Court, researchers in Hong Kong were able to count 14,446 suicides that occurred between 2003 and 2017. Based on the reported medical histories, the cases were classified according to whether or not they had cancer. Socioeconomic characteristics and in-depth accounts of the suicide incident were among the data culled from the reports.

Instruments and

Reliability/validity

of instruments

The coroner’s Court’s report was used as the data collection instrument. Independent t‐tests or chi‐square tests were used to examine differences in study variables of interest between the cancer and noncancerous groups.

Data Analysis- id statistics, LOM, findings

Comparisons of features between the two groups were analyzed using univariate analyses, as well as overall and subgroup multiple logistic regressions. The threshold of statistical significance was 0.05

Results

The outcome indicated that the patients diagnosed with cancer are highly likelihood to occur to those in a committed relationship as compared to those in single status, and a significantly higher likelihood of being retired. Among people with cancer who committed suicide, most of the people decided to commit suicide by jumping in high heights where 54.3% was recorded. On the other hand, there are those that used the hanging method and the data recorded was 27.6% and finally the use of toxicities approach like carbon monoxide poisoning (6.6%). Oncology patients are likely to employ very lethal violent tactics, greater probability to discuss their suicide intentions prior executing on them, less prone to suffer severe psychiatric illnesses, and much more likely to have one maybe more medical issues besides cancer.

Discussion/

Significance of findings

We found that suicides caused by cancer in Hong Kong differ from those caused by other diseases, and that these differences increase with the patient’s age. Patients with cancer who are elderly, more likely to commit suicide with violence, more likely to seek care for suicidal ideation, and less prone to have cognitive and emotional problems. More research on the causes of cancer patient suicide is needed to improve the quality of life for people who are afflicted with the disease.

Reliability and Validity of study, limitations

Since there is little room for data misclassification, the dataset has high validity. Comparisons of character between cancer and non-cancer suicide cases were used. Study data was extracted from the reports, comprising cancer status, sociodemographic information (age, gender, marital status, employment status, living conditions); suicide descriptions (place and method of the suicide); medical history (physical or psychiatric problems); family engagement (living alone or with family); and whether the deceased had communicated with others about suicidal intent. The methods of suicide were then classified into types of act such as hanging, cutting, crashes, burning, suffocation, and nonviolent means. Suicide rates in Hong Kong have fluctuated over the past 15 years, and these oscillations may be accurately depicted thanks to the large sample size provided by the 15-year timeframe.

Study limitation include missing information in the Coroner’s Court’s reports due to reasons such family’s inability to provide them, also lack of detailed information on the medical history of the cancer patients. Such as the cancer type, stage of cancer, and the date of the diagnosis for the patients. Other limitations include the authors inability to perform additional analyses to explore suicide‐decision pathways or temporal and casual relationships between different risk factors for suicide.

Helpful/Reliable

Compared to other articles

Superior in utility, precision, and reliability, with results that hold up under statistical scrutiny. Simple, digestible, and comprehensive findings summary. This is the only article I am aware of that examines cancer-related suicide rates explicitly in comparison to those of other diseases prevalent in the Asian community.

Author, year; Credentials Article #2


Aguinaldo, L. D.(MD), Sullivant, S.(MD), Lanzillo, E. C.(MD), Ross, A.(MD), He, J. P.(MD), Bradley-Ewing, A., Bridge, J. A. (MD), Horowitz, L. M.(MD), & Wharff, E. A (MD) (2021)

Published in General Hospital Psychiatry

Article Focus/Title

Validation of the ask suicide-screening questions (ASQ) with youth in outpatient specialty and primary care clinics

Research Design/Intervention (describe intervention)

Cross-sectional study: Nonintervention

Level of Evidence and model used to grade evidence

Level IV: Johns Hopkins (Dearholt et al., 2017).

Evaluation Tool (CASP or others- identify tool used)

JBI Critical Appraisal Checklist for Analytical Cross-Sectional Studies, 8/8

Sample/# of subjects, how recruited, power analysis?

The sample consisted of 515 young people (ages 10-21) who spoke English and used it to communicate with clinic staff in primary and specialized care outpatient clinics. In total, 918 patients were selected during the research; 827 were found to be eligible for participation, and 525 were informed and their permission sourced to take part. 7 patients that were completely engaged in the study after getting their permission participated in the outpatient specialty or primary care appointment but did not complete the study interview for personal reasons were also not included in the analyses. Three subjects were not included due to RA mistake in performing the research survey or RED Cap data collection. The remaining 515 patients were also included.

Data Collection Procedure

Suicidal Ideation Questionnaire, a self-report instrument measuring severity of suicidal ideation in adolescents, and Demographic/Clinical Variables Questionnaire, a survey containing questions on demographic and clinical information was administered, after written or verbal informed consents were received from the patients 18 or older and consents were obtained from the parents/guardians of those younger than 18.

Instruments and

Reliability/validity

of instruments

The SIQ/SIQ Jr. and the ASQ were used, the SIQ/SIQ Jr. has demonstrated high reliability (SIQ: r = 0.97; SIQ-JR: r =0.94), validity, and predictive ability using the standard criterion of the Suicidal Behavior Interview, a semi-structured clinical interview that assesses current and past suicidal behavior

Data Analysis- id statistics, LOM, findings

Associations between ASQ-detected suicide risks (i.e., heightened risk present or absent) and demographic, clinical, and other variables (i.e., age, sex, race/ethnicity, and insurance status) were analyzed using multivariate binary logistic regression modeling. ace status, and site) entered into the model simultaneously. Adjusted odds ratios were the exponentiated values of multivariate logistic regression coefficients. The statistical significance level was P =0.05.

Results

Out of 335 participants, 45 (13.4%) were found to have suicidal thoughts after taking the ASQ. Acute positive screens were found in only four individuals (1.2 percent of the total sample) who tested positive on the ASQ. In outpatient specialty clinics, the ASQ had a sensitivity of 100% (95% CI: 80.5%-100.0%), a specificity of 91.2% (95% CI: 87.5-94.1%), and a negative predictive value of 100% (95% CI: 98.7-100.0). When comparing those at risk of suicide to those who were not, a positive screen on the ASQ was 11.4 times more likely to be noticed in the former group (95% CI, 8.0-16.2). There was substantial concordance between the SIQ/SIQ Jr. and the 4-item ASQ screen (AUC = 95.6%) when assessing suicide risk.

Discussion/

Significance of findings

When compared to the gold standard SIQ/SIQ Jr., the ASQ appears to be a reliable instrument for use in identifying patients at increased suicide risk in pediatric outpatient specialty or primary care clinics. The results of this research corroborated the previous conclusion that the ASQ has adequate content validity. To aid physicians in determining which patients need further diagnostic workup, the ASQ was developed as a primary screening tool.

Reliability and Validity of study, limitations

There is little reason to doubt the accuracy of the numbers. This is owing to the fact that primary care and specialized pediatric settings are using the ASQ to identify children at risk of suicide. Multivariate binary logistic regression modeling was used to examine the associations of suicide risk as detected by the ASQ. Data preparation and data analyses were performed. Informed consent was received. Study was approved by scientific and institution review boards at the National Institute of Mental health and at the study sites.

One of the study’s limitations is that our convenience sample may not be representative of the general youthful population in regular outpatient settings. It’s possible that patients’ suicide risk screening results changed because they took part in the study (i.e., before or after their clinic appointment in outpatient specialty care). The constant probing about suicide ideation may have worn people down.

Helpful/Reliable

Compared to other articles

It was helpful because it showed this is an effective tool for identification of youths at risk for suicide when screened in an outpatient primary care setting. However, the authors should have expanded their scope to urban tertiary care facilities.

Author, year; Credentials Article #3

Farrer, L. M. (Registered Psychologist), Gulliver, A. (Research Fellow), Katruss, N. (Research Assistant), Bennett, K. (credential not identified), Bennett, A. (credential not identified), Ali, K. (PhD research student), & Griffiths, K. M. (Emeritus Professor) (2020).

Published in the British Journal of Guidance & Counselling

Article Focus/Title

Development of the Uni Virtual Clinic: an online program for improving the mental health of university students

Research Design/Intervention (describe intervention)

This was a participatory research design .Students who were traditionally underserved or socially isolated at their universities benefited from the adoption of evidence-based online interventions that helped lessen the strain on university resources.

Level of Evidence and model used to grade evidence

Level I: Johns Hopkins (Dearholt et al., 2017).

Evaluation Tool (CASP or others- identify tool used)

Tool needs to be JBI or CASP

Sample/# of subjects, how recruited, power analysis?

The qualitative methods primarily included four focus groups conducted with university students (n = 5, 5, 4, 5). Twenty people took part in three prototype testing sessions, a student leadership group workshop, and an evaluation of the pilot UVC. the project team also conducted several face-to-face discussions with university stakeholders to assess the information and service needs of different sub-groups of students

Data Collection Procedure

Systemic reviews were conducted to examine current evidence for technology which supported University students’ mental health. Peer to peer support intervention was examined, stakeholders’ perspective and feedbacks were conducted, using both qualitative and quantitative research methods. The quantitative methods comprised of two surveys, one with university staff involved in teaching and supervising students, and the other with the students.

Instruments and

Reliability/validity

of instruments

Surveys, face to face discussions and focus groups were used. Focus groups are reliable since they allow the researcher to collect detailed information and get the true customer opinion on product, beliefs, and perceptions. Surveys producing similar consistent results shows their reliability.

Data Analysis- id statistics, LOM, findings

All data obtained was documented in percentages. attitudes towards participation were positive, with each of the 19 positive items receiving average ratings above 0, indicating agreement (M = 0.96, SD = 0.24)

Results

The majority of respondents (M = 0.96, SD = 0.24) agreed with the statement that attitudes toward participation were generally favorable, with all 19 positive items having average values over zero, indicating agreement. “Each of the 13 negative items received average ratings below 0, indicating disagreement M = −1.50, SD = 0.46; ranging from −1.05 for “confused” to −1.70 for “useless”. “Adjectives that received the highest average scores indicating strongest agreement were valued M = 1.35, SD = 0.49, safe M = 1.30, SD = 0.57, helpful M = 1.20, SD = 0.41, involved M = 1.20, SD = 0.41, respected M = 1.15, SD = 0.37, and glad M = 1.15, SD = 0.67.”

Discussion/

Significance of findings

For the UVC to be developed in accordance with the needs of its intended audience, participatory design was essential. By lowering the rate of occurrence of mental disorders in high-risk youth and thereby preventing the negative attributes that accompany these conditions, UVC can help establish universities that are mentally healthy places to learn and live.

Reliability and Validity of study, limitations

By including students in the design process, the team was better able to account for students’ evolving requirements, the evolving nature of universities, and the ever-evolving nature of web design and development techniques for e-mental health service. The research had certain limitations, such as the fact that involving end users in the design and development process takes a lot of time and energy, which in turn increased the total amount of time needed for content production and website development. It was difficult to achieve harmony in the final program due to frequent conflicts over UVC content and functioning.

Helpful/Reliable

Compared to other articles

Helpful and reliable because it showed Uni Virtual Clinic (UVC) has the potential to assist universities to increase their ability to support students’ mental health. All of the study’s outcomes were easy to understand and interpret. .

Author, year; Credentials Article #4

LeCloux, M. A., Weimer, M., Culp, S. L., Bjorkgren, K., Service, S., & Campo, J. V. (2020). The authors credentials are not identified. Published by Elsevier Inc.”

Article Focus/Title

The Feasibility and Impact of a Suicide Risk Screening Program in Rural Adult Primary Care: A Pilot Test of the Ask Suicide-Screening Questions Toolkit

Research Design/Intervention (describe intervention)

Quasi experiment research design. Patients presenting for routine and sick visits were asked to participate in electronic suicide risk screening using the Ask Suicide-Screening Questions tool, patients that were positive for suicidal were further assessed with the Ask Suicide-Screening Questions Brief Suicide Safety Assessment.

Level of Evidence and model used to grade evidence

Level III: Johns Hopkins (Dearholt et al., 2017)

Evaluation Tool (CASP or others- identify tool used)

CASP for Qualitative checklist, score 9/10

Sample/# of subjects, how recruited, power analysis?

The baseline sample included 274 patients. Patients consented electronically. A 2 phased design was used to collected date from patients who presented for routine or sick visit in a rural primary care clinic. Patients who were too medically ill to participate were excluded. Electronic medical records for patients 18 or older were collected, this data was logged and deindentifeid by the clinician into spread sheet

Data Collection Procedure

Patients were directed to an electronic survey including the ASQ screening questions and a 5-item Screening Opinions Questionnaire. Results from the ASQ were uploaded and viewed by the PCP. For patients who screened positive for suicide risk, the PCP administered a brief suicide risk assessment using the BSSA as a guide and then provided disposition planning as needed. Risk level was then recorded in the EMR. Data was collected from the EMR by the researcher.

Instruments and

Reliability/validity

of instruments

Questionnaires in the form of Ask Suicide-Screening Questions (ASQ) were used. The ASQ screening tool has demonstrated high sensitivity (83–98%) 24,27,28 and good specificity 28 in samples of youths aged 10 to 21 years. Preliminary data also indicate that the ASQ has high sensitivity and specificity among adult inpatients.

Data Analysis- id statistics, LOM, findings

SPSS 26 was used to upload all data and analyze them. To assess the three indicators of viability, descriptive statistics were calculated. The client testing and suicide risk assessment process rates were compared between the two phases using a chi-square methodology with Fisher’s exact test.

Results

Screening for suicide risk was more likely to be reported throughout the intervention period (61%) compared to the baseline period 5%; X2 = 200.6, P, 0.00, In addition, 6.2% of cases during the intervention phase exhibited suicide risk, while this was only the case in 0.7% of cases during the baseline phase X2 = 12.58, P, 0.001

Discussion/

Significance of findings

This pilot research finding suggest the ASQ and BSSA are useful tools for identifying rural adults at risk of suicide, receiving primary care. Suicide risk screening programs to patients in a rural primary care are amenable and deserve further study.

Reliability and Validity of study, limitations

All study procedures received full institutional review board approval by the supporting university before the start of the study. Research validity and reliability are ensured by the use of the ASQ Toolkit, which has been shown to promote risk identification rates in adult patients receiving primary care. The study was limited by the fact that it was a pilot project, and the participants were all from the same racial background and were seen by the same primary care physician in a remote rural setting. In addition, because this was a 2-stage study, there were also some changes between the two phases of the intervention.

Helpful/Reliable

Compared to other articles

The study is reliable since these findings are consistent with findings from other empirical studies that suicide risk screening is feasible and acceptable to most patients, including those in inpatient adult medical units, pediatric emergency rooms, and urban adult primary care

Author, year; Credentials Article #5

Lindstrom, A. C. (DNP, APRN), & Earle, M. (DNP, RN) (2021).

Published in the Journal of Doctoral Nursing Practice

Article Focus/Title

Improving Suicidal Ideation Screening and Suicide Prevention Strategies on Adult nonbehavioral Health Units

Research Design/Intervention (describe intervention)

Randomized control trial.

Targeted intervention that includes nursing education, easier electronic medical record (EMR) workflow for documentation of suicide screening, was needed to improve identification and provide safety of patients at risk for suicidal.

Level of Evidence and model used to grade evidence

“Level I: Johns Hopkins” ( Dearholt et al., 2017)

Evaluation Tool (CASP or others- identify tool used)

This has to be either JBI or CASP

Sample/# of subjects, how recruited, power analysis?

There were a total of 6,229 admissions to the hospital’s adult non-behavioral health wards. A total of 6,210 of these individuals underwent screening for suicidal thoughts. Patients who visited the hospital were randomly selected.

Data Collection Procedure

The findings of focus groups with key stakeholders, such as bedside nurses, and a literature study on the early implementation of suicide screening informed the creation of interventions.

Instruments and

Reliability/validity

of instruments

Focus group discussion, nursing protocol development, data on compliance monitoring, nursing education, a pre- and post-nursing assessment, electronic medical record change where an additional answer added to the suicide screening question and making the question required documentation. The project was approved by the instructional review board, the problem was clearly defined and measured, possible solutions were analyzed and implemented and a control plan was developed to ensure sustainability. Interventions were developed based on a review of the literature discussing initial implementation of suicide screening and focus group findings from bedside nurses, and other key stakeholders.

Data Analysis- id statistics, LOM, findings

Audits of patient charts were performed to check for proper order entry and screening. According to audit results, all 63 clients had a support health provider available, 82.5 % of patients took documented safety precautions, and 76% of patients received assistant provider keep the information in the notebook within day 1 of admission.

Results

A total of 91.5% (N = 344) of all qualified nurses have finished the training program, with 89.2% of the pretest questions being answered correctly, it is clear that the participants had a solid foundation of knowledge to build upon. There was a 2.9% increase in accuracy between the pre-and post-tests, with 92.1% of questions being answered correctly on the latter.

Improvement was noted in the screening compliance which improved by 20.6%, bedside safety companion use was 100%, suicide precautions, documentation 82.5%, and a behavioral health social work note documented 76% of the time

Discussion/

Significance of findings

The nurses received in-depth training that covered every aspect of patient care, from the initial screening until the moment of release. When nurses are provided clear polices, set protocol and education, universal suicide ideation screening can be improved, and in so doing increase safety for patients.

Reliability and Validity of study, limitations

Several causal factors influencing compliance were found in this study’s medical center through chart checks and focus groups with nurses, thus the study was conducted in an environment where it was possible to maintain compliance with all regulations. Due to the fact that this research was conducted in just one location, the findings cannot be extrapolated beyond that location. Furthermore, the research was launched two months before the hospital’s experience with the COVID-19 pandemic.

Helpful/Reliable

Compared to other articles

No literature was found on how to improve screening compliance once a method had been developed, making this study both useful and credible. Other papers explore the introduction of suicide screening within the institution.

Summary section:

This literature study supported the idea that inquiring about someone’s mental health can help prevent and reduce suicide rates. The articles reviewed in this matrix include two level IV evidence articles, and three-level I evidence article.

The two cross sectional studies were valid according to the JBI checklist for cross sectional studies. The participatory, quasi, and randomised control trial were valid according to CASP, BSSA and C- SSRS, respectively. Questionnaires were used in three of the articles and past records were used in two of the articles. Four of the results of the study were reliable and helpful except one where the author didn’t carry out a detailed research on his/her topic.

In summary, all articles reviewed concluded that universal suicidal screening is feasible and helps improve patient safety. The article has changed my thinking about my research topic.

Summary will have to be revised based on changes made.

Dearholt needs to be added to references

I believe the 3rd aruticle is a qualitative article?

BSSA and C-SSRS are not tool that can be utilized. We have to pick from those listed below

Links to critical appraisal tools to evaluate research quality:

Joanna Briggs Institute (joannabriggs.org) 

Joanna Briggs Institute Links to an external site.

 

Links to an external site.


CASP checklistsLinks to an external site.


Mixed Method appraisal checklist McGill: Links to an external site.


Cincinnati Children’s Hospital Medical Center Legend tools- very helpfulLinks to an external site.


Johns Hopkins EBP Models and Tools 

References

Aguinaldo, L.D. et al. (2021) “Validation of the ask suicide-screening questions (ASQ) with youth in outpatient specialty and primary care clinics,” General Hospital Psychiatry, 68, pp. 52–58. Available at: https://doi.org/10.1016/j.genhosppsych.2020.11.006.

Farrer, L.M. et al. (2020) “Development of the UNI Virtual Clinic: An online programme for improving the Mental Health of University Students,” British Journal of Guidance &amp; Counselling, 48(3), pp. 333–346. Available at: https://doi.org/10.1080/03069885.2020.1729341.

LeCloux, M.A. et al. (2020) “The feasibility and impact of a suicide risk screening program in rural Adult Primary Care: A pilot test of the Ask Suicide-Screening Questions Toolkit,” Psychosomatics, 61(6), pp. 698–706. Available at: https://doi.org/10.1016/j.psym.2020.05.002.

Lindstrom, A.C. and Earle, M. (2021) “Improving suicidal ideation screening and suicide prevention strategies on Adult Nonbehavioral Health Units,” Journal of Doctoral Nursing Practice, 14(2), pp. 122–129. Available at: https://doi.org/10.1891/jdnp-d-20-00049.

Men, V., Emery, C. and Yip, P. (2021) “Characteristics of cancer patients who died by suicide: A quantitative study of 15‐year coronial records,” Psycho-Oncology, 30(7), pp. 1051–1058. Available at: https://doi.org/10.1002/pon.5634.

Fall 10.2020

Writerbay.net

Do you need academic writing help? Our quality writers are here 24/7, every day of the year, ready to support you! Instantly chat with a customer support representative in the chat on the bottom right corner, send us a WhatsApp message or click either of the buttons below to submit your paper instructions to the writing team.


Order a Similar Paper Order a Different Paper
Writerbay.net