Developing and validating a parser-based suicidality detection model in text-based mental health services

Background: Advances in text-mining can potentially aid online text-based mental health services in detecting suicidality. However, false positive remains a challenge. Methods: Data of a free 24/7 online text-based counseling service in Hong Kong were used to develop a novel parser-based algorithm (PBSD) to detect suicidal ideation while minimizing false alarms. Sessions containing keywords related to suicidality were extracted (N = 1267). PBSD first applies a sentence parser to work out the grammatical structure of each sentence, including subject, object, dependent and modifier. Then a set of syntax rules were applied to judge if a flagged sentence is a true or false positive. Half of the sessions were randomly selected to train PBSD, the remaining were used as the test set. A standard keywords matching model was adopted as the baseline comparison. Accuracy and recall were reported to demonstrate models' performance. Results: Of the 1267 sessions, 585 (46.2 %) were false alarms. The false alarms were categorized into four types: negation-induced false alarms (NIFA; 14 %), subject-induced false alarms (SIFA; 19 %), tense-induced false alarms (TIFA; 30 %), and other types of false alarms (OTFA; 37 %). PBSD significantly outperforms the baseline keywords matching model on accuracy (0.68 vs 0.53, 28.3 %). It successfully amended 36.8 % (105/297) lexicon matching-caused false alarms. The reduction on recall was marginal (1 vs 0.96, 4 %). Conclusions: The proposed model significantly improves the use of lexicon-based method by reducing false alarms and improving the accuracy of suicidality detection. It can potentially reduce unnecessary panic and distraction caused by false alarms among frontline service-providers.

Automated summary

This study developed a novel parser-based algorithm (PBSD) to detect suicidal ideation in online text-based counseling. The algorithm utilizes sentence parsing and syntax rules to minimize false alarms. Results showed that PBSD significantly outperforms the baseline model in accuracy and successfully reduces false alarms caused by lexicon matching.