HMS-BERT: Hybrid Multi-Task Self-Training for Multilingual and Multi-Label Cyberbullying Detection
| Authors | Zixin Feng et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.12920 |
| Download | |
| Categories | cs.CL, stat.ML |
Abstract
Cyberbullying on social media is inherently multilingual and multi-faceted, where abusive behaviors often overlap across multiple categories. Existing methods are commonly limited by monolingual assumptions or single-task formulations, which restrict their effectiveness in realistic multilingual and multi-label scenarios. In this paper, we propose HMS-BERT, a hybrid multi-task self-training framework for multilingual and multi-label cyberbullying detection. Built upon a pretrained multilingual BERT backbone, HMS-BERT integrates contextual representations with handcrafted linguistic features and jointly optimizes a fine-grained multi-label abuse classification task and a three-class main classification task. To address labeled data scarcity in low-resource languages, an iterative self-training strategy with confidence-based pseudo-labeling is introduced to facilitate cross-lingual knowledge transfer. Experiments on four public datasets demonstrate that HMS-BERT achieves strong performance, attaining a macro F1-score of up to 0.9847 on the multi-label task and an accuracy of 0.6775 on the main classification task. Ablation studies further verify the effectiveness of the proposed components.
Engineering Breakdown
Plain English
This paper addresses cyberbullying detection across multiple languages and abuse categories simultaneously—a realistic problem since abusive content on social media often mixes multiple violation types and appears in non-English languages. The authors built HMS-BERT, a system that combines a pretrained multilingual BERT model with handcrafted linguistic features and trains on two related tasks at once: fine-grained multi-label abuse classification and a coarser three-class main classification. The key innovation is using self-training to bootstrap learning from unlabeled data in low-resource languages, directly tackling the scarcity of labeled training examples in non-English contexts. While the abstract doesn't provide final metrics, the approach is designed to outperform existing monolingual or single-task methods by handling the inherent overlap and multilingual nature of real-world cyberbullying.
Core Technical Contribution
HMS-BERT's core novelty is the hybrid multi-task self-training framework that jointly optimizes two complementary classification objectives—a fine-grained multi-label task that captures overlapping abuse types and a three-class main classification task that provides coarser supervision. Rather than treating multilingual cyberbullying as a single classification problem, the authors decompose it into related subtasks and leverage the shared multilingual BERT backbone to transfer knowledge across languages and abuse categories. The integration of handcrafted linguistic features alongside learned contextual representations is intentional: it combines domain-specific signal (linguistic patterns of abuse) with neural representations, allowing the model to exploit both expert knowledge and data-driven learning. The iterative self-training loop specifically addresses low-resource language scenarios by using model predictions on unlabeled data to expand the training set, bootstrapping performance without requiring expensive annotation.
How It Works
The pipeline starts with text input in any supported language, which is tokenized and passed through a pretrained multilingual BERT encoder to produce contextual token representations. These learned representations are combined with handcrafted linguistic features (likely including n-grams, word embeddings, syntactic patterns, or dictionary-based abuse markers) to form a richer feature vector for each input. This combined representation branches into two parallel classification heads: one for multi-label prediction (outputting probabilities for multiple abuse categories that can co-occur) and one for three-class main classification (likely severity or broad category). During training, both tasks are optimized jointly—losses from both heads are backpropagated to update the shared BERT backbone and feature combination weights, creating mutual regularization. For self-training, the model makes predictions on unlabeled data (especially from low-resource languages), filters high-confidence predictions using a confidence threshold, and adds them to the training set as pseudo-labels, then retrains iteratively. This cycle continues until convergence or diminishing returns, effectively leveraging unlabeled data to improve generalization in data-scarce languages.
Production Impact
A production system using HMS-BERT would move beyond single-language or single-abuse-type detection—you'd deploy a unified model that flags multiple overlapping violations simultaneously across your entire user base regardless of language. This solves a major operational challenge: platforms today often run separate monolingual pipelines and struggle when abuse spans categories (harassment + hate speech, for instance), requiring manual review or multiple model calls. The multi-task design reduces inference latency compared to chaining separate models, since you get both fine-grained labels and main classification in a single forward pass; for platforms processing millions of posts daily, this is meaningful. However, the self-training component requires careful production engineering: you need a confident way to filter pseudo-labels to avoid training data poisoning (low-quality pseudo-labels that degrade performance), and you must monitor for language-specific drift where unlabeled data distributions shift over time. The handcrafted features also mean maintaining language-specific linguistic knowledge bases, which increases operational complexity but enables better interpretability and domain adaptation compared to pure end-to-end models.
Limitations and When Not to Use This
The paper's limitation is most acute in truly low-resource scenarios where even unlabeled data may be scarce or expensive to obtain; self-training's effectiveness depends on having sufficient confidently-labeled pseudo-data, which may not exist for languages with minimal social media presence. The reliance on handcrafted linguistic features requires domain expertise and manual feature engineering per language—this doesn't scale easily to dozens of languages and creates bottlenecks where linguistic knowledge must be periodically updated. The abstract doesn't specify how overlapping labels are resolved (are there hard constraints on label co-occurrence?) or how class imbalance is handled across abuse types, both critical for real deployments where some violation types are much rarer than others. Additionally, the approach assumes that multilingual BERT's cross-lingual transfer is strong enough for the target languages; this may fail for code-mixed text, language varieties not well-represented in BERT's pretraining, or entirely new languages, creating potential blind spots. Follow-up work would need to address robustness to adversarial reformulations (creative variations of abuse designed to evade detection) and establish whether the multi-task design actually prevents catastrophic forgetting when low-resource languages are added to the training mix.
Research Context
This work builds on the multilingual BERT pretraining paradigm (Devlin et al.'s original BERT plus mBERT variants) and multi-task learning principles established in NLP, but applies them specifically to the underexplored area of multilingual abusive content detection. Prior cyberbullying detection work typically tackled English-only datasets or single-abuse-type formulations (hate speech XOR toxicity XOR harassment), missing the real-world messiness that HMS-BERT addresses. The self-training component echoes semi-supervised learning traditions in NLP, particularly work on pseudo-labeling for low-resource tasks, but the novelty is combining it with multi-task learning in the multilingual abuse domain. The paper likely benchmarks against datasets like the HateXplain corpus, AbuseLand, or language-specific cyberbullying corpora, and opens research directions into more sophisticated pseudo-label confidence estimation, language-specific linguistic feature extraction, and the interplay between multilingual transfer and multi-task regularization in adversarial detection scenarios.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::