Longformer-es-mental-large

Model Description

Longformer-es-mental-large is a Spanish domain-adapted language model designed for the analysis of mental health–related content in long user-generated texts.

It is based on the Longformer architecture, which extends the standard Transformer attention mechanism to efficiently process long sequences. The model supports input sequences of up to 4096 tokens, enabling it to capture long-range dependencies and temporal patterns that are particularly relevant in mental health monitoring and early risk detection settings.

This model was obtained through domain-adaptive pre-training (DAP) on a large corpus of mental health–related texts translated into Spanish from Reddit communities focused on psychological support and mental health discussions. The adaptation process allows the model to better capture emotional expressions, self-disclosure patterns, and discourse structures characteristic of mental health narratives in Spanish.

Longformer-es-mental-large is released as a foundational model and does not include task-specific fine-tuning.

This is the model card of a 🤗 transformers model that has been pushed on the Hub.

• Developed by: ELiRF group, VRAIN (Valencian Research Institute for Artificial Intelligence), Universitat Politècnica de València
• Model type: Transformer-based masked language model (Longformer)
• Language(s) (NLP): Spanish
• License: Same as base model (PlanTL-GOB-ES models)
• Finetuned from model: ELiRF/RoBERTa-es-mental-large

Uses

This model is intended for research purposes in the mental health NLP domain.

Direct Use

The model can be used directly as a feature extractor or encoder for Spanish mental health–related texts, particularly when long input sequences are required.

Downstream Use

The model is primarily intended to be fine-tuned for downstream tasks such as:

• Mental disorder detection
• Early risk detection
• User-level and context-level classification
• Analysis of long social media timelines related to psychological well-being

It has been evaluated in early detection benchmarks when fine-tuned using task-specific datasets and methodologies.

Out-of-Scope Use

• Use on languages other than Spanish
• High-stakes decision-making affecting individuals’ health or safety
• Real-time intervention systems without human supervision

Bias, Risks, and Limitations

• The training data originates from social media platforms, which may introduce demographic, cultural, and linguistic biases.
• The data was automatically translated into Spanish, which may introduce translation artifacts or subtle semantic drift.
• Mental health language is highly contextual and subjective; predictions may be unstable when evidence is limited.
• The model does not provide explanations or clinical interpretations of its outputs.

How to Get Started with the Model

Use the code below to get started with the model.

  from transformers import AutoTokenizer, AutoModel

  tokenizer = AutoTokenizer.from_pretrained("ELiRF/Longformer-es-mental-large")
  model = AutoModel.from_pretrained("ELiRF/Longformer-es-mental-large")

  inputs = tokenizer(
      "Ejemplo de texto relacionado con salud mental.",
      return_tensors="pt",
      truncation=True,
      max_length=4096
  )

  outputs = model(**inputs)

Training Details

Training Data

The model was domain-adapted using a merged corpus composed of:

• Reddit SuicideWatch and Mental Health Collection (SWMH)
• Reddit Mental Health Narratives (RMHN)

All texts were automatically translated into Spanish using neural machine translation. The final dataset contains approximately 1.9 million posts from multiple mental health–related communities.

Training Procedure

The model was trained using domain-adaptive pre-training (DAP) with a masked language modeling objective. Pre-training was performed for 20 epochs using multiple GPUs, following the same procedure applied to the other foundational models described in the paper “Improving Mental Health Screening and Early Risk Detection in Spanish”.

No task-specific fine-tuning is included in this checkpoint.

Preprocessing [optional]

• Automatic translation to Spanish
• Text normalization and tokenization using the base model tokenizer

Training Hyperparameters

• Training regime: fp 16 mixed precision

Speeds, Sizes, Times [optional]

• Model size: ~435M parameters
• Training duration: 4 days
• Checkpoint size: 1.7GB

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was evaluated after fine-tuning on Spanish mental health benchmarks (e.g., MentalRisk shared tasks).

Factors

• Disorder type
• Amount of available user context
• Early vs full-context settings

Metrics

• Macro F1-score
• ERDE (Early Risk Detection Error)
• Latency True Positive (LTP)

Results

When fine-tuned on Spanish mental health benchmarks, Longformer-es-mental shows competitive performance and improves upon the state of the art in both full-context (user-level) and early detection mental health tasks.

Summary

Longformer-es-mental is a Spanish domain-adapted long-context language model for mental health text analysis. It is based on the Longformer architecture and supports input sequences of up to 4096 tokens, enabling the modeling of long user message histories. The model shows strong performance on Spanish mental health detection and early risk detection tasks when fine-tuned on domain-specific datasets.

Technical Specifications [optional]

Model Architecture and Objective

• Longformer architecture
• Masked Language Modeling objective

Compute Infrastructure

Hardware

• Multiple NVIDIA A40 GPUs

Citation

This model is part of ongoing research currently under review. The final version of the paper will be linked once it is published.

Model Card Authors

ELiRF research group (VRAIN, Universitat Politècnica de València)