Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding
SemKey is a novel multi-stage framework for EEG-to-text decoding that addresses three key limitations: semantic bias, signal neglect, and the BLEU Trap. It enforces signal-grounded generation through four decoupled semantic objectives (sentiment, topic, length, and surprisal) and redesigns neural encoder-LLM interaction by injecting semantic prompts as Queries with EEG embeddings as Key-Value pairs. The approach achieves state-of-the-art performance on robust evaluation protocols including N-way Retrieval Accuracy and Fréchet Distance, effectively eliminating hallucinations on noise inputs.
arXiv:2603.03312v1 Announce Type: cross
Abstract: Decoding natural language from non-invasive EEG signals is a promising yet challenging task. However, current state-of-the-art models remain constrained by three fundamental limitations: Semantic Bias (mode collapse into generic templates), Signal Neglect (hallucination based on linguistic priors rather than neural inputs), and the BLEU Trap, where evaluation metrics are artificially inflated by high-frequency stopwords, masking a lack of true semantic fidelity. To address these challenges, we propose SemKey, a novel multi-stage framework that enforces signal-grounded generation through four decoupled semantic objectives: sentiment, topic, length, and surprisal. We redesign the interaction between the neural encoder and the Large Language Model (LLM) by injecting semantic prompts as Queries and EEG embeddings as Key-Value pairs, strictly forcing the model to attend to neural inputs. Furthermore, we move beyond standard translation metrics by adopting N-way Retrieval Accuracy and Fr\'echet Distance to rigorously assess diversity and alignment. Extensive experiments demonstrate that our approach effectively eliminates hallucinations on noise inputs and achieves SOTA performance on these robust protocols. Code will be released upon acceptance at https://github.com/xmed-lab/SemKey.