Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent
Cross Language Agent Team - ByteDance Research
Abstract. In this paper, we present Cross Language Agent - Simultaneous Interpretation, CLASI, a high-quality and human-like Simultaneous Speech Translation (SiST) System. Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency. To address the challenge of translating in-domain terminologies, CLASI employs a multi-modal retrieving module to obtain relevant information to augment the translation. Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information. Experimental results show that CLASI outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate CLASI with a better human evaluation metric, valid information proportion (VIP), which measures the amount of information that can be successfully conveyed to the listeners. In the real-world scenarios, where the speeches are often disfluent, informal, and unclear, CLASI achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese translation directions, respectively. In contrast, state-of-the-art commercial or open-source systems only achieve 35.4% and 41.6%. On the extremely hard dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70% VIP.
Figure 1: Performance evaluation. CLASI significantly outperforms the leading commercial and open-source systems using a more reliable VIP metric, achieving human interpreter parity.
Gallery
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Framework
Figure 2: Overall framework of CLASI. The diagram illustrates the flow of operations for speech translation. In Step 1, CLASI processes the current input audio data. Optionally, the retriever is activated to obtain the relevant information from External Knowledge. In this demonstration, using the translation pair “伊辛模型: Ising model” from the External Knowledge leads to the precise translation of the provided speech. In Step 3, the CLASI reads from its memory to load the previous historical context. Lastly (Steps 4 and 5), the agent may engage CoT to output both transcription and translation as well as update its memory. Then the agent returns to Step 1 to process the speech of the next round.
Figure 3: Architecture of CLASI agent. At round r, our model takes the current audio stream, the previous memory (r − 1), and the retrieved knowledge (if have) as the input. CLASI outputs the response based on the given instruction and then updates the memory. Note that our model also outputs the cut-off timestamp of the last semantic chunk. For the given example, the phrase “就在” is not treated as a full semantic chunk, the cut-off timestamp is right before this phrase.
Comparison Demonstrations
Chinese English
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
English Chinese
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag
Replay
Play
00:00 / 00:00
00:00
Fullscreen
Cssfullscreen
1x
  • 2x
  • 1.5x
  • 1x
  • 0.75x
  • 0.5x
Click and hold to drag