Yoko Nishihara, Shuang Wu, Ryosuke Yamanishi: A method to select comic frames for supporting mimetic onomatopoeia study, Proc. of International Conference on Human-Computer Interaction 2022, pp. 348-353, 2022年6月

Japanese comics include onomatopoeias to represent sounds and situations while other types of comics usually use words semantically representing the source of sound and the event itself. The onomatopoeias are rarely translated. The untranslation of onomatopoeias may prevent non-native readers of Japanese language from enjoying Japanese comics. There are two types of onomatopoeias: sound-based and mimetic onomatopoeias. The mimetic onomatopoeias reflect situations and atmosphere in comic frames. However, to best of our knowledge, there might not be any method to translate those words to other languages with the language features. So, we change our way of thinking from translation to supporting study of mimetic onomatopoeias. This paper proposes a method to select comic frames for supporting mimetic onomatopoeia study. As a comic frame with a mimetic onomatopoeia is given, the proposed method evaluates similarities between speech texts in the comic frame and other comic frames in a database. We assume that, if a speech text around a mimetic onomatopoeia in a comic frame is similar to that in the other comic frame, the semantics and usage of those comic frames are similar to each other. It is expected that the proposed approach selects comic frames with high similarities as hints to understand the semantic and usage of the mimetic onomatopoeias. In this paper, the application vision and the limitation of the proposed approach are detailed and discussed.