Shared Task on Scene Segmentation (STSS)

Schedule

• 13:30 – 14:00: Opening
• 14:00 – 14:25: Detecting Scenes in Fiction Using the Embedding Delta Signal (Felix Schneider, Björn Barz, Joachim Denzler)
• 14:25 – 14:50: “LTUHH@STSS: Applying Coreference to Literary Scene Segmentation” (Hans Ole Hatzel, Chris Biemann)
• 14:50 – 15:15: Coffee Break
• 15:15 – 15:40: “Scene Segmentation Using Temporal, Spatial and Entity Feature Vectors” (Florian Barth, Tillmann Dönicke)
• 15:40 – 16:05: “Twin BERT Contextualized Sentence Embedding Space Learning and Gradient-Boosted Decision Tree Ensembles for Scene Segmentation in German Literature“ (Sebastian Gombert)
• 16:05 – 16:30: “Breaking the Narrative: Scene Segmentation through Sequential Sentence Classification” (Murathan Kurfali, Mats Wirén)
• 16:30 - ?: Discussion

Task Description

Our shared task has the goal of developing methods to detect scenes in narrative texts automatically. A scene can be understood as a segment of a text where the story time and the discourse time are more or less equal, the narration focuses on one action and space and character constellations stay the same. Scenes can be found predominantly in narrative texts like novels or biographies, which can be understood as a sequence of segments where some of the segments are scenes and others are not. Scene segmentation is of great interest for the high-level analysis of longer texts, for example the reconstruction of plot, but also for many areas of NLP that deal with longer narrative texts, since even modern methods struggle with processing text longer than a couple of sentences or paragraphs. The shared task thus also provides a testbed to explore methods to handle long texts.

Datasets

The data set used in the shared task consists of German-language dime novels annotated with scene boundaries. The annotation guidelines are already available here. A partial data set of 15 novels has already been published in the context of an EACL paper (preprint). As the annotation process continues, test data annotated according to the same guidelines will be available soon.

We structure our shared task into two tracks distinguished by the evaluation dataset: Track 1 focuses on dime novels, which are ‘simple texts’ without strong variation. In track 2, contemporary high literature is used as a test set, thus allowing to evaluate transfer across different narrative text types.

While novels are substantially longer than ‘typical NLP texts’, the number of annotated novels in our dataset is not very large. Participants are thus encouraged to incorporate additional knowledge sources and/or data sets.

Evaluation Metrics

Evaluating segmentation with variable length is not straightforward. Attempts to allow for some leeway (i.e., penalise near misses not as harshly) introduce parameters that are difficult to optimise. The ranking metric will therefore be the exact F1 score over all boundaries. For informative reasons, we will also publish various other metrics and visualisations that allow for a deeper understanding of the performances of the submitted systems.

We will also offer an interim evaluation, i.e., allow participants to test their systems on unknown test data before the final submission deadline.

Important Dates

This is the timeline for this challenge. Further information will be announced in the future. All dates are given in the AoE (anywhere on earth) timezone.

• April 15, 2021: Trial Data will be provided ✅
• May 15, 2021: Training Data will be provided ✅
• June 7, 2021: Registration Deadline ✅
• June 7, 2021: Start of interim evaluation ✅
• June 15, 2021 June 20, 2021: End of interim evaluation ✅
• June 30, 2021: Start of final evaluation ✅
• July 7, 2021July 14, 2021: End of final evaluation ✅
• July 15, 2021July 22, 2021: Paper submission due ✅
• August 10, 2021August 17, 2021: Camera ready due ✅
• September 6-9, 2021: KONVENS conference

Contact

If you have any questions regarding the challenge, please write an email to stss2021@informatik.uni-wuerzburg.de.

Organisers

• Albin Zehe, University of Würzburg
• Leonard Konle, University of Würzburg
• Lea Dümpelmann, University of Heidelberg
• Evelyn Gius, TU Darmstadt
• Svenja Guhr, TU Darmstadt
• Andreas Hotho, University of Würzburg
• Fotis Jannidis, University of Würzburg
• Lucas Kaufmann, University of Würzburg
• Markus Krug, University of Würzburg
• Frank Puppe, University of Würzburg
• Nils Reiter, University of Cologne
• Annekea Schreiber, TU Darmstadt