WIP

site/_config.yml

@@ -1,5 +1,5 @@
 # _config.yml
-
-title: "Jekyll Actions Demo"
+theme: jekyll-theme-cayman
+title: "GermEval2024 - GerMS"
 
 

site/announcement.md

@@ -0,0 +1,6 @@
+* Text to use for announcing the shared task by email or in social media.
+* may contain a copy of [overview](overview.md) text
+
+## Text
+
+

site/index.md

@@ -1,6 +1,72 @@
----
----
+* Text giving an overview of the shared task
+* links to the details of [subtask 1](subtask1.md) and [subtask 2](subtask2.md)
 
-Welcome to My Home Page
+## GermEval-2024 Task - Sexism Detection in German Online News Fo
+ra
 
+This shared task is about the detection of sexism/misogyny in comments
+posted in (mostly) German language to the comment section of an Austrian
+online newspaper. 
 
+The data was originally collected for the development of a classifier 
+that supports the human moderators in detecting potentially sexist 
+comments or identify comment fora with a high rate of sexist comments.
+
+The texts identified as being sexist are often especially challenging
+for automatic classification because they often refer to some implied 
+context which is not available or are formulated in a subtle way, avoiding
+strong or outright offensive language. 
+
+Texts have been annotated by several human annotators, with a large portion
+of the corpus being annotated by at least two annotators out of 7 who 
+are forum moderators. 
+
+The main aim of annotating the presence and strength of sexism/misogyny in 
+the corpus was to identify comments which make it less welcoming to 
+women to participate in the conversation. 
+The full annotator guidelines with examples are available in an 
+[English translation of the German original](guidelines.pdf)
+
+Since the sexism/misogyny present in this corpus is often present in 
+a subtle form that avoids outright offensiveness or curse words, 
+there are many texts where annotators have different opinios on whether
+the text should be regarded as sexist, or which degree of sexism should
+be assigned to it. 
+
+This shared task therefore also provides an opportunity to learn about
+how to deal with diverging opinions among annotators and how to train 
+models on such a corpus which potentially can also inform about how 
+diverging the opinions on a new text might be. 
+
+## Subtasks
+
+The shared task is divided into two subtasks:
+
+* [Subtask 1](subtask1.md): predict a binary label indicating the presence or absence of sexism in different ways, based on the original grading of the texts by several annotators; also predict the majority grading assigned by annotators. 
+* [Subtask 2](subtask2.md): predict binary soft labels, based on the different opinions of annotators about the text; predict the distribution of the original gradings by annotators
+
+## Closed and open tracks
+
+Each of the [subtask 1](subtask1.md) and [subtask 2](subtask2.md) competitions 
+are organized into two different tracks:
+
+* [Closed Track](closed-track.md): in this track, models can only be trained with the provided training set. Models are limited as to what kind of data for pretraining is allowed. Only the closed track counts towards the competition of the shared task and a closed track submission is required for the submission of a paper. See the linked document for details.
+* [Open Track](open-track.md): in this track, anything goes really: you can use language models, use your own training data (but you have to share it with the community) or use other interesting approaches. The open track does NOT count towards the competition ranking but has been added to allow for the exploration of interesting strategies which may be hard to reproduce. 
+
+
+
+## Timeline
+
+* **Development phase**: April 14 - May 17, 2024
+* **Testing phase**: May 18 - June 12, 2024
+* **Evaluation phase**: June 13 - June 25, 2024
+* **Paper submission due**: July 1, 2024
+* **Camera ready due**: July 20, 2024
+* **Shared Task @KONVENS**: 9 September, 2024
+
+## Organizers
+The task is organized by the **Austrian Research Institute for Artificial Intelligence (OFAI)**. The organizing team are:
+
+* [Brigitte Krenn](https://www.ofai.at/~brigitte.krenn/) (brigitte.krenn (AT) ofai.at)
+* [Johann Petrak](https://johann-petrak.github.io/) (johann.petrak (AT) ofai.at)
+* [Stephanie Gross](https://www.ofai.at/~stephanie.gross/) (stephanie.gross (AT) ofai.at)

site/subtask1.md

@@ -0,0 +1 @@
+

site/subtask2.md

@@ -0,0 +1,10 @@
+# How to participate
+
+Please submit your results as a .tsv file. It needs to contain the following columns: 
+* *ID* : the ID of the post
+* *0_rate* : percentage of annotators rating this post as not sexist
+* *1_rate* : percentage of annotators rating this post as 1 on a scale from 0-4 with 4 being extremely sexist
+* *2_rate* : percentage of annotators rating this post as 2 on a scale from 0-4 with 4 being extremely sexist
+* *3_rate* : percentage of annotators rating this post as 3 on a scale from 0-4 with 4 being extremely sexist
+* *4_rate* : percentage of annotators rating this post as 4 on a scale from 0-4 with 4 being extremely sexist
+* *soft_label* : rating this post as sexist on a scale from 0 to 1.

site/terms.md

@@ -0,0 +1,18 @@
+# Terms and Conditions
+
+**Participation in the competition**: Any interested person may freely participate in the competition. By participating in the competition, you agree to the terms and conditions in their entirety, without amendment or provision. By participating in the competition, you consent to the public release of your scores and submissions at the GermEval-2024 workshop and in the associated proceedings. Participation is understood as any direct or indirect contributions to this site or the shared task organizers, such as, but not limited to: results of automatic scoring programs; manual, qualitative and quantitative assessments of the data submitted; task and systems papers submitted.
+
+**Individual and Team Participation**: Participants may create teams, but participants may not be part of more than one team. Teams and individual participants must create exactly one account to participate in the Codabench competition. Team composition may not be changed once the Test Phase starts. Your system may be named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organizers.
+
+**Scoring of submissions**: Submissions may be evaluated with automatic and manual quantitative judgements, qualitative judgements, and any other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers. Organizers are under no obligation to release scores. Official scores may be withheld if organizers judge the submission incomplete, erroneous, deceptive, or violating the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science. If multiple submission files are uploaded during the Test Phase, the last submission file per group will be understood as the team's or participant's definitive submission and ranked as such in the task description paper.
+
+**Data usage**: The provided data should be used responsibly and ethically. Do not attempt to misuse it in any way, including, but not limited to, reconstructing test sets, any none-scientific use of the data, or any other unconscionable usage of the data. You may not redistribute the task data except in the manner prescribed by its licence.
+
+**Submission of systems description papers**: Participants having made at least one submission during the Test Phase will be invited to submit a paper describing their system. We strongly encourage a link to the code of systems being described will be made available to organizers or the public at large. We also encourage you to upload any systems and models to an open-source repository such as the HuggingFace Hub.
+
+**Specific conditions for closed tasks**: Participants agree to use no (i) existing models trained on additional data labelled for sexism or misogyny or (ii) additional data labelled for sexism or misogyny.
+
+**Specific conditions for open tasks**: If participants use additional models or embeddings trained on data labelled for sexism or misogyny, these models and data need to be open source and provided by the participants upon request. Participating in the closed task is a precondition to participate in the open task. However, participating in the open task is no precondition for participating in the closed task.
+
+
+**Acknowledgements**: This shared task was created by OFAI with funding from the FFG project EKIP.