Skip to content

Commit 4b23c79

Browse files
johann-petrakjohann-petrak
committed
WIP
1 parent 151b4c6 commit 4b23c79

File tree

5 files changed

+19
-28
lines changed

5 files changed

+19
-28
lines changed

‎site/announcement.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,13 @@ GermEval2024 Shared Task: GerMS-Detect -- Sexism Detection in German Online News
22

33
CALL FOR PARTICIPATION
44

5-
9 September 2024 at KONVENS 2024, Vienna, Austria
5+
10 September 2024 at KONVENS 2024, Vienna, Austria
66

77
[https://ofai.github.io/GermEval2024-GerMS/](https://ofai.github.io/GermEval2024-GerMS/)
88

99
---- Task description ----
1010

11-
This shared task is about the detection of sexism/misogyny in comments posted in (mostly) German language to the comment section of an Austrian online newspaper. The data was originally collected for the development of a classifier that supports the human moderators in detecting potentially sexist comments or identify comment fora with a high rate of sexist comments. The texts identified as being sexist are often especially challenging for automatic classification because they often refer to some implied context which is not available or are formulated in a subtle way, avoiding strong or outright offensive language. The main aim of annotating the presence and strength of sexism/misogyny in the corpus was to identify comments which make it less welcoming to women to participate in the conversation. Since the sexism/misogyny present in this corpus is often present in a subtle form that avoids outright offensiveness or curse words, there are many texts where annotators have different opinios on whether the text should be regarded as sexist, or which degree of sexism should be assigned to it. This shared task therefore also provides an opportunity to learn about how to deal with diverging opinions among annotators and how to train models on such a corpus which potentially can also inform about how diverging the opinions on a new text might be.
11+
This shared task is about the detection of sexism/misogyny in comments posted in (mostly) German language to the comment section of an Austrian online newspaper. The data was originally collected for the development of a classifier that supports human moderators in detecting potentially sexist comments or identify comment fora with a high rate of sexist comments. The texts identified as being sexist are often especially challenging for automatic classification because they often refer to some implied context which is not available or are formulated in a subtle way, avoiding strong or outright offensive language. The main aim of annotating the presence and strength of sexism/misogyny in the corpus was to identify comments which make it less welcoming to women to participate in the conversation. Since the sexism/misogyny present in this corpus is often present in a subtle form that avoids outright offensiveness or curse words, there are many texts where annotators have different opinios on whether the text should be regarded as sexist, or which degree of sexism should be assigned to it. This shared task therefore also provides an opportunity to learn about how to deal with diverging opinions among annotators and how to train models on such a corpus which potentially can also inform about how diverging the opinions on a new text might be.
1212

1313
The shared task is divided into two subtasks:
1414

‎site/open-track.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ There is an _Open Track_ competition for each of the two subtasks. Please note t
55

66
* In the open tracks, participants are encouraged to use whatever approach they prefer
77
* Additional labeld data or models or embeddings trained on labelled data are allowed.
8-
* HOWEVER Additional labelled data, embeddings or models must be publically available as open source or with a creative-commons license
8+
* HOWEVER, additional labelled data, embeddings or models must be publically available as open source or with a creative-commons license
99
* IMPORTANT: Participants submitting in open tracks are only invited to submit a paper for the Shared Task at KONVENS 2024 describing their system, if they also made a submission in a closed track during the Competition Phase.
10-
* Due to reproducibility issues, e.g. when including results from commercial or closed-source models we do not accept papers which solely present approaches for the open tracks.
10+
* Due to reproducibility issues, e.g. when including results from commercial or closed-source models, we do not accept papers which solely present approaches for the open tracks.
1111
* We do look forward however to find out how the results in the open tracks will compare to the closed track results.

‎site/subtask1.md

+7-9
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,13 @@ To submit your predictions to the competition:
8585
* enter the name of your team which has been registered for the competition
8686
* give a name to your method
8787
* confirm that you have checked that you are indeed submitting to the correct competition for the subtask and track desired
88+
89+
**Submission errors and warnings**
90+
91+
* Always make sure a phase is selected before trying to upload your submission.
92+
* A submission is successful, if it has the submission status 'finished'. 'Failed' submissions can be investigated for error sources by clicking at '?' next to 'failed' and looking at LOGS > scoring logs > stderr.
93+
* If you experience any issue such as a submission file stuck with a "scoring" status, please cancel the submission and try again. In case the problem persists you can contact us using the Forum.
94+
* Following a successful submission, you need to refresh the submission page in order to see your score and your result on the leaderboard.
8895

8996
## Phases
9097

@@ -99,13 +106,4 @@ over all classes.
99106

100107
The final `score` which is used for ranking the submissions is calculated as the unweighted average over all 5 scores.
101108

102-
103-
## Submission errors and warnings
104-
105-
Always make sure a phase is selected before trying to upload your submission.
106-
107-
A submission is successful, if it has the submission status 'finished'. 'Failed' submissions can be investigated for error sources by clicking at '?' next to 'failed' and looking at LOGS > scoring logs > stderr.
108-
109-
If you experience any issue such as a submission file stuck with a "scoring" status, please cancel the submission and try again. In case the problem persists you can contact us using the Forum.
110-
111109
Following a successful submission, you need to refresh the web page in order to see your score and your result on the leaderboard.

‎site/subtask2.md

+5-12
Original file line numberDiff line numberDiff line change
@@ -85,8 +85,12 @@ To submit your predictions to the competition:
8585
* give a name to your method
8686
* confirm that you have checked that you are indeed submitting to the correct competition for the subtask and track desired
8787

88+
**Submission errors and warnings**
8889

89-
**note**: do we provide example submissions?
90+
* Always make sure a phase is selected before trying to upload your submission.
91+
* A submission is successful, if it has the submission status 'finished'. 'Failed' submissions can be investigated for error sources by clicking at '?' next to 'failed' and looking at LOGS > scoring logs > stderr.
92+
* If you experience any issue such as a submission file stuck with a "scoring" status, please cancel the submission and try again. In case the problem persists you can contact us using the Forum.
93+
* Following a successful submission, you need to refresh the submission page in order to see your score and your result on the leaderboard.
9094

9195
## Phases
9296

@@ -101,14 +105,3 @@ distance metric which is between 0 and 1. It is the square root of the Jensen-Sh
101105

102106
The overall score which is used for ranking the submissions is calculated as the unweighted average between the two JS-distances.
103107

104-
105-
## Submission errors and warnings
106-
107-
Always make sure a phase is selected before trying to upload your submission.
108-
109-
A submission is successful, if it has the submission status 'finished'. 'Failed' submissions can be investigated for error sources by clicking at '?' next to 'failed' and looking at LOGS > scoring logs > stderr.
110-
111-
If you experience any issue such as a submission file stuck with a "scoring" status, please cancel the submission and try again. In case the problem persists you can contact us using the Forum.
112-
113-
Following a successful submission, you need to refresh the submission page in order to see your score and your result on the leaderboard.
114-

‎site/terms.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,14 @@
22

33
**Participation in the competition**: Any interested person may participate in the competition. By your participation, you agree to the terms and conditions in their entirety, without amendment or provision. By participating in the competition, you consent to the public release of your scores and submissions at the GermEval-2024 workshop and in the associated proceedings. Participation is understood as any direct or indirect contributions to this site or the shared task organizers, such as, but not limited to: results of automatic scoring programs; manual, qualitative and quantitative assessments of the data submitted; task and systems papers submitted.
44

5-
**Individual and Team Participation**: Participants can participate as individuals or as part of one team. Teams and individual participants must create exactly one account to participate in the Codabench competition. Team composition may not be changed once the Test Phase starts. Your system is named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organizers.
5+
**Individual and team participation**: Participants can participate as individuals or as part of one team. Teams and individual participants must create exactly one account to participate in the Codabench competition. Team composition may not be changed once the Test Phase starts. Your system is named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organizers.
66

77
**Scoring of submissions**: Submissions are evaluated with automatic and manual quantitative judgements, qualitative judgements, and any other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers. Organizers are under no obligation to release scores. Official scores may be withheld if organizers judge the submission incomplete, erroneous, deceptive, or violating the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission. If multiple submission files are uploaded during the Test Phase, the last submission file per group will be understood as the team's or participant's definitive submission and ranked as such in the task description paper.
88

9-
**Data usage**: The provided data should be used responsibly and ethically. Do not attempt to misuse it in any way, including, but not limited to, reconstructing test sets, any none-scientific use of the data, or any other unconscionable usage of the data. You may not redistribute the task data except in the manner prescribed by its licence.
9+
**Data usage**: The provided data should be used responsibly and ethically. Do not attempt to misuse it in any way, including, but not limited to, reconstructing test sets, any non-scientific use of the data, or any other unconscionable usage of the data. You may not redistribute the task data except in the manner prescribed by its licence.
1010

1111
**Specific conditions for closed and open tracks**: Participants agree to follow the specific conditions for [closed tracks](link-tbd) and [open tracks](link-tbd), which specify the type of data allowed for pretraining the model.
1212

13-
**Submission of systems description papers**: Participants having made at least one submission for a closed track during the Test Phase will be invited to submit a paper describing their system. Participants having made only submissions for open tracks will not be invited to submit a paper describing their system (see the specific conditions for closed and open tracks). For both tracks, we strongly encourage participants to provide a link to the code of their system(s) to organizers or the public at large (on the submission page?). We also encourage you to upload any systems and models to an open-source repository such as the HuggingFace Hub.
13+
**Submission of systems description papers**: Participants having made at least one submission for a closed track during the Test Phase will be invited to submit a paper describing their system. Participants having made only submissions for open tracks will not be invited to submit a paper describing their system (see the specific conditions for closed and open tracks). For both tracks, we strongly encourage participants to provide a link to the code of their system(s) to organizers or the public at large. We also encourage you to upload any systems and models to an open-source repository such as the HuggingFace Hub.
1414

1515
**Acknowledgements**: This shared task was created by OFAI with funding from the FFG project EKIP.

0 commit comments

Comments
 (0)