Papers

See the datasets used in these publications

2019

• Oana Inel and Lora Aroyo: Validation Methodology for Expert-Annotated Datasets: Event Annotation Case Study. LDK 2019 (in publication)
• Anca Dumitrache, Lora Aroyo and Chris Welty: A Crowdsourced Frame Disambiguation Corpus with Ambiguity. NAACL 2019 (in publication).

2018

• Anca Dumitrache, Lora Aroyo and Chris Welty: Crowdsourcing Semantic Label Propagation in Relation Classification. FEVER workshop at EMNLP 2018.
• Anca Dumitrache, Oana Inel, Benjamin Timmermans, Carlos Ortiz, Robert-Jan Sips, Lora Aroyo, Chris Welty: Empirical Methodology for Crowdsourcing Ground Truth, Semantic Web Journal (in publication), 2018.
• Oana Inel, Giannis Haralabopoulos, Dan Li, Christophe Van Gysel, Zoltán Szlávik, Elena Simperl, Evangelos Kanoulas, Lora Aroyo: Studying Topical Relevance with Evidence-based Crowdsourcing. In the Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM), 2018 .
• Anca Dumitrache, Oana Inel, Lora Aroyo, Benjamin Timmermans, Chris Welty: CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement.
• Tommaso Caselli, Oana Inel: Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation. Proceedings of the Workshop Events and Stories in the News 2018.
• Oana Inel, Sabrina Sauer and Lora Aroyo: A Study of Narrative Creation by Means of Crowds and Niches. Work-in-Progress at the AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2018).
• Anca Dumitrache, Lora Aroyo and Chris Welty: Capturing and Interpreting Ambiguity in Crowdsourcing Frame Disambiguation. HCOMP 2018.
• Anca Dumitrache, Lora Aroyo and Chris Welty: False Positive and Cross-relation Signals in Distant Supervision Data. Collective Intelligence 2018.
• Chantal van Son, Oana Inel, Roser Morante, Lora Aroyo, Piek Vossen: Resource Interoperability for Sustainable Benchmarking: The Case of Events. LREC 2018.
• Anca Dumitrache, Lora Aroyo and Chris Welty: Crowdsourcing Ground Truth for Medical Relation Extraction. ACM TiiS Special Issue on Human-Centered Machine Learning Vol. 8, Issue 2.

2017

• Anca Dumitrache, Lora Aroyo, Chris Welty: False Positive and Cross-relation Signals in Distant Supervision Data. AKBC at NIPS 2017.
• Lora Aroyo and Chris Welty: The Quantum Collective. Collective Intelligence 2017
• Benjamin Timmermans, Lora Aroyo and Tobias Kuhn, Kaspar Beelen, Evangelos Kanoulas and Bob van de Velde, Gerben van Eerten: ControCurator: Understanding Controversy Using Collective Intelligence . Collective Intelligence 2017
• Oana Inel and Lora Aroyo: Harnessing Diversity in Crowds and Machines for Better NER Performance . Research Track at ESWC 2017.
• Robert Iepsma, Theo Gevers, Oana Inel, Lora Aroyo, Robert-Jan Sips and Zoltan Szlavik: Crowdsourcing for Video Event Detection. Collective Intelligence 2017
• Oana Inel and Lora Aroyo: Improving NER with Diversity in Machines and Crowds. Collective Intelligence 2017
• Anca Dumitrache, Oana Inel, Benjamin Timmermans and Lora Aroyo: Crowdsourcing Ambiguity-Aware Ground Truth. Collective Intelligence 2017
• Anca Dumitrache, Lora Aroyo and Chris Welty: Disagreement in Crowdsourcing and Active Learning for Better Distant Supervision Quality. Collective Intelligence 2017
• Benjamin Timmermans, Kaspar Beelen, Lora Aroyo, Evangelos Kanoulas, Tobias Kuhn, Bob van de Velde, Gerben van Eerten: ControCurator: Human-Machine Framework For Identifying Controversy. ICT Open 2017

2016

• Zhaochun Ren, Oana Inel, Lora Aroyo, Maarten de Rijke: Time-aware Multi-Viewpoint Summarization of Multilingual Social Text Streams . CIKM 2016: 25th ACM Conference on Information and Knowledge Management.
• Benjamin Timmermans and Lora Aroyo: Capturing a Collective Intelligence for Cognitive Computing using CrowdTruth. Collective Intelligence 2016
• Oana Inel: Machine-Crowd Annotation Workflow for Event Understanding Across Collections and Domains . PhD Symposium at ESWC 2016.
• Benjamin Timmermans: Exploiting disagreement through open-ended tasks for capturing interpretation spaces. PhD Symposium at ESWC 2016
• Oana Inel, Tommaso Caselli and Lora Aroyo: Crowdsourcing Salient Information from News and Tweets . LREC 2016
• Emiel van Miltenburg, Benjamin Timmermans and Lora Aroyo: The VU Sound Corpus: Adding more fine-grained annotations to the Freesound database. LREC 2016
• Tommaso Caselli, Rachele Sprugnoli and Oana Inel: Temporal Information Annotation: Crowd vs. Experts . LREC 2016
• Anca Dumitrache, Oana Inel, Benjamin Timmermans, Lora Aroyo, Robert-Jan Sips and Chris Welty: CrowdTruth: Methodology for Harnessing Disagreement in Gathering Annotated Data. ICT Open 2016

2015

• Anca Dumitrache, Oana Inel, Benjamin Timmermans, Lora Aroyo and Robert-Jan Sips: CrowdTruth: Machine-Human Computation Framework for Harnessing Disagreement in Gathering Annotated Data. DIR 2015
• Carlos Martinez-Ortiz, Lora Aroyo, Oana Inel, Stavros Champilomatis, Anca Dumitrache and Benjamin Timmermans: Provenance-driven Representation of Crowdsourcing Data for Efficient Data Analysis. IEEE eScience 2015
• Anca Dumitrache, Lora Aroyo, Chris Welty: CrowdTruth Measures for Language Ambiguity: The Case of Medical Relation Extraction. LD4IE at ISWC 2015.
• Anca Dumitrache, Lora Aroyo, Chris Welty: Achieving Expert-Level Annotation Quality with CrowdTruth: The Case of Medical Relation Extraction. BDM2I at ISWC 2015.
• Benjamin Timmermans, Lora Aroyo and Chris Welty: Crowdsourcing ground truth for Question Answering using CrowdTruth. WebSci 2015.
• Anca Dumitrache: Crowdsourcing Disagreement for Collecting Semantic Annotation. PhD Symposium at ESWC 2015.

2014

• Lora Aroyo, Chris Welty: Truth is a Lie: 7 Myths about Human Annotation, AI Magazine 2014. (pdf)
• Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014.
• Oana Inel, Khalid Khamkham, Tatiana Cristea, Arne Rutjes, Jelle van der Ploeg, Lora Aroyo, Robert-Jan Sips, Anca Dumitrache and Lukasz Romaszko: Crowd Truth: Machine-Human Computation Framework for Harnessing Disagreement in Gathering Annotated Data. ISWC-RBDS 2014.

2013

• Lora Aroyo, Chris Welty: Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. ACM WebSci 2013.
• Lora Aroyo, Chris Welty: Measuring crowd truth for medical relation extraction. AAAI Fall Symposium on Semantics for Big Data (2013)
• Oana Inel, Lora Aroyo, Chris Welty, Robert-Jan Sips: Domain-independent quality measures for Crowd Truth Disagreement. Detection, Representation, and Exploitation of Events in the Semantic Web, 2 (2013)
• Anca Dumitrache, Lora Aroyo, Chris Welty, Robert-Jan Sips, Anthony Levas: Dr. Detective: combining gamification techniques and crowdsourcing to create a gold standard in medical text. CrowdSem 2013: 16 – 31.
• Oana Inel, Lora Aroyo, Chris Welty, Robert-Jan Sips: Exploiting crowdsourcing disagreement with various domain-independent quality measures. DeRiVe 2013.
• Guillermo Soberon, Lora Aroyo, Chris Welty, Oana Inel, Manfred Overmeen, Hui Lin: Content and Behaviour Based Metrics for Crowd Truth. CrowdSem 2013: 45 – 58.
• Lora Aroyo, Chris Welty: Harnessing disagreement in crowdsourcing a relation extraction gold standard Technical report RC25371 (WAT1304-058), IBM Research 4 (2013)
• IBM Inspire: IBM en Vrije Universiteit slaan handen in elkaar. (Dutch)

2012

• Lora Aroyo, Chris Welty: Harnessing disagreement for event extraction. DeRiVe 2012: 31 – 40.