Welcome to the CrowdTruth blog!

The CrowdTruth Framework implements an approach to machine-human computing for collecting annotation data on text, images and videos. The approach is focussed specifically on collecting gold standard data for training and evaluation of cognitive computing systems. The original framework was inspired by the IBM Watson project for providing improved (multi-perspective) gold standard (medical) text annotation data for the training and evaluation of various IBM Watson components, such as Medical Relation Extraction, Medical Factor Extraction and Question-Answer passage alignment.

The CrowdTruth framework supports the composition of CrowdTruth gathering workflows, where a sequence of micro-annotation tasks can be configured and sent out to a number of crowdsourcing platforms (e.g. Figure Eight and Amazon Mechanical Turk) and applications (e.g. Expert annotation game Dr. Detective). The CrowdTruth framework has a special focus on micro-tasks for knowledge extraction in medical text (e.g. medical documents, from various sources such as Wikipedia articles or patient case reports). The main steps involved in the CrowdTruth workflow are: (1) exploring & processing of input data, (2) collecting of annotation data, and (3) applying disagreement analytics on the results. These steps are realised in an automatic end-to-end workflow, that can support a continuous collection of high quality gold standard data with feedback loop to all steps of the process. Have a look at our presentations and papers for more details on the research.

CrowdTruth at HCOMP 2018

The CrowdTruth team is preparing to attend the sixth AAAI Conference on Human Computation and Crowdsourcing (HCOMP), taking place in Zurich, Switzerland, July 5-8. We are happy to announce we will be presenting two papers in the main track:

  • Capturing Ambiguity in Crowdsourcing Frame Disambiguation (Anca Dumitrache, Lora Aroyo, Chris Welty):
  • FrameNet is a computational linguistics resource composed of semantic frames, high-level concepts that represent the meanings of words. We present an approach to gather frame disambiguation annotations in sentences using a crowdsourcing approach with multiple workers per sentence to capture inter-annotator disagreement. We perform an experiment over a set of 433 sentences annotated with frames from the FrameNet corpus, and show that the aggregated crowd annotations achieve an F1 score greater than 0.67 as compared to expert linguists. We highlight cases where the crowd annotation was correct even though the expert is in disagreement, arguing for the need to have multiple annotators per sentence. Most importantly, we examine cases in which crowd workers could not agree, and demonstrate that these cases exhibit ambiguity, either in the sentence, frame, or the task itself, and argue that collapsing such cases to a single, discrete truth value (i.e. correct or incorrect) is inappropriate, creating arbitrary targets for machine learning.

  • A Study of Narrative Creation by Means of Crowds and Niches (Oana Inel, Sabrina Sauer, Lora Aroyo):
  • Online video constitutes the largest, continuously growing portion of the Web content. Web users drive this growth by massively sharing their personal stories on social media platforms as compilations of their daily visual memories, or with animated GIFs and memes based on existing video material. Therefore, it is crucial to gain understanding of the semantics of video stories, i.e., what do they capture and how. The remix of visual content is also a powerful way of understanding the implicit aspects of storytelling, as well as the essential parts of audio-visual (AV) material. In this paper we take a digital hermeneutics approach to understand what are the visual attributes and semantics that drive the creation of narratives. We present insights from a nichesourcing study in which humanities scholars remix keyframes and video fragments into micro-narratives i.e., (sequences of) GIFs. To support the narrative creation for humanities scholars a specific video annotation is needed, e.g., (1) annotations that consider literal and abstract connotations of video material, and (2) annotations that are coarse-grained, i.e., focusing on keyframes and video fragments as opposed to full length videos. The main findings of the study are used to facilitate the creation of narratives in the digital humanities exploratory search tool DIVE+.

We will also appear in the Collective Intelligence co-located event, where we will be discussing our paper False Positive and Cross-relation Signals in Distant Supervision Data (Anca Dumitrache, Lora Aroyo, Chris Welty), previously published at AKBC 2017:

Distant supervision (DS) is a well-established method for relation extraction from text, based on the assumption that when a knowledge-base contains a relation between a term pair, then sentences that contain that pair are likely to express the relation. In this paper, we use the results of a crowdsourcing relation extraction task to identify two problems with DS data quality: the widely varying degree of false positives across different relations, and the observed causal connection between relations that are not considered by the DS method. The crowdsourcing data aggregation is performed using ambiguity-aware CrowdTruth metrics, that are used to capture and interpret inter-annotator disagreement. We also present preliminary results of using the crowd to enhance DS training data for a relation classification model, without requiring the crowd to annotate the entire set.

If you are attending HCOMP 2018, we hope you will stop by our presentations!

Figure Eight & CrowdTruth Dataset on Medical Relation Extraction

As part of an initiative to highlight highly curated datasets that have been collected using crowdsourcing, Figure Eight (the AI and crowdsourcing platform formerly known as Crowdflower) has teamed up with CrowdTruth to highlight our work on medical relation extraction from sentences. Both the dataset and the task templates have been made available, and it is now possible to re-use the task template directly in any Figure Eight account. For more information, read the post on the Figure Eight website, as well as our papers:

Watson Innovation Course – Invited Lecture by Ken Barker, IBM Watson US

This week, the Watson Innovation course, a collaboration between the Vrije Universiteit, University of Amsterdam and IBM Netherlands, Centre for Advanced Studies (CAS) starts. The course offers a unique opportunity to learn about IBM Watson, cognitive computing and the meaning of such artificial intelligence systems in a real world and big data context. Students from Computer Science and Economics faculties join their complimentary efforts and creativity in cross-disciplinary teams to explore the business and innovation potential of such technologies.


This year, on 13th of November, Ken Barker from IBM Watson US will give an invited lecture. Here is an abstract of his invited lecture entitled “Question Answering Post-Watson”:

There is a long, rich history of Natural Language Processing and Question Answering research at IBM. This research achieved a significant milestone when the autonomous Question Answering system called “Watson” competed head-to-head with human trivia experts on the American television show, “Jeopardy!” Since that event, both Watson and QA/NLP research have barreled forward at IBM, though not always in the same direction.

In this talk, I will give a brief, biased history of Question Answering research and Watson at IBM, before and after the Jeopardy! challenge. But most of the talk will be a more technical presentation of our path of QA research “post-Watson”. The discussion will be in three parts: 1) Continuing research on traditional Question Answering technology beyond Jeopardy! 2) Work on transferring QA technology to Medicine and Healthcare; and 3) Recent research into exploratory, collaborative Question Answering against scientific literature.


Ken Barker Bio:

Ken Barker heads the Natural Language Analytics Department in the Learning Health Systems Organization at IBM Research AI. His current research examines the weaknesses of existing information gathering tools and applies Natural Language Processing to collaborative, exploratory question answering against scientific literature. Before joining IBM in 2011, he was a Research Faculty Member at the University of Texas at Austin, serving as Investigator on DARPA’s Rapid Knowledge Formation and Machine Reading Projects, as well as on Vulcan’s Digital Aristotle Project to build intelligent scientific textbooks. He was also an Assistant Professor of Computer Science at the University of Ottawa. His research there focused on Natural Language Semantics and Semi-Automatic Interpretation of Text.

Watson Innovation Course – Invited Lecture by Vanessa Lopez, IBM Ireland

This week, the Watson Innovation course, a collaboration between the Vrije Universiteit, University of Amsterdam and IBM Netherlands, Centre for Advanced Studies (CAS) starts. The course offers a unique opportunity to learn about IBM Watson, cognitive computing and the meaning of such artificial intelligence systems in a real world and big data context. Students from Computer Science and Economics faculties join their complimentary efforts and creativity in cross-disciplinary teams to explore the business and innovation potential of such technologies.


This year, on 16th of November, Vanessa Lopez from IBM Ireland Research will give an invited lecture. Here is an abstract of her invited lecture entitled “Cognitive solutions for Integrated Care”:

Cognitive technologies promise to have significant societal impact in domains where there is a need to transform multidisciplinary information into actionable services. From an industry perspective, the abundance ofdigital information gives an unprecedented opportunity to use data science to improve health and social care delivery.However,healthcare professionals have to quickly cope with large volume of information often scattered among unstructured case notes and health records to construct a care plan that addressessthe needs of the individual. In this talk, we look at the role of cognitive approaches to support care professionals to take better informeddecision,by capturing and interpreting patient-centric informationand learningfrom the actual practice of care professionals to suggest courses of action based on this holistic picture.With most of information still unstructured, we discuss the technologies, lessons learned and challenges behind this societal use case, in regards to knowledge acquisition, to find and combinemeaningful pieces of knowledge acrosssources with evidence for users’ information needs, and to facilitate intuitive human interaction, in which professionalsinteract with the system and the systems reacts and adapts its knowledge to give better suggestions,andfinally on how to validate the value of congitive systems with domain experts.


Vanessa Lopez Bio:

Vanessa Lopez is a researcher at IBM Research Ireland since 2012, where she investigates AI solutions for harnessing urban and web data as knowledge and to support users to query and find insights across data sources in a natural way, through a combination ofLinked Data, NLP and learning technologies for data integration. Her research has been applied to develop applications for smarter cities and Social and Health care to support care professionals to take better informed decisions.

Previous to joining IBM, she was a researcher at KMi (Open University) from 2003, where she investigated Question Answering interfaces for the Web of Data and received a PhD degree. She graduated in 2002 with a degree in computer engineer from the Technical University of Madrid (UPM), where she held an internship at the AI Lab. She has co-authored more than 40 publications in high impact conference and journals.

IBM Watson Masterclasses with VU Amsterdam and TU Delft

Why would decision makers attend these masterclasses?
To make informed business decisions on and around cognitive technologies, decision makers must understand the foundations, as well as the context, of these technologies.

What do we offer?
The IBM Benelux Center for Advanced Studies (CAS) teamed up with our long term collaborators in academia to deliver two 2-day masterclasses to educate decision makers about the technology. The first day of a masterclass will be at the university, covering the academic basics in an accessible way, while the second day is at IBM providing a more industrial angle of the topic.

The first masterclass, titled Foundations of Cognitive Computing, is delivered together with the Vrije Universiteit Amsterdam featuring renowned professors such Lora Aroyo, Guszti Eiben, and Frank van Harmelen (final list to be confirmed), but we will also have talks by IBM Research (Ken Barker), and demonstrations of past and present projects delivered locally by CAS. The preliminary dates for this Masterclass are 16-17 November 2017.

Topics covered (subject to change upon demand):

  • The past, present and future of Artificial Intelligence
  • Introduction to Cognitive Computing and Watson
  • How do Cognitive Systems learn?
  • Where is Watson now?
  • AI for the Masses (AI services in the cloud)
  • When AI Goes Bad (Ethics)
  • Demos and Corresponding Deep Dives

The second masterclass, titled The Internet of Everything and Everyone, is delivered with TU Delft, containing talks by prof. Geert-Jan Houben, dr. Alessandro Bozzon and several other researchers and students at the university, but also from IBM (John Cohn, Victor Sanchez, etc), CAS researchers and students. The date for for this masterclass are 7-8 December 2017.

Topics covered (subject to change upon demand):

  • Data collection for Cognitive systems:
    • big data,
    • sensor data,
    • human data
  • Solving real problems with IoT and with human computation
  • Data Science to connect machines and humans
  • Demos by students, faculty and IBM on how the technology can be used

What benefits can the masterclasses bring?
By gaining and understanding of what the technology is based on and what it can do, attendees can engage in deeper, more meaningful conversations about Cognitive, IoT, etc. They might become inspired to start projects using the technology, or perhaps the class can help them become convinced about the value a proposed IBM project.

Interested?  Contact us via casbnl@nl.ibm.com, or via Zoltan Szlavik and Benjamin Timmermans directly.