Benjamin Timmermans

Collective Intelligence 2017 – Trip Report

On June 15-16 the Collective Intelligence conference took place at New York University. The CrowdTruth team was present with Lora Aroyo, Chris Welty and Benjamin Timmermans. Together with Anca Dumitrache and Oana Inel we published a total of six papers at the conference.

Keynotes

The first keynote was presented by Geoff Mulgan, CEO of NESTA. He set the context of the conference by stating that there is a problem with technological development, namely that it only takes knowledge out of society and does not put it back in. Also, he made it clear that many of the tools we see today like Google Maps are actually nothing more than companies that were bought and merged together. This combination of things is what creates the power. He also defined what the biggest trends are in collective intelligence: the observation e.g. citizen generated data on floods, predictive models e.g. fighting fires with data, memory e.g. what works centers on crime reduction, and judgement e.g. adaptive learning tool for schools. Though, there are a few issues with collective intelligence: Who pays for all of this? What skills are needed for CI? What are the design principles of CI? What are the centers of expertise? These are all not yet clear. However, what is clear is that there is a new field emerging through combining AI with CI: Intelligence Design. We used to think systems resolve this intelligence, but actually we need to steer and design it.

In a plenary session there was an interesting talk on public innovation by Thomas Kalil. He defined the value of concreteness as things that happen when particular people or organisations take some action in pursuit of a goal. These actions are more likely to affect change if you can articulate who would needs to do what. He said he would like to identify the current barriers to prediction markets and areas where governments could be a user and funder of collective intelligence. This can be achieved through connecting people that are working to solve similar problems locally, e.g. in local education. Then change can be driven realistically, by making clear who needs to do what. Though, it was noted also that people need to be willing and able for change to work.

Parallel Sessions

There were several interesting talks during the parallel sessions. Thomas Malone spoke about using contest webs to address the problem of global climate change. He claims that funding science can be both straightforward and challenging, for instance government policy does not always correctly address the need of a domain issues, and even conflicts of interest may exist. Also, fundamental research can be tough to convince the general public of its use, as it is not sexy. Digital entrepreneurship is furthermore something that is often overlooked. There are hard problems, and there are new ways of solving them. It is essential now to split the problems up into parts, solve each of them with AI, and combine them back together.

Chris Welty presented our work on Crowdsourcing Ambiguity Aware Ground Truth at Collective Intelligence 2017.

Also Mark Whiting presented his work on Daemo, a new crowdsourcing platform that has a self-governing marketplace. He stress the fact that crowdsourcing platforms are notoriously disconnected from user interests. His new platform has a user driven design, in order to get rid of the flaws that exist in for instance Amazon Mechanical Turk.

Plenary Talks

Daniel Weld from the University of Washington presented his work on argumentation support in crowdsourcing. Their work uses argumentation support in crowd tasks to allow workers to reconsider their answers based on the argumentation of others. They found this to significantly increase the annotation quality of the crowd. He also claimed that humans will always need to stay in the loop of machine intelligence, for instance to define what the crowd should work on. Through this, hybrid human-machine systems are predicted to become very powerful.

Hila Lifshitz-Assaf of NYU Stern School of Business gave an interesting talk on changing innovation processes. The process of innovation has changed from a lane inventor, to labs, to collaborative networks, and now into open innovation platforms. The main issue with this is that the best practices of innovation fail in the new environment. In standard research and development there is a clearly defined and selectively permeable, whereas with open innovation platforms this is not the case. Experts can participate from in and outside the organisation. It is like open innovation: managing undefined and constantly changing knowledge in which anyone can participate. For this to work, you have to change from being a problem solve to a solution seeker. It is a shift from thinking: The lab is my world, to the world is my lab. Still, problem formulation is key as you need to define the problems in ways that cross boundaries. The question always remains, what is really the problem?

Poster Sessions

In the poster sessions there were several interesting works presented, for instance work on real-time synchronous crowdsourcing using “human swarms” by Louis Rosenberg. Their work allows people to change their answers through the influence of the rest of the swarm of people. Another interesting poster was by Jie Ren of Fordham University, who presented a method for comparing the divergent thinking and creative performance of crowds compared to experts. We ourselves had a total of five posters covering both poster sessions, which were received well by the audience.

Collective Intelligence Slides and Papers

Here is the full list of our papers published at the Collective Intelligence 2017 conference:


Chris Welty also presented our work on Crowdsourcing Ambiguity Aware Ground Truth at Collective Intelligence. The slides are available here:

Amsterdam Data Science – Coffee & Data: Controversy in Web Data

On 9th of June we are organising a Coffee & Data event with the Amsterdam Data Science community. The topic is “How to deal with controversy, bias, quality and opinions on the Web” and will be organised in the context of the COMMIT ControCurator project. In this project VU and UvA computer scientists and humanities researchers investigate jointly the computational modeling of controversial issues on the Web, and explore its application within real use cases in existing organisational pipelines, e.g. Crowdynews and Netherlands Institute for Sound and Vision.

The Agenda is as follows:

09:00-09:10 Coffee

Introduction & Chair by Lora Aroyo, Full Professor at the Web & Media group (VU, Computer Science)

09:10 – 09:25: Gerben van Eerten – Crowdynews deploying ControCurator

09:25 – 09:40: Kaspar Beelen – Detecting Controversies in Online News Media (UvA, Faculty of Humanities)

09:40 – 09:50: Benjamin Timmermans – Understanding Controversy Using Collective Intelligence (VU, Computer Science)

09:50 – 10:00: Davide Ceolin – (VU, Computer Science)

10:00 – 10:15: Damian Trilling – (UvA, Faculty of Social and Behavioural Sciences)

10:15 – 10:30: Daan Oodijk (Blendle)

10:30 – 10:45: Andy Tanenbaum – “Unskewed polls” in 2012

10:45 – 11:00: Q&A Coffee

The event takes place at the Kerkzaal (HG-16A00) on the top floor of the VU Amsterdam main building.

ControCurator demonstration at ICT Open 2017

Our demo of ControCurator titled “ControCurator: Human-Machine Framework For Identifying Controversy” will be shown at ICT Open 2017. In this demo the ControCurator human-machine framework for identifying controversy in multimodal data is shown. The goal of ControCurator is to enable modern information access systems to discover and understand controversial topics and events by bringing together crowds and machines in a joint active learning workflow for the creation of adequate training data. This active learning workflow allows a user to identify and understand controversy in ongoing issues, regardless of whether there is existing knowledge on the topic.

Watson Innovation Course wins ICT project of the year in education

watsoneducation

Yesterday at the Computable Awards the Vrije Universiteit, University of Amsterdam and IBM won the prize for “ICT project of the year in education” with the Watson Innovation Course. Furthermore, the project was highest rated across all nominees of all prize categories. The course is ongoing at the moment for the second time, with an improved setup and new state of the art tools for the students.

The course is run by Lora Aroyo, Anca Dumitrache, Benjamin Timmermans and Oana Inel from the VU, and Robert-Jan Sips and Zoltan Szlavik from IBM. In the course the students were challenged by Amsterdam Marketing to solve the issue of the increasing overcrowdedness of tourists in the city center of Amsterdam. The city is culturally rich with many places to visit, yet most visitors cluster around a limited set of popular locations. The students came up with ideas to motivate visitors to spread in the city and provide them with relevant information for their visit.

computable-award

Interested in working with IBM and ABN AMRO on an exciting innovation project?

Are you interested in working with IBM and ABN AMRO on an exciting innovation project?

Although the mortgage application process and the regulations surrounding that are clearly mapped, institutionalised and supported by automated systems, the process of orientation in the housing market is not. When looking for an appropriate place to live / open a business, clients of the bank are confronted with questions and considerations like the location, image of and facilities in the neighbourhood and the safety of this neighbourhood, energy labels, average price levels, future development plans, etc., surrounding one of the largest decisions of their life: the purchase of a house. At current, the bank is unclear about the steps customers take in the orientation process and with which extra services / answers / information their bank could support them.

Within this 3 month project, IBM, VU and UvA will join forces and will use the IBM Open Innovation approach to come up with a new data-driven service concept for one of the leading banks in the Netherlands. The team will consist of IBM Consultants (design / business strategy), an IBM programmer, 2 IBM researchers and researchers from VU Amsterdam and UvA.

To strengthen our team we are looking for 3 students for a 3-month-internship at IBM, potentially followed by a MSc thesis project deepening/continuing upon their work, in the following disciplines:

(1) Business/Service Innovation: Students with an entrepreneurial / service innovation background, looking to gain experience in the development of a real-life business case. The student should ideally have experience with focus groups and qualitative interviews, to help gain initial insights into the house orientation process.

(2) Crowdsourcing: Using crowdsourcing and the social web, to get a clear(er) picture of the demands, questions, uncertainties surrounding the purchase of a house. This project will complement the work done by student (1) with quantitative results and a larger scope.

(3) Open Data / Information Retrieval: Finding (open) datasets and retrieving datasources which would be able to provide insights in the questions identified by the work done by student (1) and (2).

If you are interested in an internship with IBM / ABNAMRO within the context of this project, please contact Lora Aroyo via lora.aroyo@vu.nl with a short motivation why you would like to work on it, CV and your availability in the coming 3-4 months.

Sign up for the Watson Innovation Course!

Have you ever wondered how we could provide tourists in Amsterdam with the best experience? Now is your chance to develop ideas, business cases and real prototypes of Watson to answer all questions tourists have.

The Watson Innovation course is a collaboration between the Vrije Universiteit, University of Amsterdam and IBM Netherlands. It offers a unique opportunity to learn about IBM Watson, cognitive computing and the meaning of such artificial intelligence systems in a real world and big data context. Students from Computer Science and Economics faculties will join their complimentary efforts and creativity in cross-disciplinary teams to explore the business and innovation potential of such technologies. Visit the course page to find out all the details.

Crowdsourcing brainstem tumors at Lowlands 2016

lowlands

Brainstem tumors are a rare form of childhood cancer for which there is currently no cure. The Semmy Foundation aims to increase the survival of children with this type of cancer by supporting scientific research. The Center for Advanced Studies at IBM Netherlands is supporting this research by developing a cognitive system that allows doctors and researchers to quicker analyse MRI-scans and better detect anomalies in the brainstem.

In order to gather training data, a crowdsourcing event was held at the festival Lowlands, which is a 3-day music festival that took place from 19-21 August 2016 and welcomed 55k visitors. At the science fair, IBM had a booth that hosted both this research and showcase of the Weather stations of the Tahmo project with TU Delft.

screenshot

In the crowdsourcing task, the participants were asked to draw the shape of the brainstem and tumor in an MRI scan. Gathering data on whether a particular layer of a scan contains the brainstem and determining its size should allow a classifier to recognize the tumors. Furthermore, the annotator quality can be measured with the CrowdTruth methodology by analysing the precision of the edges that were drawn in relation to their alcohol and drug use that we collected. The hypothesis is that people under influence can still make valuable contributions, but that these are of lower quality than sober people. This may make the reliability of online crowd workers more clear, because it is unknown under what conditions they make their annotations.

heatmap

The initial results in the heatmap of drawn pixels give an indication of the overall location of the brainstem, but further analysis will follow on the individual scans in order to measure the worker quality and generating 3d models.