There is an increasing interest in using crowdsourcing to collect labels for machine learning. Currently, many companies provide crowdsourcing services such as Amazon Mechanical Turk and CrowdFlower. A remarkable advantage of crowdsourcing is that we can obtain a large number of labels at the low cost of pennies per label. However, these labelers are not experts, so the labels collected from them...