Ever since the Crusades when Christians from western Europe were fighting holy wars against Muslims in the near east western people have often perceived Islam as a violent and intolerant faith - even though when this prejudice took root Islam had a better record of tolerance than Christianity.
I've always felt that if you back down from a fear the ghost of that fear never goes away. It diminishes people.