Earlier this month, a heated confrontation between Sam Harris and Ben Affleck about Islam and violence on the Bill Maher show set off an internet firestorm that is only just beginning to cool off. Harris, the New Atheist author and intellectual, argued that modern Islam is “the mother lode of bad ideas,” and that it has an inherent propensity toward illiberalism and violence. Affleck argued that Harris’s view was racist, “like saying you’re a shifty Jew.”

The politics of the fallout were predictable: Conservatives tended to defend Harris, while liberals tended to charge him with generalizing from the actions of a small minority of Muslims to tar an entire culture. Before this episode recedes too far into memory, I want to note an interesting tension in the liberal approach.

A common theme in liberal commentary on this issue is that it is not fair — or at least not useful — to draw conclusions about a broader culture based on the actions of extremists. The vast majority of Muslims are horrified by ISIS and al-Qaeda, the argument goes, so it is bigoted to suggest that these groups are indicative of broader defects within Islamic culture.

This sentiment was neatly summarized in a response to the Harris-Affleck episode posted on the liberal website ThinkProgress:

In using the same kind of reasoning that officials have espoused to perpetuate Islamophobia … Harris mixes the violent actions of a few with millions of Muslims who are leading the initiative to show Islam as a peaceful religion.

This is not an unreasonable response. But it is manifestly at odds with the way liberals usually think about the relationship between extremism, culture and violence. Isolated acts of violence in the United States are frequently taken to be symptomatic of larger cultural ills. Consider ThinkProgress’s response to the UC Santa Barbara shootings from last Spring, in which Elliot Rodgers murdered six people and left behind a misogynistic rant as a suicide note:

While the debate in the aftermath of the shooting will likely focus on gun legislation … it is also becoming a discussion about widespread misogyny. The hashtag #YesAllWomen became a venue on Twitter for women to share personal stories and experiences. As the country tries to reckon with the tragedy, it will have to grapple with a climate in which men perpetrate violence against women on a daily basis, violence that is deeply embedded within our society.

Prominent liberal commentators similarly detected broad cultural roots to the shooting of Congresswoman Gabrielle Giffords in 2011. The New York Times editorialized that the shooter was part of a culture of “fear, anger and intolerance” that had pervaded U.S. politics and needed to be addressed.

These responses are not necessarily unreasonable either (at least at the time; in retrospect, the New York Times’ Giffords commentary seems questionable). Even though the shootings were perpetrated by psychopaths and horrified virtually all Americans, it is perfectly fair to ask whether they reflect toxic elements lurking beneath the surface of American society. Similarly, it is perfectly fair to ask if there is a link between terrorist acts perpetrated by a small minority of Muslims and certain aspects of Muslim societies at large.

In the past, liberals have quite rightly sought to examine the relationship between extremists, their ideas, and the society that incubates them. So maybe they shouldn’t be so quick to dismiss Sam Harris’ efforts to do the same.

Jason Willick, a senior studying history, is the editor in chief of Stanford Political Journal.