Last week, conservative Fox News star Bill O’Reilly seemed to defend the supporters of gay marriage, saying that they have a “compelling argument.” At the same time, the pundit criticized opponents of gay marriage, stating that they haven’t been able “to do anything but thump the Bible.”

Now, O’Reilly is having to defend his comments against a backlash of Christians who feel the host was criticizing their religion. O’Reilly brought conservative talk radio host Laura Ingraham onto The O’Reilly Factor to provide a warm body to whom he could further explain his comments.

“If you’re going to stand up for heterosexual marriage and exclude gay marriage, if you’re going to do that, you’ve got to do it outside the Bible,” said O’Reilly. “You can’t cite the Bible, ’cause you’ll lose if you do it.”

Ingraham told O’Reilly that she thinks his comments are “disrespectful” to Christians. O’Reilly went on to try and explain that he makes a distinction between Americans’ privately held beliefs and how public policy should be made, insisting that he meant no offense to Christians. He also went on to double-down on his “thump the Bible” comment.

“There are Bible thumpers, and all they do is say, ‘I object to gay marriage ’cause God objects to it,” said O’Reilly. “You don’t win a policy debate in America with that.”