Jussie Smollett, who plays Jamal Lyon on the hit show Empire, is being targeted with death threats over the gay storyline he’s a part of on the show.

Smollett and show creator Lee Daniels have both been the subject of nastiness from homophobes, but Daniels says he won’t let it stop him from telling Jamal’s story, in part because it is also his own story. At a conference in L.A. over the weekend, Daniels said that a scene in a recent episode involving Jamal being thrown into a trash can as a child by his intolerant father was based on something that happened to him.

“I have to relate to these people. That trash can scene? It happened.”

Jussie hasn’t publicly responded to the hate mail, but said earlier this year–after coming out to Ellen DeGeneres–that he hopes his decision to be open about his sexuality will help others, particularly in the African-American community.

“It was a bigger deal to everyone than it was for me,” he said. “But at the same time I do understand why it is something to talk about…I didn’t talk to Ellen so that people could, be like ‘oh my God, let’s see what Jussie does in his bedroom in his private life.’ But I did talk to her so that people understood that they’re not alone. That’s all,” Smollett said.

For Jussie, the ability to play out storylines that have a real impact on people has been amazing.

“You don’t always get these experiences as an artist of [playing] roles where you feel the impact of what you’re doing, but I do. And that as an artist, as an activist, as a man, it’s feeding my soul,” Jussie Smollett said.