BREAKING NEWS: One Tree Hill’s Tyler Hilton Was Told Audiences ‘Hated’ His Character: ‘What Have I Done?’.giang

Though Tyler Hilton now travels the world to meet adoring One Tree Hill fans regularly, it turns out they didn’t initially welcome Chris Keller with open arms.

Hilton, 41, tells PEOPLE that when his character was first introduced on season 2 of the beloved series, the network told him that audiences had some thoughts.

“I came across as much more of an a—— than I meant to,” he recalls. “And the network was like, ‘People hated you on it.’ ”

While Hilton of course thought the review was a negative one, it turned out it was quite the opposite.

“I was like, ‘Oh, I’m so sorry’ and they were like, ‘We love it! We’ve got to bring you back,’ ” he adds.

Though landing a recurring character is usually a good thing for an actor, Hilton was nervous about fans’ reception.

“I remember thinking, ‘What have I done?’ And they were like, ‘We’ve got to keep bringing you back and you’ve got to keep doing that thing,’ ” he says. “I remember being like ‘What? This business is so weird.’ But they kept giving me lines that were funnier and funnier.”

Off-camera, Hilton was bonding with the rest of the cast and the group “became best friends instantly.” The fans on the other hand, took some time to come around, he adds.

“When I’d be going through airports or grocery stores or anything, people would be like, they just hated me,” he says.

Now, more than 10 years since the show went off the air, Hilton is still connecting with both fans of the show and his former costars. Just recently, he and his former costar Kate Voegele wrapped up a trio of shows for their One Tree Holiday tour in Boston, New York City and Philadelphia. The cast as a whole often reunites at various fan festivals in places across the world from Wilmington, North Carolina — where the show was filmed — to Paris, where cast members including Hilarie Burton, James Lafferty and Austin Nichols traveled this fall for the Tree Hill Is Your Home convention.

Even when Hilton is touring on his own, he notices the One Tree Hill fans come out in full force.

“People love One Tree Hill so much internationally,” he says. “When I tour all over the world, still at least half the crowd is One Tree Hill fans. I mean Australia, anywhere in Europe. It’s wild. It’s totally, totally wild.”

“It’s been long enough that the fans are like, a lot of them you recognize and see again, and now we all share this thing in common,” he adds. “Not to be cheesy, but it’s like we all have the same college alma mater. It’s like going to a college event ’cause that’s the age we were, that’s what these fans are celebrating — that time in our of life. And it’s like we all get to just celebrate the magic and the randomness that was One Tree Hill together.”