The winger says that the Kirkcaldy club put on a good performance and would have taken all three points if captain Kyle Benedictus hadn’t received a red card from referee Colin Steven, who Connolly says “killed the game”.
The result sees Rovers three and four points behind Partick Thistle and Inverness respectively but with six games left of the league season, Connolly says Raith have enough to finish in the top four.
“I thought we played well against Inverness,” he said.
“The boys have been doing really well recently. In the last four of five games they've played really well.
“You could see at the end of the game the fans were clapping us off because they know we played well and the ref has killed the game.
“Bene has given him a chance to make a decision. He won the ball but I think it was because of the reaction, although I don't think the referee even saw it.
“He was doing that with a lot of things during the game. There was one where we should have had a corner but he says he hasn't seen it.
“It was a bad refereeing performance.
"The boys were down but it wasn't like a defeat where we weren't good enough.
“Up until the sending off we were the dominant team and we were creating chances, they didn't create anything until the last few minutes.
“Everything seemed to go against us and it's felt like that for the second part of the season, but we just have to keep going.
"We're not out of it yet.
“Obviously Inverness have an advantage on us now but I think we can still catch them.
“If we keep playing the way we are and working hard I think we'll get wins.”
The 26-year-old, who is in his second spell at Stark’s Park, opened the scoring with his eighth goal of the season, but says the goal ultimately didn’t matter due to the defeat.
“It's come to me from a corner and I took a touch and just struck it.
“It went through a sea of bodies and I couldn't really see it after that, then I just saw it go into the back of the net.
“But it doesn't really mean anything. I would have rather we won and not scored.
“I've scored a few individually which is really good but I always think the team comes first and I thought we deserved to win.
“Once we had a man sent off though they started to come on to us and it's difficult after that because you can't get up the park.”
Connolly hopes Raith can take their sense of injustice out on Arbroath when they make the trip to Gayfield this weekend to face Dick Campbell’s second-placed side, who have only lost four games all season.
“Arbroath have been excellent this season. they deserve to be up there,” Connolly says.
“They're the team to look at.
"They work hard, everyone knows what they're doing and they're getting results.
“Up there the wind machine can be on which can make it a difficult game but we'll just keep going and do our best.
“We've been fighting and everyone is working really hard and if we continue that we'll have more wins than defeats in the few games that are left.”