Blades keeper George Long twice saved well from Adam Armstrong and made a flying stop from a Ruben Lameiras shot.
John Fleck also failed to connect with an Armstrong pass and when Long spilled a cross it was adjudged not to have gone over the line.
Substitute Sharp headed home a Paul Coutts corner from close range before Sam Ricketts was sent off late on.
Sheffield United manager Nigel Adkins:
"We played against a very good footballing side and we had a game-plan. The lads worked very hard and we were solid, resolute and committed, which is great to see.
"The players were disciplined and organised. We want to be competitive and there is a resilience about the players.
"We want to be attack-minded, especially at home, but we've been punished before when teams have caught us on the counter-attack so we need to be nice and solid. We want to have that platform of being hard to beat and keep clean sheets."
Coventry manager Tony Mowbray:
"You've got to take your opportunities when they come along. It wasn't a game with lots of chances.
"The boys were saying in the dressing room that the ball was definitely over the line (Murphy shot). Joe Cole span away celebrating when he probably could have put it in himself.
"The referee had a difficult day. He got some right and he got some wrong so we have to accept that.
"The nature of the goal we conceded was frustrating. It's always disappointing and frustrating to concede a goal from a corner-kick. Reice has decided to come for the ball and missed it. It happens and I'm not sure whether the boy who scored knew much about it.
"Credit to them because I thought they set up well and defended well. We have to take it on the chin and move on to the next one."
Match ends, Sheffield United 1, Coventry City 0.
Second Half ends, Sheffield United 1, Coventry City 0.
Attempt saved. Ryan Flynn (Sheffield United) left footed shot from the centre of the box is saved in the centre of the goal.
Second yellow card to Sam Ricketts (Coventry City) for a bad foul.
Chris Basham (Sheffield United) wins a free kick in the attacking half.
Foul by Sam Ricketts (Coventry City).
Foul by Paul Coutts (Sheffield United).
Chris Stokes (Coventry City) wins a free kick on the right wing.
Paul Coutts (Sheffield United) wins a free kick in the defensive half.
Foul by Chris Stokes (Coventry City).
Corner,  Sheffield United. Conceded by Chris Stokes.
John Fleck (Coventry City) is shown the yellow card for a bad foul.
Stefan Scougall (Sheffield United) wins a free kick in the defensive half.
Foul by John Fleck (Coventry City).
Attempt missed. Billy Sharp (Sheffield United) right footed shot from the right side of the box is close, but misses the top left corner.
Attempt missed. Ryan Kent (Coventry City) right footed shot from outside the box is close, but misses the top right corner.
Substitution, Sheffield United. Ryan Flynn replaces Matt Done.
Billy Sharp (Sheffield United) is shown the yellow card.
Foul by Matt Done (Sheffield United).
John Fleck (Coventry City) wins a free kick on the right wing.
Foul by Dean Hammond (Sheffield United).
Ryan Kent (Coventry City) wins a free kick in the attacking half.
Substitution, Coventry City. Marc-Antoine Fortuné replaces Romain Vincelot.
Goal!  Sheffield United 1, Coventry City 0. Billy Sharp (Sheffield United) header from very close range to the high centre of the goal. Assisted by Paul Coutts with a cross.
Corner,  Sheffield United. Conceded by Aaron Martin.
Foul by Billy Sharp (Sheffield United).
John Fleck (Coventry City) wins a free kick in the defensive half.
Sam Ricketts (Coventry City) is shown the yellow card for a bad foul.
Stefan Scougall (Sheffield United) wins a free kick in the defensive half.
Foul by Sam Ricketts (Coventry City).
Corner,  Coventry City. Conceded by John Brayford.
Attempt blocked. Ryan Kent (Coventry City) right footed shot from the right side of the box is blocked.
Attempt saved. Aaron Martin (Coventry City) header from the left side of the box is saved in the centre of the goal.
Corner,  Coventry City. Conceded by Dean Hammond.
Corner,  Coventry City. Conceded by John Brayford.
Attempt missed. Stefan Scougall (Sheffield United) right footed shot from outside the box is too high.
Billy Sharp (Sheffield United) wins a free kick in the defensive half.
Foul by Aaron Martin (Coventry City).
Substitution, Coventry City. Ryan Kent replaces Ruben Lameiras.
Attempt saved. Billy Sharp (Sheffield United) right footed shot from outside the box is saved in the centre of the goal.
Coventry lost ground on League One leaders Burton after Billy Sharp sealed victory for Sheffield United.