Sheffield came through the youth system at Norwich City, but made just one appearance for their first team when he deputised for the suspended Bryan Gunn in a 2-2 draw against Aston Villa at Carrow Road on 15 April 1989. While with Norwich, he had loan spells with Aldershot, Ipswich Town and Cambridge United before joining Cambridge on a permanent basis.
Whilst at Cambridge, Sheffield fought for the number 1 shirt with John Vaughan and enjoyed success with the club as they reached the old Division Two, as he eventually made the number 1 shirt at Cambridge his own following the departure of John Vaughan and his replacement John Filan. While at Cambridge had loan spells with Colchester United, Swindon Town and Hereford United.
After some excellent performances he left The Abbey Stadium to join local rivals Peterborough United in a £150,000 transfer where he played 62 games before moving to Plymouth Argyle for £100,000.
In 2001 Sheffield joined Yeovil Town on a free transfer and played 21 games, giving him a career total of 341 league appearances. He later also represented Saltash United F.C. in Cornwall.
Whilst playing for Swindon Town in the FA Premier League against Aston Villa, Jon Sheffield became the first player to wear the no. 40 shirt in the Premier League.
On 29 December, he was announced as Yeovil Town's new goalkeeping coach as a replacement for Ben Roberts.