Kavanagh's first clubs were Ballybrack and St Joseph’s boys where his talent was soon spotted by scouts.
Wayside Celtic signed him and he became an instant hit helping them win the FAI Intermediate Cup in his first season. A year later they reached the top of the table and in the last round of games there was a play off to decide the champions. That game still lives in the memory as his side clinched the title winning 5-0 and he scored twice and credited with assists for the other three goals.
During this time he played an under-20 amateur international against Spain and helped the Irish win 3-0.
After that game the offers flooded in and the first offer came from UCD AFC , not far from home and he signed in June 2006 for four months on amateur terms. He scored five minutes into his League of Ireland debut as a substitute against Waterford United on June 26. His season was interrupted as he had already booked holidays before signing for UCD; however, he got a run in the side towards the end of the season and scored three goals in UCD's last three games to help the Students to sixth place in the league. The highlight was the winner at Dalymount Park against Bohemians on the 27th of October 2006. A week later he was sprung from the bench and scored the winner against Sligo and UCD finished mid-table.
Next up was an approach from overseas scouts that finally brought him face to face with Steve Bruce at St Andrews. The former Manchester United star was very honest and explained that his side were struggling in the Premiership but he would be guaranteed games with the reserves and he was pleased with the offer. He got regular football and scored regularly but the highlight was training with the first team. As the season progressed the ‘Brum’ were losing the survival battle and although the reserves were going well, relegation cast a cloud over his future participation in the second biggest English city. Bruce was under greater pressure when he met Paddy for pre-season training later in the year and told him he had been given a war chest to buy players but the best he could guarantee him was another season in the reserves. After careful consideration he returned to Ireland in the knowledge there was plenty of offers on the table at home.
Bray Wanderers welcomed him with open arms and he got great experience and again attracted attention from the bigger boys like Derry City and Bohemians but then an offer came from Shamrock Rovers he could not refuse.
He made his Rovers debut in Cork on 11 August 2009, and made his home debut at Tallaght Stadium on 15 August 2009 against UCD in the FAI Cup, scoring his first goal for the club. His younger brother James currently plays for Bray Wanderers.
He earned a Republic of Ireland U23 cap in November 2007. He was named Player of the Month in the League of Ireland in April 2008. On 27 July 2009, he signed for Shamrock Rovers.