Phelan started his career as part of the Leeds United youth system, and made 19 appearances for Leeds in the 1985-86 season, but was released on a free transfer by Leeds manager Billy Bremner. Phelan then spent a year at Swansea City, earning a move to top division Wimbledon a year later.
In his first season at Wimbledon, Phelan was a member of the team which won the FA Cup with a shock win over Liverpool in the final. He also helped Wimbledon finish seventh in the league on two occasions during his time there.
After five years at Wimbledon in which he made 198 appearances, Phelan was transferred to Manchester City for £2.5 million at the start of the 1992-93 season, equalling the British record transfer fee for a defender as well as equalling the club's record fee. At Manchester City, Phelan made 122 appearances over three and a half seasons. Arguably the best goal of Phelan's career came at Maine Road for Manchester City in an FA Cup tie against Tottenham Hotspur when the full-back ran the length of the pitch skipping several challenges and keeping his cool to slot home.
Phelan moved to Chelsea in November 1995, where he spent two years on the fringes of the first team (playing just 15 times in the league) before moving to Everton, where he played regularly under Joe Royle, until he suffered an injury from which he took nearly 18 months to recover.
During Phelan's injury, Walter Smith took over as Everton manager (after Howard Kendall's third spell as manager lasted just one season), and did not select Phelan following his recovery, instead loaning him to Crystal Palace. Phelan moved to Fulham in February 2000, and was part of the Fulham team which won promotion to the Premiership in 2001. However, Phelan was released by Fulham following promotion. Phelan then had a short spell with Sheffield United before moving to the United States, where he played for Charleston Battery and also coached young players. In October 2005, Phelan was named as player-coach of Otago United in New Zealand Football Championship. He remained as coach for four seasons up until the end of the 2008/09 season when due to poor results he was replaced by his assistant Malcolm Fleming for the 2009-10 season. Now, during the summer, Terry leaves his family to teach youth soccer with One on One Soccer.
He is now living in McAndrew Bay in Dunedin, New Zealand and is Football Development Manager for Football South, a football federation that manages the game from the provinces of South Canterbury southwards.