Born in Barry, John played for Barry Town and Caerphilly, before joining English club Arsenal, who signed him amongst stiff competition, in January 1922 for a fee of £750. John made his Arsenal first-team debut on October 28, 1922 in a 2-1 defeat at home to Newcastle United, and quickly became a regular, succeeding Tom Whittaker at left half. His ability was such that soon after, made his debut for the Welsh national side, against Scotland on March 17, 1923; it was the first of fifteen caps.
John was displaced from the Arsenal side in 1923-24 thanks to competition from Billy Blyth and Andrew Young, but after being switched to left back to cover for Andy Kennedy, he was a near ever-present in 1924-25. Eventually however, John was switched back to left half, and this time he remained a first-team regular. A prodigious ball-winner and noted passer of the ball, John reached (but lost) the FA Cup Final with Arsenal in 1926-27, after a mistake by his compatriot and close friend Dan Lewis; it was John who consoled Lewis after the final whistle, assuring him he would get another chance to a win a medal (although Lewis never did).
John remained in the Arsenal side through the first half of the 1930s, despite competition from the likes of Charlie Jones. He finally won some silverware with an FA Cup win in 1929-30, followed by three First Division titles, in 1930-31, 1932-33 and 1933-34. John also scored Arsenal's only goal of the 1932 FA Cup Final, in which Arsenal were controversially beaten by Newcastle United. By this time he was one of the senior members of the Arsenal squad, and mentored many of the club's younger new arrivals, such as Alex James.
Although John played for Arsenal until his retirement in 1938, for the final three years of his career he was mainly a reserve player, having lost his place to Wilf Copping, and thus missed out on a medal in the Gunners' League win of 1934-35. Nevertheless, he played 470 times for the club in total, the most of any of Arsenal's pre-World War II players; as of 2006 he is tenth in the club's all-time appearances table.
On retiring in 1938, John joined the coaching staff at West Ham United. He later joined Torquay United as trainer, working under his former Highbury teammate Jack Butler, but when Butler left for Crystal Palace in May 1947, John followed, again as trainer. In 1949 he was appointed trainer-coach at Cardiff City, a position he held until March 1950 when he was appointed manager of Torquay United. He had an unsuccessful spell as manager at Plainmoor, winning only 7 of the 28 games he was in charge for and left his post in November 1950.
He finished his football career as a scout for Cardiff City. He died in Barry in 1982, at the age of 83. In 2008, his descendants loaned his shirts from the 1927, 1930 and 1932 FA Cup Finals to the Arsenal FC Museum.