Rudley was born in 1910 (some sources say 1911) in Philadelphia, and attended Temple University. He left Temple after winning a scholarship to Eva Le Gallienne's Civic Repertory Theatre.
He began appearing on stage in 1926. His Broadway debut was in Did I Say No in 1931. He also appeared in stage productions of The Threepenny Opera, Abe Lincoln in Illinois and Macbeth.
In 1940, he appeared in the film version of Abe Lincoln in Illinois. For the next four decades he appeared in dozens of supporting film roles, including The Seventh Cross and Rhapsody in Blue, the film biography of George Gershwin in which he portrayed Ira Gershwin. He also appeared in A Walk in the Sun, Joan of Arc, and The Young Lions, in which he played an unsympathetic army officer.
On television, he appeared in both drama, often as a military person, and comedy. He also appeared on My Friend Flicka. Im 1957, he guest starred as a Prussian cavalry officer in an episode of the syndicated western series, Boots and Saddles. From 1957-1959, he co-starred in the role of Sam Brennan in NBC's western drama, The Californians, set in the San Francisco gold rush of the 1850s. His co-stars included Sean McClory and Adam Kennedy.
In 1959, he appeared as John McAuliffe on the syndicated television series Border Patrol, with Richard Webb. Rudley guest starred twice in the role of Jeremy
Thorne NBC's western series Laramie. In the 60s, he was a regular in a short-lived television
vehicle for Juliet Prowse called "Mona McCluskey." In 1963, he appeared in two episodes of The Beverly Hillbillies as the psychiatrist, Dr Twombey. In 1973, he guest starred in one episode of Lorne Greene's ABC crime drama Griff.
Rudley, however, is best remembered for his role as Eve Arden's husband, attorney Herb Hubbard, in NBC's sitcom, The Mothers-in-Law.