The site was earmarked for a new Corn Exchange in 1868 to replace the existing corn exchange on St Andrews Street. The site was originally the remains of the Priory of Friars Hermits which had been on the site since the Middle Ages. The priory was passed to a museum in New Zealand. The foundation stone was laid by the Mayor in 1874 and the building was opened in 1875. A quarter of a million local bricks were used in various colours. The opening concert was a performance on 9 November by the Coldstream Guards and a local choral society.

A less popular usage is as an examination hall for students at Cambridge University.