The palace was built in the 15th century by Bishop John Alcock however just two towers from the original building remain. Bishops over the following centuries expanded and remodelled the palace.
During the Second World War the palace was used as a base for the British Red Cross and after this period it became a home for disabled children until its closure in the 1980s. Following this, the palace was purchased on a 99 years lease by the Sue Ryder Care organization although the palace went up for sale again in 2010. Later that year, local private school, King's Ely took over the lease from Sue Ryder Care and refurbished the palace to become the school new sixth form centre. It was opened by the Prince Richard, Duke of Gloucester on Friday 25 January 2013.