This was initially called Fort Carolusborg when it was built in 1655 by the Swedes. It was taken by the Dutch but fell in British hands in 1664 after a Danish/British force captured it.In 1680 the British built their own fort here, using about three quarters of the old Swedish Fort. They then used it as an operation base for the Royal African Company.
In 1821, after the abolition of the slave trade, the British decided to take over the Cap Coast Case as an administrative base for the new British Crown Colony on the Gold Coast. This developed into a major defence point as well as base of operations. Large towers were built on the outskirts, Fort Victoria and Fort William (Smith’s Tower), in order to deny the outlying hills to attackers.