Initially fort St. Jago was a fortified chapel built by the Portuguese. In 1637 burned to the ground by the Dutch who placed large guns on the hill and forced the surrender of St. George Castle. To avoid the same faith as the Portuguese the Dutch built fort St. Jago, also called Fort Coenraadsburg, on this hill, opposite Elmina Castle to protect and defend Elmina (St. George) Castle.
The fort was always well protected by large garisons, and often served as a prison for European convicts. It has been used as a prison, hospital and rest house. Today it is in good condition and is now an inn and a restaurant.