The building was designed by David Hamilton in the neoclassical style, built in ashlar stone at a cost of £12,000 and was completed in December 1816. The design involved a symmetrical main frontage with five bays facing northwest along Fore Street; the central section of three bays featured a full-height tetrastyle portico with Doric order columns supporting an entablature and a central roundel which was flanked by volutes and contained the town's coat of arms. The outer bays were fenestrated with sash windows and flanked by full-height pilasters supporting an entablature and a balustrade. At roof level, there was a central three stage tower, with round headed windows and Ionic order columns in the first stage, an octagonal piece with clock faces in the second stage and a belfry in the third stage. The tower was surmounted by a spire, which was 150 feet (46 m) high, and a weather vane. Internally, the principal room was the council chamber, which had a coffered and vaulted ceiling; the building also contained courtrooms and a series of prison cells.
The town buildings continued to serve as the headquarters of Port Glasgow Burgh Council, but ceased to be the local seat of government when the enlarged Inverclyde District Council was formed in 1975. After the building had stood empty for twelve years and had become seriously dilapidated, an extensive programme of refurbishment works was completed in August 1996. The works involved the removal of the rear wall, which had originally been harled, and the erection of a modern extension. These changes allowed the ground floor of the building to be used as a public library, and the first floor to become offices for a public body, Caledonian Maritime Assets Limited. Following further works, which included the reconfiguration of the library space and the creation of a new atrium, the building re-opened in August 2021