A silver corner stone........................built over with sandstone from Virginia.............

Construction was a laborious and time-consuming process: the sandstone used for the building had to be ferried on boats from the quarries at Aquia, Virginia; workers had to be induced to leave their homes to come to the relative wilderness of Capitol Hill; and funding was inadequate. By August 1796 the commissioners were forced to focus the entire work effort on the building's north wing so that it at least could be ready for government occupancy as scheduled. Even so, some third-floor rooms were still unfinished when the Congress, the Supreme Court, the Library of Congress, and the courts of the District of Columbia occupied the U.S. Capitol in late 1800.

In 1803, Congress allocated funds to resume construction. A year earlier, the office of the commissioners had been abolished and replaced by a Superintendent of the City of Washington. To oversee the renewed construction effort, Benjamin Henry Latrobe was appointed architect. The first professional architect and engineer to work in America, Latrobe modified Thornton's plan for the south wing to include space for offices and committee rooms; he also introduced alterations to simplify the construction work.