President John Adams, elected in 1796 as the second President, was the first resident of the White House. Abigail Adams, President Adams' wife, was known to have complained about the largely unfinished new residence. President Thomas Jefferson, upon moving to the house in 1801, was also not impressed, and dismissed the house as being too big. Jefferson made several structural changes under architect Benjamin Henry Latrobe such as the addition of terrace-pavilions on either side of the main building and single-story wings for storage.

In addition to replacing the slate roof with one of sheet iron, Jefferson further improved the grounds by landscaping them in a picturesque manner. While James Madison was President from 1809 to 1817, the White House was torched by the British in the War of 1812. Although the fire was put out by a summer thunderstorm, all that remained were the outside, charred walls and the interior brick walls. Madison brought Hoban back to restore the mansion, which took three years. It was during this construction that the house was painted white. Hoban later added the South and North Porticos, using a slightly altered design by Latrobe. Thereafter, further more alterations were made.

Today, the White House Complex includes the executive residence, west wing, east wing, the Eisenhower executive office building—the former state department, which now houses offices for the president's staff and the vice president—and Blair House, a guest residence. The executive residence is made up of six stories—the ground floor, state floor, second floor, and third floor, as well as a two-story basement. The term White House is often used as a metonym for the Executive Office of the President of the United States and for the president's administration and advisers in general. The property is a National Heritage Site owned by the National Park Service and is part of the President's Park. In 2007, it was ranked second on the American Institute of Architects list of America's Favorite Architecture.