Brick Academy is the nickname for a Federal-style brick building built in 1809 to meet the growing needs of the Basking Ridge Classical School located in the Basking Ridge section of Bernards Township, Somerset County, New Jersey. That school existed prior to 1799, at least 10 years before the construction of this building in 1809. The brick building was constructed for the elementary school run by local Presbyterian pastor, Rev. Robert Finley. This was about halfway through Rev. Finley's time at Basking Ridge. During the time he ran the school, attendance grew from fewer than 12 to an average near 25 students, and sometimes as high as 40 students. Students came from near & far, mostly from prominent families. The school was a high end preparatory school for boys who generally continued on to the College of New Jersey, later, known as Princeton University. In 1817, Rev. Finley quit Basking Ridge to briefly become president of the University of Georgia in Athens, GA. By 1828, the "Brick Academy" corporation was formed and the building continued use as a private, then public school in 1853, before being used for other purposes.

Samuel Lewis Southard was a prominent American statesman of the early 19th century, serving as a U.S. Senator, Secretary of the Navy, and the tenth governor of New Jersey. He also served as President pro tempore of the Senate, and was briefly first in the presidential line of succession.