Anchoring the line of buildings that make up the Tarbox Inn is the federal style homestead built by Samuel Tarbox ca. 1825. An expansion soon after at the back of the house served as the kitchen until 1956. The oldest part of the house is the 1763 ell, which was originally situated on a hill to the north of the property.
This building was likely built by the original property owner Gabriel Stone. The 18th century ell retains its original beams and fireplace and connects the main homestead with the carriage house. It has served as the dining room since 1956.