Millbury, first settled by Europeans in 1716, has a long history as a New England mill town, from which the town's name is derived. The Blackstone River flows through the town and during the Industrial Revolution provided much of the water power to the town's many textile mills and factories. Its industrial history can be traced to the early 18th century. John Singletary began operating a mill on Singletary Brook and around 1753, built the S & D Spinning mill, which is still in operation, making it one of the oldest continuously operating mills in the United States. The mill also makes the red stitching that is on major league baseballs.

In 1808, Asa Waters II and his brother Elijah Asa and Elijah erected an armory which manufactured firearms for the U.S. Government during the Civil War.  With this wealth Asa Waters II began construction, in 1826, of a Federal-style mansion, near the town center on Elm Street. Designed by Boston architect Asher Benjamin, it was completed in 1832. It took two years to collect materials for construction of the house, including marble from Italy and bricks from Baltimore. Known as the Asa Waters Mansion, it is an icon of the town.

President William Howard Taft spent many summer vacations in Millbury as a young boy, attending the public schools for a season. When he grew older, he visited his grandparents most summers. He visited his aunt, Delia C. Torrey, during his presidency for the occasion of Millbury's 100th birthday. The Torrey House, where President Taft stayed during his visit, is commonly called the Taft House today.