Baylor Walker changed the name of the Walker estate from Rye Field to Locust Grove and moved the residence from the fort hill to the riverside. He built a large frame house on the river in the 1760s, about the time that he married. It was several stories, a square with two rooms on each side of a center hall and a basement.

Baylor participated on the King and Queen County court in 1765. In 1766 he acquired 400 acres located in Orange County from his sister Susannah Walker Fleet and her husband William Fleet. In May 1770 he was named to a commision to examine witnesses and take depositions between the parties to a complaint of several misdemeanors against Mr. Roger Gregory.

Like his mother his estate gave brandy (used like cash) to the American Revolution cause.

In his will Baylor Walker divided his slaves among his sons. To his son Humphrey he gave the Locust Grove estate - approximately 800 acres and the house. His son Robert received a large tract of land in Amherst County on which he settled and where he died. Thomas received a large tract of land in Orange County, which he sold and then bought a tract of land in King William County. To his daughter Susannah he left three thousand pounds to be paid her by her brothers and some household items such as silver plate.