"Joseph Rufus Short "collected a barnful of the best heart pine he could find" until he had enough to build his house. He built the house on a corner lot in the town of Fremont where he owned and operated a ladies ready-to-wear store and a general store. The Register of Deeds shows that he purchased the land in 1891, however, the house was not built until 1901. Mr. Short was married to Dida Dickinson and together they had five children (Mana, Pansy, Wilber, Frank and Jurney). It is not clear if the children were born in the home or prior to it being built.

When Dida died, Mr. Short went out to Dickinson's home near Fremont and married Dida's younger sister Alice Irene. They had four children (Evenlyn, Pauline, John and Leonard.) Alice raised all of the children, with some calling her Mama and some calling her Aunt Alice.

According to a story told by Evelyn Short Brown, the front porch had a loose brick where her brother John hid his money. She said that she considered it her personal bank and often took money to buy herself a cone of ice cream. She said John never did find out. Mrs. Brown also said that her sister Pauline, who taught at Fremont School, would send a chosen student to ger her lunch from the house cook each day. Several Fremont residents recall the honor it was to be the chosen student.

Pauline Short lived in the house and took care of her mother until she died in 1971. Then, at age 60, Pauline married for the first time and put the house up for sale. When Pauline said that she was going to sell the family home, her brother John said he would burn the house down before he would let anyone else live in it. Fortunately, he did not and the house was sold to the Claiborne family, who lived in it for six years.

In 1977, Robert and Janet Lesser bought the house and began it's restoration. At Christmas of 1978, Evelyn and Pauline brought John to see the house. He stood in the front hall and cried, saying he was so happy to see his home was loved.

The house has 12 foot ceilings on the entire first floor and 9 foot ceilings on the second floor. It still showcases all of the original heart pine wainscot, floors and other woodwork. The original doorbell is still in the front door. The shed where the horse and cart were kept is still standing in the back yard and there are ancient oaks, magnolias, gardenias, camellias, daffodils and azaleas throughout the yard. There are a total of eight working fireplaces with beautiful ornate mantels and two belly stoves.

In 2005, Jeffery and Doris Reyes purchased the house and began the outdoor restoration project, cutting down 30 trees. Five of the downed pine trees had grown to 130 feet. This white house was painted in pastel colors to reflect a true Victorian house. Residents of Fremont routinely knock on the front door to comment on the beauty of the previously hidden home. In December 2005, Evelyn Short's son stopped by for a visit and had a truly emotional experience."