James and Mary were in East Troy on the 1905 census just six months before his death. After his death, Mary went to live with her daughter Anna and her husband Albert Sengbusch in Lafayette.