Joel married 3 times: to Margarett Nix, Mary Elizabeth Cochran, and Vinnie Langley. He and Mary had 4 children: Howard Brock, Verna Rainwater, Vera Thomason, and Vassie Rainwater.
