When the Rivington Men began supplying AK-47s, Ballentine had become so ingrained with the company that the men gave him one of the new repeaters. Benny Lang observed him strut toward the Company H camp as he rode up and demanded to know from 1st Sgt. Nate Caudell why he was letting a "kaffir" go about armed. Caudell explained the situation and indicated he wasn't in Company H. Lang demanded to know who commanded the company and stormed off to confront Capt. Sidney Mitchell.
Lang browbeat Mitchell into taking the rifle from Ballentine. Ballentine seemed to take it calmly but that night ran off. He attempted to cross the Rapidan but was spotted by a Confederate patrol and shot dead. His body was brought back to the 47th where the men of Company H gave him a formal funeral.